subword tokenization solves: handles rare words by breaking into known pieces

WordPiece tokenization does: similar to BPE but uses likelihood instead of frequency

WordPiece tokenization splits words into subwords based on token likelihood rather than frequency

Unigram tokenization does: starts with large vocabulary and prunes using EM

Unigram tokenization starts with a large vocabulary and prunes using EM

the tokenizer's special tokens do: [CLS], [SEP], [PAD], [MASK] have specific roles

[CLS] marks the start of input, [SEP] denotes separation, [PAD] fills space, [MASK] hides words for prediction

[CLS] pooling does: uses the first token's embedding as the sentence representation

CLS pooling: uses the first token's embedding as the sentence representation

Greedy vs beam search decoding: greedy picks best token, beam maintains k candidates

Greedy picks best token, beam maintains k candidates

BPE tokenization does: iteratively merges the most frequent byte pairs

BPE tokenizes text by iteratively merging the most frequent byte pairs