Tokens and Embeddings

- Tokenization as dictionary for model input - Tokens → IDs → contextual embeddings - Semantic meaning emerges only in embeddings - Transformer layers reshape embeddings by context - Pretrained embeddings accelerate domain understanding - Good tokenization reduces loss, improves learning - Tokenizer choice impacts RAG chunking - Compression tradeoffs differ by domain needs - Tokenization affects inference cost and speed - Compare BPE, SentencePiece, custom tokenizers - Emerging trend: byte-level latent transformers - Generations of embeddings add deeper semantics - Similarity measured via dot products, distance - Embeddings enable search, clustering, retrieval systems