Tagged
#tokenization
2 articles
models
Tokens and tokenization: why models see text strangely
Models don't read letters or words — they read tokens. Understanding that one fact explains spelling slips, odd costs, and why context limits work as they do.
#tokens#tokenization#context-window
05-14 16:37·7 min read
research
Tokenizers and why they matter for languages
A language model never sees words. It sees tokens. How text gets chopped into tokens quietly decides cost, speed, and fairness across languages.
#tokenization#languages#nlp
05-05 08:17·7 min read