Tagged
#attention
4 articles
research
Why context length is hard to scale
A longer context window sounds like a simple knob to turn. Underneath it fights a cost that grows faster than the text — and attention that spreads thin.
#context-window#attention#scaling
06-08 18:48·7 min read
models
Context windows explained: tokens, attention, and where long context breaks
A bigger context window is not the same as better memory. Here is what a context window really is, why long inputs degrade, and how to design around it.
#context-window#tokens#attention
06-02 10:06·7 min read
research
Attention, in plain language
Attention sounds technical, but the idea is something you do every time you read. Here is what it really means inside a language model, without the math.
#attention#transformers#context
04-30 11:26·7 min read
research
The transformer architecture, explained without math
The transformer is usually drawn as a wall of equations. Strip that away and it is one elegant idea: let every word decide which other words matter.
#transformers#architecture#attention
04-15 10:54·7 min read