Tagged
#transformers
3 articles
research
Why context length is hard to scale
A longer context window sounds like a simple knob to turn. Underneath it fights a cost that grows faster than the text — and attention that spreads thin.
#context-window#attention#scaling
06-08 18:48·7 min read
research
Attention, in plain language
Attention sounds technical, but the idea is something you do every time you read. Here is what it really means inside a language model, without the math.
#attention#transformers#context
04-30 11:26·7 min read
research
The transformer architecture, explained without math
The transformer is usually drawn as a wall of equations. Strip that away and it is one elegant idea: let every word decide which other words matter.
#transformers#architecture#attention
04-15 10:54·7 min read