Tagged
#training
7 articles
Catastrophic forgetting and continual learning
Teach a neural network something new and it tends to forget what it knew. This stubborn problem is why models learn in big batches, not in a stream.
How large language models are trained, in plain language
Training a language model happens in stages, not one magic step. Here is what each stage does, in plain language, and why the order matters.
AI and your data: what training on your inputs means
When a service says it may train on your inputs, what does that actually mean for your text, files, and ideas? A plain-language guide to the trade.
Distillation: teaching small models from big ones
Knowledge distillation trains a small model to imitate a large one. The trick is not copying answers, but copying the way the big model is unsure.
Synthetic data: training models on model output
When real data runs short, models can generate their own training data. It is powerful, slightly circular, and dangerous if you forget where it came from.
Scaling laws: bigger, but why
"Make it bigger" sounds like a slogan, not a science. Scaling laws are what turned it into one. Here is what they actually say, and what they do not.
Pretraining vs fine-tuning vs alignment
Three words get blurred together when people describe how models are made. They are different stages with different jobs. Here is what each one does.