Tagged
#distillation
2 articles
research
Distillation: teaching small models from big ones
Knowledge distillation trains a small model to imitate a large one. The trick is not copying answers, but copying the way the big model is unsure.
#distillation#compression#training
05-21 13:52·7 min read
models
Quantization and distillation: making models smaller
"Two different ways to shrink a model — one changes its numbers, the other trains a smaller copy. Here is how each works and when to reach for it."
#quantization#distillation#model-compression
04-12 16:37·7 min read