How language models keep learning
A collaborative review of eleven continual-learning methods and four approaches to reducing catastrophic forgetting in language models.
The question
When a model learns something new, it can lose capabilities it learned before. Our review explored approaches to this problem of catastrophic forgetting in natural language processing.
My contribution
I co-authored the academic review with Cristóbal Mora Ramírez, Francisco Javier Suárez Pedraza and Joy Zhong as part of our NLP studies at IE. Our contribution was to compare and explain published methods; the underlying research and figures belong to their cited authors.
The approach
- Weight regularisation: constrain changes to parameters that matter for earlier tasks.
- Experience replay: revisit previous examples while learning new ones.
- Architectural techniques: adapt model structure to accommodate new tasks.
- Model distillation: transfer knowledge between models.
What came out of it
A 21-page review covering eleven published methods, shared on LinkedIn as an accessible starting point for deeper reading. It is a literature review, not a claim to have invented or benchmarked those methods.
Explore the source.
Read the review announcement and documentLet’s talk.
About a challenge, an idea, or something we could do together.