To tackle this and make AI training cost-effective, researchers have found a new method called "Selective Forgetting."
What is Selective Forgetting? Selective forgetting is a technique used in machine learning models to improve their ability to learn new languages. It involves periodically erasing information from the model's memory during training. This can be done by erasing the embedding layer, which stores information about the building blocks of words, or resetting the entire model at regular intervals.
Why Selective Forgetting? Suppose you want to add more language to an AI model trained before, say, 100 languages. To add a new one conventionally, you must retrain the entire AI model, which will be expensive and time-consuming.
How Selective Forgetting Works ? Selective Forgetting works by periodically erasing information ( adaptive forgetting ) during the training process, allowing the model to learn a new language more easily.
Scores On a common measure of language accuracy, with full training data: Standard approach: 86.1 Forgetting approach: 85.1
After retraining on new languages using much smaller datasets (only 5 million tokens instead of 70 billion): Standard approach: Accuracy dropped to 53.3 on average Forgetting approach: Accuracy dropped to 62.7 on average
When computational limits were imposed during retraining (training length cut from 125,000 steps to 5,000 steps): Standard approach: Accuracy plunged to 37.2 (essentially random guessing) Forgetting approach: Accuracy decreased to 57.8 on average