"Ai2 tested DataDecide across a wide range of datasets and model sizes, using 10 benchmarks to evaluate how well small models predict large-scale performance. The findings arenโt earth-shattering, but they present useful takeaways for AI developers and researchers.
For one, Ai2 found that small models (around 150 million parameters) can predict large-scale outcomes with surprising accuracy. Some benchmarks reached over 80% decision accuracy using just 0.01% of the compute compared to billion-parameter models.
Since small-model experiments use less compute than other methods, developers donโt need to run full-scale tests just to predict outcomes. โThe promise of this work is lower compute costs during training,โ said Pijanowski.
Ai2 found that scaling laws didnโt outperform the simpler method of ranking datasets by small-model results. Scaling laws, a more sophisticated and more costly testing method, aim to predict how accuracy improves with model size. For now, โjust stick with ablating things at one scale,โ advised Magnusson.
The findings should give LLM devs pause for thought, Hunt said: โThere are scaling laws that have been derived from empirical studies between data volume, compute resources and performance. Ai2โs research points out that we may want to revisit some of those assumptions.โ"
https://thenewstack.io/new-tools-help-llm-developers-choose-better-pre-training-data/
#AI #GenerativeAI #LLMs #AITraining #SLMs
