Research collection
#pretraining
1 paper tagged Pretraining. Each paper states what it shows and what it does not prove.
1 results
A learning-rate schedule built around update size
AIR uses an effective-update-based schedule and has lower terminal training loss than cosine in two tested model scales.