Skip to main content

We gratefully acknowledge support from the Simons Foundation, member institutions, and all contributors. Donate

stat > arXiv:2503.13751

Help | Advanced Search

All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
Search

GO

quick links

Login Help Pages About

Statistics > Machine Learning
arXiv:2503.13751 (stat)

[Submitted on 17 Mar 2025] Title:Optimizing ML Training with Metagradient Descent Authors:Logan Engstrom, Andrew Ilyas, Benjamin Chen, Axel Feldmann, William Moses, Aleksander Madry View a PDF of the paper titled Optimizing ML Training with Metagradient Descent, by Logan Engstrom and 5 other authors View PDF Abstract:A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients -- gradients through model training -- at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we greatly improve on existing dataset selection methods, outperform accuracy-degrading data poisoning attacks by an order of magnitude, and automatically find competitive learning rate schedules.
Subjects: Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) Cite as: arXiv:2503.13751 [stat.ML] (or arXiv:2503.13751v1 [stat.ML] for this version) https://doi.org/10.48550/arXiv.2503.13751

Focus to learn more
arXiv-issued DOI via DataCite

Submission history From: Andrew Ilyas [view email] [v1] Mon, 17 Mar 2025 22:18:24 UTC (368 KB)

Full-text links: Access Paper:

View a PDF of the paper titled Optimizing ML Training with Metagradient Descent, by Logan Engstrom and 5 other authors View PDF TeX Source Other Formats view license
Current browse context: stat.ML < prev | next >

new | recent | 2025-03 Change to browse by: cs cs.AI cs.LG stat

References & Citations

NASA ADS Google Scholar Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation ×
loading...
Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle   

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle   

Connected Papers (What is Connected Papers?)

Litmaps Toggle   

Litmaps (What is Litmaps?)

scite.ai Toggle   

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle   

alphaXiv (What is alphaXiv?)

Links to Code Toggle   

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle   

DagsHub (What is DagsHub?)

GotitPub Toggle   

Gotit.pub (What is GotitPub?)

Huggingface Toggle   

Hugging Face (What is Huggingface?)

Links to Code Toggle   

Papers with Code (What is Papers with Code?)

ScienceCast Toggle   

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle   

Replicate (What is Replicate?)

Spaces Toggle   

Hugging Face Spaces (What is Spaces?)

Spaces Toggle   

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower   

Influence Flower (What are Influence Flowers?)

Core recommender toggle   

CORE Recommender (What is CORE?)

Author Venue Institution Topic

About arXivLabs  

arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

About Help

Contact
Subscribe

Copyright Privacy Policy

Web Accessibility Assistance

arXiv Operational Status Get status notifications via email or slack