NeurIPS 2024 Wrapped 🌯

What happens when you bring over 15,000 machine learning nerds to one city? If your guess didn't include racism, sabotage and scandal, belated epiphanies, a spicy SoLaR panel, and many fantastic research papers, you wouldn't have captured my experience. In this episode we discuss the drama and takeaways from NeurIPS 2024.

Posters available at time of episode preparation can be found on the episode webpage.

EPISODE RECORDED 2024.12.22

• (00:00) - Recording date
• (00:05) - Intro
• (00:44) - Obligatory mentions
• (01:54) - SoLaR panel
• (18:43) - Test of Time
• (24:17) - And now: science!
• (28:53) - Downsides of benchmarks
• (41:39) - Improving the science of ML
• (53:07) - Performativity
• (57:33) - NopenAI and Nanthropic
• (01:09:35) - Fun/interesting papers
• (01:13:12) - Initial takes on o3
• (01:18:12) - WorkArena
• (01:25:00) - Outro

Links

Note: many workshop papers had not yet been published to arXiv as of preparing this episode, the OpenReview submission page is provided in these cases. 

• NeurIPS statement on inclusivity
• CTOL Digital Solutions article - NeurIPS 2024 Sparks Controversy: MIT Professor's Remarks Ignite "Racism" Backlash Amid Chinese Researchers’ Triumphs
• (1/2) NeurIPS Best Paper - Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
• Visual Autoregressive Model report this link now provides a 404 error
  • Don't worry, here it is on archive.is
• Reuters article - ByteDance seeks $1.1 mln damages from intern in AI breach case, report says
• CTOL Digital Solutions article - NeurIPS Award Winner Entangled in ByteDance's AI Sabotage Accusations: The Two Tales of an AI Genius
• Reddit post on Ilya's talk
• SoLaR workshop page

Referenced Sources

• Harvard Data Science Review article - Data Science at the Singularity
• Paper - Reward Reports for Reinforcement Learning
• Paper - It's Not What Machines Can Learn, It's What We Cannot Teach
• Paper - NeurIPS Reproducibility Program
• Paper - A Metric Learning Reality Check

Improving Datasets, Benchmarks, and Measurements

• Tutorial video + slides - Experimental Design and Analysis for AI Researchers (I think you need to have attended NeurIPS to access the recording, but I couldn't find a different version)
• Paper - BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
• Paper - Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
• Paper - A Systematic Review of NeurIPS Dataset Management Practices
• Paper - The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track
• Paper - Benchmark Repositories for Better Benchmarking
• Paper - Croissant: A Metadata Format for ML-Ready Datasets
• Paper - Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox
• Paper - Evaluating Generative AI Systems is a Social Science Measurement Challenge
• .css-j9qmi7{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;font-weight:700;margin-bottom:1rem;margin-top:2.8rem;width:100%;-webkit-box-pack:start;-ms-flex-pack:start;-webkit-justify-content:start;justify-content:start;padding-left:5rem;}@media only screen and (max-width: 599px){.css-j9qmi7{padding-left:0;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;}}.css-j9qmi7 svg{fill:#27292D;}.css-j9qmi7 .eagfbvw0{-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;color:#27292D;}