A must-listen episode with Dr. Dieuwke Hupkes, a research scientist at #Meta AI Research, where we dive into AI generalization, LLM robustness, and model evaluation in large language models.
We explore how LLMs handle grammar and hierarchy, how they generalize across tasks and languages, and what consistency tells us about AI alignment.
We also talk about Dieuwke’s journey from physics to NLP, the challenges of peer review, and sustaining a career in research—plus, how pole dancing helps with focus 💪
REFERENCES:
- Dieuwke Hupkes - Google Scholar profile
- A taxonomy and review of generalization research in NLP
- What's in My Big Data
- GenBench workshop ( Youtube, website)
- Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses
- From form(s) to meaning: Probing the semantic depths of language models using multisense consistency
- MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages
- How much do language models memorize?
Chapters
00:00 Introduction to Dieuwke Hupkes and Her Journey
05:15 Navigating Challenges in Research
07:17 The Peer Review Process: Insights and Frustrations
16:23 Being a Woman in AI: Representation and Challenges
19:57 Balancing Research and Personal Life
23:37 Exploring Consistency and Generalization in Language Models
33:31 Generalization Across Modalities
35:15 Exploring Generalization Taxonomy
40:55 Challenges in Evaluating Generalization
44:12 Data Contamination and Generalization
50:43 Consistency in Language Models
57:23 The Intersection of Consistency and Alignment
01:01:15 Current Research Directions
🎧 Subscribe to stay updated on new episodes spotlighting brilliant women shaping the future of AI.
WiAIR website:
♾️ https://women-in-ai-research.github.io
Follow us at:
♾️ LinkedIn
♾️ Bluesky
♾️ X (Twitter)
#LLMs #AIgeneralization #LLMrobustness #AIalignment #ModelEvaluation #MetaAIResearch #WiAIR #WiAIRpodcast
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Special Summer Offer: Exclusively on Apple Podcasts, try our Dateline Premium subscription completely free for one month! With Dateline Premium, you get every episode ad-free plus exclusive bonus content.
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!