In this episode of Future Proof, we sit down with Aman Khan, the Head of Product at Arize AI. Aman reveals why traditional product metrics fail for AI systems and shares Arize's framework for building evaluation systems that actually predict real-world AI performance, plus the emerging PM skills that separate successful AI products from failed experiments.
We discuss:
- How AI builders should think about evaluations
- The role of the AI pm and how product management is evolving
- How you should build with the expectations of foundation models changing.
(0:00) Highlights
(0:37) Intro
(1:40) What is an AI pm
(4:10) How PMs are evolving with AI
(8:10) The Aha moment in AI
(11:50) What AI builders should think about evaluations
(19:40) How AI builders best leverage their time in AI evaluations
(23:40) Prompt iteration - if your evaluations are not ideal, how do you iterate?
(27:40) What’s the minimum viable eval someone should write
(30:40) How would prioritization change based on the future of AI models
(36:40) Final thoughts
(38:00) Ethan's reflection
Ship integrations 7x faster https://www.useparagon.com/
Watch all Future Proof episodes: https://www.useparagon.com/future-proof
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
New Heights with Jason & Travis Kelce
Football’s funniest family duo — Jason Kelce of the Philadelphia Eagles and Travis Kelce of the Kansas City Chiefs — team up to provide next-level access to life in the league as it unfolds. The two brothers and Super Bowl champions drop weekly insights about the weekly slate of games and share their INSIDE perspectives on trending NFL news and sports headlines. They also endlessly rag on each other as brothers do, chat the latest in pop culture and welcome some very popular and well-known friends to chat with them. Check out new episodes every Wednesday. Follow New Heights on the Wondery App, YouTube or wherever you get your podcasts. You can listen to new episodes early and ad-free, and get exclusive content on Wondery+. Join Wondery+ in the Wondery App, Apple Podcasts or Spotify. And join our new membership for a unique fan experience by going to the New Heights YouTube channel now!
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com