July 2, 2025 • 26 mins
The Code[ish] Podcast is back! Join Heroku superfan Jon Dodson and Hillary Sanders from the Heroku AI Team for the latest entry in our “Deeply Technical” series. In this episode, the pair discuss Heroku Managed Inference and Agents—what it is, what it does, and why developers should be using it.
Hillary also shares tips for new developers entering the job market, and Jon pits 10 principal developers against one hundred fresh bootcamp graduates (hypothetically, of course).
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Rachel West (00:04):
Hello and welcome to Code[ish].
An exploration of thelives of modern developers.
Join us as we dive into topicslike languages and frameworks,
data and event-driven architectures,artificial intelligence,
and individualand team productivity.
Tailored to developersand engineering leaders,
this episode is partof our Deeply Technical series.
Jon (00:27):
Hello everyone, my name is
Jon Dodson and I work forHeroku on the Builds Team.
I’ve been programmingever since my parents bought a VIC-20
and I wanted to make my ownsoftware and, if I’m being honest,
it was mostly just weird games.
I’m a huge Heroku superfan and I’mexcited to talk to you about what’s
awesome in Heroku and today to talk
about what’s awesome at Heroku, I’mjoined by Heroku’s own Hillary Sanders.
(00:49):
Hello, Hillary.
Hillary (00:50):
Hey there.
Jon (00:51):
So, Hillary, we're just going to jump right into it.
So, I wonder if you could tell us a bitabout yourself and what you do at Heroku.
Hillary (00:57):
Yeah, I am an AI engineer and researcher
with a backgroundin statistics and neural networks,
and I work on the Heroku AI Team
where we build products to help Heroku
customers use AI and integrateAI into their applications.
And we also use AI to sort ofbuild custom internal
tools and pipelines and improveexisting Heroku products.
Jon (01:21):
That sounds fun.
So, your journey as a software engineer,what's it been like up to this point?
Any tips for developersjust starting out in the game?
Hillary (01:30):
Yeah, my journey has been super fun and lucky.
I think it started outbecause I was in college
and I fell in love withstatistics, obviously.
Jon (01:42):
Common path. Hillary
I was puttering around taking waytoo many classes and took a stats one
and it maybe was not exactly
like a religious calling,but something like it.
I think stats is, like, the studyof how to optimally evaluate evidence
about the world,which I find to be very beautiful
(02:02):
and important and reallyspeaks to how my mind works
and how I felt the world shouldmake decisions to maximize good.
But I realize that statementmight sound horrifying to some people.
If it does, then they should talk to me.
[Laughs] Hillary
But that’s what gotme interested in stats.
What I realized is that if you combinethat with the power of big data
(02:25):
and compute, stats becomesnot only beautiful but very powerful,
i.e. machine learning.
And in this day and age,
and certainly in the next couple decadesperhaps, scary powerful.
And so that’s how I got into machinelearning, I just fell in love with stats
and started doing research with professorsand then went to the Bay Area.
I got into neural networks, honestly,because I was confused
(02:48):
by descriptions of them becauseno one could explain them back in the day.
I actually didn’t study that in college,it was all about hierarchical
Bayesian networks and Markovchains. Amazing things.
But I ended up
getting into neural networksbecause I was just burningly curious why
no one could explain how they worked,which made me really
(03:09):
want to figure out how they worked.
And that helped me end upgetting some jobs for quite a few years
doing neural network optimizationand eventually led me to Heroku.
So that was sort of my journeyto become an AI software engineer.
Insofar as tips, I wouldcomment on the job market today,
which feels in part inauthenticbecause I haven’t experienced a job
(03:32):
tech market this bad before.
Agree, it’s rough out there.
Hillary (03:36):
And I’m in a better position than most.
Jon (03:38):
Yeah.
Hillary (03:39):
Yeah, because like I think if you’re a junior dev entry,
it’s worse than the experienceI’m going to have.
But I’ve also made lots of mistakes,so might have some tips.
My first ignorable tip
would be to not write cover letters.
I don’t know if this is correct,but I think they take a long time.
(04:00):
Often people are just going to assumethat you’re writing them with an LLM.
You can apply to a lot of jobsin the time it takes for you to write one.
So, unless you have somethingreally meaningful and specific to say
and the question is pretty unique,I think just skip it.
And it’s not just for the time tradeoff,I think it’s to avoid depression
because I am friends with peoplewho have interviewed for a year
(04:24):
and have gotten really depressed
and it’s like hundredsand hundreds of applications.
And it feels so depressingto write all these cover letters
and kind of pour your heart outeven though you’re trying not to,
and then essentially be ghostedby all these companies.
So, you know what? Just skip it.
I think it’s too depressingand not very useful.
That’s my first tipthat is not really informed by data.
Jon (04:46):
Right. No cover letters. Check. Hillary
Another one is
if bad companies rejectyou don’t pay too much heed
because that is anincredibly noisy signal.
Absolutely.
Hillary (04:59):
I and others I know have gotten job offers for like twice the salary
in the same week that othercompanies paying half as much,
and that sounded wayless cool, rejected us.
And that’s like very standard.
And in fact, I would argue I’venoticed a positive relationship
between the job and company qualityand the simplicity of the questions
(05:21):
being asked during the interview process,for maybe reasons that I won’t get into.
So, if you get rejected from places,don’t take it too personally.
Maybe take it as very, verynoisy data on what to focus on.
My third tip is themost uncomfortable one.
Jon (05:38):
I can’t wait [chuckles].
Hillary (05:39):
Yeah, it’s just awful.
A big part of interviews,you know, once you get in
and get an interview, you have a muchhigher probability of getting a job.
It’s pretty important, you know, the codeyou write and how you do,
but how you present yourselfand how you communicate
is also very importantand maybe often underestimated,
because how you communicate and presentyourself is really important to the job.
(06:02):
So I recommend doing mock interviews
and videotaping yourselfand then gasp, watching them back over,
which may cause you significant nauseaand emotional discomfort.
But I think like per unit time investment,
it’s super, super effective at makingyou get better at interviews.
(06:23):
And I really recommend it, even thoughit’s just the worst and terrible.
Jon (06:27):
I agree with you there.
I think doing mock interviews is one ofthe most important things you could do.
Maybe practice with a friend.
You’ve got a job.
There’s oftentimesan employment office around
that you can do that with or you could,just like you said, record yourself.
There’s plenty of interview questionsonline that you can practice with.
So yeah, I think that’s great,that’s great advice.
So, Hillary, you’re on the Heroku AI Team,which is still a pretty
(06:50):
new team at Heroku, and I’mwondering if you can tell us a bit
about why it was created and whatproblems the team is trying to solve.
Hillary (06:57):
Yeah, I think we’re trying to solve lots of problems.
Essentially, Heroku was incredibly cool15 years ago, very, like, hot.
I think it’s still very cool and we’retrying to do a great job of keeping up
with the times and makingsure we do a very good job of doing that.
And so that leads us to focusingon two main areas when it comes to AI.
(07:19):
Like one, making itreally easy and seamless
to incorporate AI intoyour Heroku applications
and making sure those AI componentscan easily interact with your databases
and other components in your Heroku space.
And then also just usingAI to make our existing products better.
(07:41):
So, if you have databases onHeroku or apps on Heroku
or if you want to be vibecoding with Cursor or VS Code,
we should be helping to make sure that
AI is making that experiencereally, really good.
Cursor should have an extensionthat makes it easy to have
the LLM understand how to deploy your appas a Heroku app, that kind of thing.
(08:04):
It’s kind of enabling all of thatis the main goal and I think that leads
to lots of really interesting,fun products and features.
Jon (08:12):
So Hillary, this is a really important question.
My son’s eight,
he enjoys watching variousbrainrot videos on YouTube, you know,
as one does, and onesuch piece of nonsense are these versus
videos such as 10,000Harry Potters versus a million Predators?
Like I said, real important.
So, we’re going to apply this to AI.
Which team do you think would makea better application faster in six months?
(08:35):
Ten principal developers with no AI or 100
fresh bootcamp graduates with all currentAI tools at their disposal?
Hillary (08:44):
Woo... Okay.
That... I have complicated feelings here,because there’s a lot of trade-offs.
Jon (08:50):
[Laughs] There are, absolutely.
Hillary (08:53):
I will first raise a concern or discussion topic with the implicit hypothesis
that a hundred engineerson a single application
makes things better,unless you’re organizing it very well.
I feel like that’simplied by the question.
Jon (09:09):
Uh huh, that’s the first trap of the question. Absolutely. [Laughs]
Hillary (09:12):
That is really hard. Jon
If they all have to work together on one app and one code base…
Jon (09:19):
Right.
Hillary (09:20):
And have six months, in this specific situation,
I would bet on the 10principal developers.
However, there aremany permutations to this
that I think would lead me to beton the fresh boot camp grads, for sure.
Jon (09:35):
Which permutation?
Hillary (09:36):
Okay, so if they’re allowed to break up into groups of five or 10,
grab eight smartest peopleyou know that do different things
and the 100 people and run into a room.
If they can all go be siloedand work together, and then maybe you take
like the best thing that they’ve built of all those groups after six months.
(09:57):
That I could see beating outon the 10 principal developers for sure.
Jon (10:01):
Agree. Agree on that. Yeah.
Hillary (10:03):
Because like if you’re a fresh boot camp grad with like a good product vision
and you’re using AI, like, sure,
a lot of the time you mightend up going in bad directions.
Kind of because you just don’thave all of the lovely mistakes and
learnings that the experienced developershave made along their career paths.
You might go in the wrong direction,but one or two of the teams
will probably goin a really good direction.
(10:24):
So, I would bet on themin that circumstance.
Another permutation isif you shorten the time period,
if you have one dayto build a thing or a week
or even a month, then maybeI bet on the bootcamp grads
because you can do so much,so fast with AI and
that is just a lot harder without it.
So that’s like a thing.
(10:45):
And additionally, if you have six monthswhen you’re not integrating
into existing complicatedservices, bureaucracy etc.,
you can build a lot and that means
your project gets pretty complexand your code base gets fairly big.
Especially if you’re using too much AI and
existing LLM models strugglewith various things.
They’re like amazing,but they also struggle
(11:08):
with common sense reasoningand they struggle with understanding
a really complex code base and thenmaking changes on just a little bit.
There’s a lot of really cool toolsthat people are working on,
and that Heroku’s interestedin, to make that easier.
Like shoving like a whole repointo the context window
of a bigger and bigger modelor trying to have good ways to look up
(11:29):
the relevant parts of your codeto put into your context window
for your LLM so that it can edityour code or adjust things.
But that still doesn’t workamazingly on pretty complex code bases.
So, on a six-month time span,I think there are
decreasing marginal returns to like AI,especially when
if you made wrong turns in the beginning,that’s not going to be great.
(11:50):
But if it’s a short time periodand you have a clear vision like yeah,
junior devs with AI, they can build a lot.
Jon (11:58):
Awesome.
So, a different track here.
Ruby on Rails.
Ruby on Rails framework are the language
and platform of choice for Heroku,at least historically.
So, we support much more than that,including .NET,
which we just added,which I’m really excited about.
But historically Ruby on Railsare the bread and butter of what we do.
So, for you, like, what language would youconsider to be the language of AI
(12:19):
if there even is one?
Hillary (12:21):
I mean, Python for sure.
Jon (12:24):
Mm hmm.
Hillary (12:24):
It’s super popular. Jon
If you’re trying to like develop
on neural networks, you’retypically almost always using Python.
It’s great.
It’s very popular amongst AI enthusiasts.
It’s very easy to use, lots of supportand lots of packages.
So, yeah.
Jon (12:40):
All right. So, moving to
talking a little bit about MIA,which is our newly released product here.
So, question for you, what is MIA?
What does it do?
And why should customers use it?
Hillary (12:54):
MIA, yay.
MIA is Heroku’sManaged Inference and Agents add-on
and it does a lot of thingsand it will do even more things.
But essentially it is an add-onthat makes it really easy to integrate AI,
so specifically like largefoundational models, like an LLM,
like Claude Sonnet 3.7,into your Heroku apps.
(13:16):
So, it lets you kind of just add
the add-on, attach a modellike Claude directly to your app
without the hassle of externalaccount setups or like sending
your data out of Herokuor API key management,
data security issuesand inference calls will just work.
That is very niceand avoids a lot of hassle,
but there’s also a lot of kindof special sauce features
(13:39):
that we have been adding to makethe experience nicer and to take away
a lot of boilerplate code that you oftenhave to write in certain situations.
So, we have really coolfeatures relating to like
agentic automatic tool execution,
including some toolsthat just come built into MIA
that will just work off the bat
(14:00):
automatically without youhaving to deploy your own tool servers.
We have nice dashboards and dashboardsfor the tools you attach to your models.
And we’re alsoproviding a lot of really cool
features relating to MCP.
MCP is an open-source protocol.
It stands for Model Context Protocol,
(14:20):
and it basically helps definehow AI applications should interact
with tools and databases and resourcesin like a nice standardized way.
And we’re betting pretty heavy on MCP,and I’m very happy that we’re doing that.
So, a lot of our features arebuilt around making sure
deploying your own MCPservers to Heroku is really easy
(14:41):
and adding in yetmore special sauce around that.
Jon (14:44):
Great.
I like MIA because it’s as easyto add AI to your application
as adding a database or adding Redis,and it’s going to be first class.
When I was originally looking atthe API that the team designed,
I loved the simplicity of itand the extensibility of it.
It’s really great work.
Hillary (15:03):
Yay!
Jon (15:04):
[Laughs] Yeah, yay indeed.
So, what’s the biggest problem
MIA solves for Heroku customersand where do you think developers
should absolutely consideradding MIA to their applications?
Hillary (15:15):
Yeah. Okay. So, the biggest problem MIA solves,
MIA solves like a lot oftiny annoying problems.
Jon (15:22):
Right.
Hillary (15:23):
Similar to what Heroku does really well.
Like, we solve a lot of tinyannoying problems for you,
so it’s not a frustratingexperience to deploy apps.
So, if I think about the biggest problemMIA solves, it’s probably pretty boring.
You don’t have to go make anexternal account with like OpenAI
and give them your credit card and worry
that your data is beingtransferred to them.
(15:44):
That’s pretty simple, but it is nice.
And then you have a very easy-to-set-upLLM or image model or embedding model
that can easily connect to your otherHeroku apps or databases if you so choose.
So maybe that’s the biggestproblem solved. It’s perhaps
not the most interesting one,but it’s relevant to everyone who uses it.
(16:05):
So, it’s kind of high impact.
Jon (16:07):
Mm hmm. Yeah, absolutely.
Hillary (16:09):
What was the other question?
It was where do you thinkdevelopers should be adding it?
If they have an applicationwhere they were going to call out
to an external third party,large foundational model,
I would strongly considerusing MIA because you’re
getting often the same thing,but it’s just going to work
really nicely on Heroku because you getall of these features built in for free.
(16:33):
Like dashboards and token consumptionand all of those really nice features
relating to MCP.
And if you want like an MCP server,oh, all of a sudden it’s pretty easy
to add OAuth authenticationand do server registration
that works really well with MIA.
And that kind of stuff is just going tobe totally doable, but harder
if you use an external third party LLM.
(16:56):
So, I would say if you want touse large foundational models
like these APIs in your apps, that isawesome and consider using MIA.
Jon (17:05):
So, this is really, really important.
In the Star Wars filmAttack of the Clones, Padmé and Anakin
travel to Naboo via the H-typeNubian yacht, which is a luxury vessel
which is part of Naboo’s fleet
and known for its sleek designand strong deflector shields.
Obviously.
According to a Tumblr blog post that didthe math, the total travel time to Naboo
(17:26):
was 10 days and 10 hours,so my question for you, Hillary, is
how many movies or Disney Plus seasonsdo we need to cover this journey?
Ten days, 10 hours, really important.
Hillary (17:38):
Yeah. I think that’s a really excellent question.
Jon (17:42):
Oh, good.
Hillary (17:43):
And I have an answer, and that answer is it’s Andor season two.
Jon (17:47):
[Laughing] Oh my gosh, I’m watching that right now.
It’s so great.
Hillary (17:50):
It's a great show. Jon
Ten out of 10. Jon
I’m literally watching it for movie night tonight with friends.
So, no spoilers.
Jon (17:58):
Oh yeah, it’s fantastic.
Hillary (18:00):
Yeah. Jon
Yeah. If you’re listening now, like, what are you doing?
Go watch it. Oops! After.
Yeah, it’s really good.
Jon (18:08):
So, what kinds of technologies did you use and the team use
in the development of MIA?
Hillary (18:14):
Ooh, that’s a fun question!
So insofar as like MIA as it exists today,
we are mostly using technologiesthat Heroku provide.
So we’re dogfoodingHeroku, which I think is just
typically an excellent thing to do.
So, we’re using Heroku Dynos and
Heroku Private Spaces andHeroku Postgres and Heroku Redis.
And despite what I saidearlier about Python,
(18:36):
a lot of our code is written in Gobecause we wanted routing to be fast.
Jon (18:40):
Right.
Hillary (18:40):
And we’re doing less neural network development.
We want API routing to work really well.
The actual models are hosted byAmazon and secure AWS accounts.
We might add more models in thefuture without the might part actually.
So that’s sort of like the roughtech stack we’re using today.
I will say that a while back we wereplaying with the idea of hosting
(19:02):
our own models,
which I think it was definitelythe right decision to move away from that.
But we got to play with areally fun set of technologies.
So like Triton and TensorRTand vLLM and PyTorch
and doing a bunch of cost-performanceanalysis on different EC2 instances
like Inferentia and Trainium,classic GPU powered EC2s.
(19:22):
That was super fun.
I think that was just fun.
But those pieces of technology we’renot using in the current MIA.
We are though using a bit of Pythonlike we’re publishing various,
you know, open-source MCP reposso people can use our first-party tools.
If that floats their boat,they can deploy their own MCP servers
and have that work with MIA,and they can also kind of just clone
(19:44):
some of our getting-started reposto do that kind of thing even more easily.
Jon (19:48):
Can you walk me through what you think is the coolest feature of MIA?
Hillary (19:53):
Yeah, I don’t know if this feature
will be for general audienceby the time this podcast comes out.
If it’s not…Jon: Right.
It will be very soon.
Probably the coolest featurefor MIA, for me is what we’re
calling First-PartyAutomatic Tool Execution.
So, MIA offers like an agents endpoint,
(20:15):
which allows you to tell your model like,hey, you can call X, Y, Z tools
and normally with an inference provider,the model would select a tool
and then call back to you and you,the client would have to be like, oh,
the model wants to call this tool.
Now I have to like handle thatand call it to some server
I’ve deployed or do somethingand then give it back a response,
but if you use the agents endpointyou have the option of us
(20:38):
just doing all of thatcontrol loop nonsense for you.
And what is extra cool and I thinkis maybe the coolest feature,
is we are also offering tools that just comeworking built-in natively with MIA.
So the idea is you can createan app on Heroku and attach it to MIA
and maybe attach Claude Sonnet3.7 and say like, hey Claude,
(21:00):
you have all these tools availablethat I know like Heroku makes available
and I want you to like write Python codeand it’s going to run on a one-off dyno
in my Heroku account.
Or like, I want you to like,be able to search Google
or I want you to be able
to look at one of my read-onlydatabases and tell me about it.
Or I want you to be able to like parsethis random PDF and talk about it.
There’s various tools that we’rejust thinking are very useful
(21:24):
and we want to justoffer natively for free.
And what I think is really cool
is that that’s something that peoplecan just use in the first three minutes
of using MIA, and it just doesn’ttake a lot of boilerplate
and a lot of work to get thatworking because again, like that’s
something you can buildyourself, but it is really nice
if you don’t have to build ityourself and it just works
(21:46):
beautifully in three minutesof writing a couple lines of code.
Jon (21:50):
That's my favorite kind of development.
Hillary (21:52):
It’s, it’s pretty nice. Jon
So, oftentimes when teams get togetherto mix up the stuff that comes out
in terms of products, oftentimeswe have technical disagreements.
It happens. People disagree.
I was wondering when it came to buildingMIA, were there any disagreements
the team had or did the productjust sort of naturally evolve without any?
(22:13):
I guess we’ll have to define disagreements, right?
Because I will say honestlythe Engineering Team
and the Heroku AI Team is the most
pro-social set of engineersI’ve ever worked with.
And I’ve been doing this like 12 years,so that should be a significant statement.
So, if I define disagreements like
high uncertainty, high impact,hot topic decisions that we waffle on
(22:36):
as a team, we’ve totally had thosebecause there are a lot of decisions that
there’s not a clear answer and we reallyneed to work through it together.
Like we’re still wafflingover some things like this.
If you want the hot gossip that mayor may not be deleted from this,
one really interesting problemthat we’re working through is
what is the best API schemafor like future models that we release,
(22:59):
or future end points that release.
Like what’s the best API format?
And there are just so many conflictingincentives here that are really important.
I don’t think there is a clear,perfect answer.
There’s just a lot of different tradeoffs.
You don’t want to overwhelmcustomers with choice.
That’s exhausting.
I don’t want to evaluate 250 types of jam.
(23:20):
I just want like a delicious jamand I want to eat a snack.
Like, that’s what I want. Same with Heroku.
On the other hand, though, there’stradeoffs between different API schemas.
Like you can take your underlyingmodel providers and use their API schema,
but then you have like adifferent one per model.
You can convert everything tolike a really popular schema, like OpenAI,
(23:41):
but then you’ll kind offall short on the edge cases
and you can do a lot of workto fix that in many situations,
but at the end of the day,you’ll still have little edge cases
that might make theexperience less elegant.
When you want to use a featurethat our model supports
but OpenAI doesn’t,or vice versa, or want to be explicit
(24:03):
that we’re actually ignoring a featurebut you also don’t want to break things
when people are running thecode through OpenAI SDKs.
It’s like there’s so manybenefits with doing that too.
Like, it’s so nice to just useall of the example code online
and all the packages onlinethat really know OpenAI really well.
And then to add to that party,you also have all of the custom stuff
we’re building that needs to belike a superset on whatever API format
(24:26):
we decide on that relates to the agenticcapabilities that we’re offering
with like the automatic toolexecution and that kind of thing.
And so that’s a really fun maybequote/unquote disagreement.
But I think more just like superinteresting problem that we will continue
to think about solving really wellas we release more and more things.
(24:48):
And honestly, if you’re listeningand have a strong opinion,
email me because I’mcurious to hear what you think
because it’s a super tough problemand people have a lot of good opinions.
Jon (24:57):
Well, inbox flooded, I’m sure.
Hillary (24:59):
I hope.
Jon (25:00):
[Laughing] That would be great.
So finally, I wonder,to wrap things up here.
Thank you, Hillary,for talking to me today.
Hillary (25:07):
Oh, thank you.
Jon (25:08):
Yeah, you’re welcome.
What’s coming next with MIA?What can we look forward to?
I know it just came out.
I know.
And everyone’s looking to seewhat’s next, but I’m just curious.
What’s next for MIA?
So much. I mean, mostly
the things that we wanted to squeezeinto our initial release that
didn’t get squeezed in.
The biggest category of stuffis a lot of really cool features
(25:30):
relating to MCP, the ModelContext Protocol I talked about.
So, we’re offering different typesof models, like an embedding model
for RAG—Retrieval Augmented Generationimage model and a couple chat models,
but you can really superchargechat models with tools.
And so we’re making that bigbet on MCP and really seeing
(25:51):
a lot of super cool featuresrelating to that pretty soon.
So, I’m excited about that.
Me too.
And thank you again, Hillary,for talking to me today.
Hillary (26:01):
Thank you very much.
Rachel (26:04):
Thanks for joining us for this episode of the Code[ish] Podcast.
Code[ish] is produced by Heroku.
The easiest way to deploy, manage,and scale your applications in the cloud.
If you’d like to learn more aboutCode[ish] or any of Heroku’s podcasts,
please visit Heroku.com/podcasts.
Hello and welcome to Code[ish].
An exploration of thelives of modern developers.
Join us as we dive into topicslike languages and frameworks,
data and event-driven architectures,artificial intelligence,
and individualand team productivity.
Tailored to developersand engineering leaders,
this episode is partof our Deeply Technical series.
Jon (00:27):
Hello everyone, my name is
Jon Dodson and I work forHeroku on the Builds Team.
I’ve been programmingever since my parents bought a VIC-20
and I wanted to make my ownsoftware and, if I’m being honest,
it was mostly just weird games.
I’m a huge Heroku superfan and I’mexcited to talk to you about what’s
awesome in Heroku and today to talk
about what’s awesome at Heroku, I’mjoined by Heroku’s own Hillary Sanders.
(00:49):
Hello, Hillary.
Hillary (00:50):
Hey there.
Jon (00:51):
So, Hillary, we're just going to jump right into it.
So, I wonder if you could tell us a bitabout yourself and what you do at Heroku.
Hillary (00:57):
Yeah, I am an AI engineer and researcher
with a backgroundin statistics and neural networks,
and I work on the Heroku AI Team
where we build products to help Heroku
customers use AI and integrateAI into their applications.
And we also use AI to sort ofbuild custom internal
tools and pipelines and improveexisting Heroku products.
Jon (01:21):
That sounds fun.
So, your journey as a software engineer,what's it been like up to this point?
Any tips for developersjust starting out in the game?
Hillary (01:30):
Yeah, my journey has been super fun and lucky.
I think it started outbecause I was in college
and I fell in love withstatistics, obviously.
Jon (01:42):
Common path. Hillary
I was puttering around taking waytoo many classes and took a stats one
and it maybe was not exactly
like a religious calling,but something like it.
I think stats is, like, the studyof how to optimally evaluate evidence
about the world,which I find to be very beautiful
(02:02):
and important and reallyspeaks to how my mind works
and how I felt the world shouldmake decisions to maximize good.
But I realize that statementmight sound horrifying to some people.
If it does, then they should talk to me.
[Laughs] Hillary
But that’s what gotme interested in stats.
What I realized is that if you combinethat with the power of big data
(02:25):
and compute, stats becomesnot only beautiful but very powerful,
i.e. machine learning.
And in this day and age,
and certainly in the next couple decadesperhaps, scary powerful.
And so that’s how I got into machinelearning, I just fell in love with stats
and started doing research with professorsand then went to the Bay Area.
I got into neural networks, honestly,because I was confused
(02:48):
by descriptions of them becauseno one could explain them back in the day.
I actually didn’t study that in college,it was all about hierarchical
Bayesian networks and Markovchains. Amazing things.
But I ended up
getting into neural networksbecause I was just burningly curious why
no one could explain how they worked,which made me really
(03:09):
want to figure out how they worked.
And that helped me end upgetting some jobs for quite a few years
doing neural network optimizationand eventually led me to Heroku.
So that was sort of my journeyto become an AI software engineer.
Insofar as tips, I wouldcomment on the job market today,
which feels in part inauthenticbecause I haven’t experienced a job
(03:32):
tech market this bad before.
Agree, it’s rough out there.
Hillary (03:36):
And I’m in a better position than most.
Jon (03:38):
Yeah.
Hillary (03:39):
Yeah, because like I think if you’re a junior dev entry,
it’s worse than the experienceI’m going to have.
But I’ve also made lots of mistakes,so might have some tips.
My first ignorable tip
would be to not write cover letters.
I don’t know if this is correct,but I think they take a long time.
(04:00):
Often people are just going to assumethat you’re writing them with an LLM.
You can apply to a lot of jobsin the time it takes for you to write one.
So, unless you have somethingreally meaningful and specific to say
and the question is pretty unique,I think just skip it.
And it’s not just for the time tradeoff,I think it’s to avoid depression
because I am friends with peoplewho have interviewed for a year
(04:24):
and have gotten really depressed
and it’s like hundredsand hundreds of applications.
And it feels so depressingto write all these cover letters
and kind of pour your heart outeven though you’re trying not to,
and then essentially be ghostedby all these companies.
So, you know what? Just skip it.
I think it’s too depressingand not very useful.
That’s my first tipthat is not really informed by data.
Jon (04:46):
Right. No cover letters. Check. Hillary
Another one is
if bad companies rejectyou don’t pay too much heed
because that is anincredibly noisy signal.
Absolutely.
Hillary (04:59):
I and others I know have gotten job offers for like twice the salary
in the same week that othercompanies paying half as much,
and that sounded wayless cool, rejected us.
And that’s like very standard.
And in fact, I would argue I’venoticed a positive relationship
between the job and company qualityand the simplicity of the questions
(05:21):
being asked during the interview process,for maybe reasons that I won’t get into.
So, if you get rejected from places,don’t take it too personally.
Maybe take it as very, verynoisy data on what to focus on.
My third tip is themost uncomfortable one.
Jon (05:38):
I can’t wait [chuckles].
Hillary (05:39):
Yeah, it’s just awful.
A big part of interviews,you know, once you get in
and get an interview, you have a muchhigher probability of getting a job.
It’s pretty important, you know, the codeyou write and how you do,
but how you present yourselfand how you communicate
is also very importantand maybe often underestimated,
because how you communicate and presentyourself is really important to the job.
(06:02):
So I recommend doing mock interviews
and videotaping yourselfand then gasp, watching them back over,
which may cause you significant nauseaand emotional discomfort.
But I think like per unit time investment,
it’s super, super effective at makingyou get better at interviews.
(06:23):
And I really recommend it, even thoughit’s just the worst and terrible.
Jon (06:27):
I agree with you there.
I think doing mock interviews is one ofthe most important things you could do.
Maybe practice with a friend.
You’ve got a job.
There’s oftentimesan employment office around
that you can do that with or you could,just like you said, record yourself.
There’s plenty of interview questionsonline that you can practice with.
So yeah, I think that’s great,that’s great advice.
So, Hillary, you’re on the Heroku AI Team,which is still a pretty
(06:50):
new team at Heroku, and I’mwondering if you can tell us a bit
about why it was created and whatproblems the team is trying to solve.
Hillary (06:57):
Yeah, I think we’re trying to solve lots of problems.
Essentially, Heroku was incredibly cool15 years ago, very, like, hot.
I think it’s still very cool and we’retrying to do a great job of keeping up
with the times and makingsure we do a very good job of doing that.
And so that leads us to focusingon two main areas when it comes to AI.
(07:19):
Like one, making itreally easy and seamless
to incorporate AI intoyour Heroku applications
and making sure those AI componentscan easily interact with your databases
and other components in your Heroku space.
And then also just usingAI to make our existing products better.
(07:41):
So, if you have databases onHeroku or apps on Heroku
or if you want to be vibecoding with Cursor or VS Code,
we should be helping to make sure that
AI is making that experiencereally, really good.
Cursor should have an extensionthat makes it easy to have
the LLM understand how to deploy your appas a Heroku app, that kind of thing.
(08:04):
It’s kind of enabling all of thatis the main goal and I think that leads
to lots of really interesting,fun products and features.
Jon (08:12):
So Hillary, this is a really important question.
My son’s eight,
he enjoys watching variousbrainrot videos on YouTube, you know,
as one does, and onesuch piece of nonsense are these versus
videos such as 10,000Harry Potters versus a million Predators?
Like I said, real important.
So, we’re going to apply this to AI.
Which team do you think would makea better application faster in six months?
(08:35):
Ten principal developers with no AI or 100
fresh bootcamp graduates with all currentAI tools at their disposal?
Hillary (08:44):
Woo... Okay.
That... I have complicated feelings here,because there’s a lot of trade-offs.
Jon (08:50):
[Laughs] There are, absolutely.
Hillary (08:53):
I will first raise a concern or discussion topic with the implicit hypothesis
that a hundred engineerson a single application
makes things better,unless you’re organizing it very well.
I feel like that’simplied by the question.
Jon (09:09):
Uh huh, that’s the first trap of the question. Absolutely. [Laughs]
Hillary (09:12):
That is really hard. Jon
If they all have to work together on one app and one code base…
Jon (09:19):
Right.
Hillary (09:20):
And have six months, in this specific situation,
I would bet on the 10principal developers.
However, there aremany permutations to this
that I think would lead me to beton the fresh boot camp grads, for sure.
Jon (09:35):
Which permutation?
Hillary (09:36):
Okay, so if they’re allowed to break up into groups of five or 10,
grab eight smartest peopleyou know that do different things
and the 100 people and run into a room.
If they can all go be siloedand work together, and then maybe you take
like the best thing that they’ve built of all those groups after six months.
(09:57):
That I could see beating outon the 10 principal developers for sure.
Jon (10:01):
Agree. Agree on that. Yeah.
Hillary (10:03):
Because like if you’re a fresh boot camp grad with like a good product vision
and you’re using AI, like, sure,
a lot of the time you mightend up going in bad directions.
Kind of because you just don’thave all of the lovely mistakes and
learnings that the experienced developershave made along their career paths.
You might go in the wrong direction,but one or two of the teams
will probably goin a really good direction.
(10:24):
So, I would bet on themin that circumstance.
Another permutation isif you shorten the time period,
if you have one dayto build a thing or a week
or even a month, then maybeI bet on the bootcamp grads
because you can do so much,so fast with AI and
that is just a lot harder without it.
So that’s like a thing.
(10:45):
And additionally, if you have six monthswhen you’re not integrating
into existing complicatedservices, bureaucracy etc.,
you can build a lot and that means
your project gets pretty complexand your code base gets fairly big.
Especially if you’re using too much AI and
existing LLM models strugglewith various things.
They’re like amazing,but they also struggle
(11:08):
with common sense reasoningand they struggle with understanding
a really complex code base and thenmaking changes on just a little bit.
There’s a lot of really cool toolsthat people are working on,
and that Heroku’s interestedin, to make that easier.
Like shoving like a whole repointo the context window
of a bigger and bigger modelor trying to have good ways to look up
(11:29):
the relevant parts of your codeto put into your context window
for your LLM so that it can edityour code or adjust things.
But that still doesn’t workamazingly on pretty complex code bases.
So, on a six-month time span,I think there are
decreasing marginal returns to like AI,especially when
if you made wrong turns in the beginning,that’s not going to be great.
(11:50):
But if it’s a short time periodand you have a clear vision like yeah,
junior devs with AI, they can build a lot.
Jon (11:58):
Awesome.
So, a different track here.
Ruby on Rails.
Ruby on Rails framework are the language
and platform of choice for Heroku,at least historically.
So, we support much more than that,including .NET,
which we just added,which I’m really excited about.
But historically Ruby on Railsare the bread and butter of what we do.
So, for you, like, what language would youconsider to be the language of AI
(12:19):
if there even is one?
Hillary (12:21):
I mean, Python for sure.
Jon (12:24):
Mm hmm.
Hillary (12:24):
It’s super popular. Jon
If you’re trying to like develop
on neural networks, you’retypically almost always using Python.
It’s great.
It’s very popular amongst AI enthusiasts.
It’s very easy to use, lots of supportand lots of packages.
So, yeah.
Jon (12:40):
All right. So, moving to
talking a little bit about MIA,which is our newly released product here.
So, question for you, what is MIA?
What does it do?
And why should customers use it?
Hillary (12:54):
MIA, yay.
MIA is Heroku’sManaged Inference and Agents add-on
and it does a lot of thingsand it will do even more things.
But essentially it is an add-onthat makes it really easy to integrate AI,
so specifically like largefoundational models, like an LLM,
like Claude Sonnet 3.7,into your Heroku apps.
(13:16):
So, it lets you kind of just add
the add-on, attach a modellike Claude directly to your app
without the hassle of externalaccount setups or like sending
your data out of Herokuor API key management,
data security issuesand inference calls will just work.
That is very niceand avoids a lot of hassle,
but there’s also a lot of kindof special sauce features
(13:39):
that we have been adding to makethe experience nicer and to take away
a lot of boilerplate code that you oftenhave to write in certain situations.
So, we have really coolfeatures relating to like
agentic automatic tool execution,
including some toolsthat just come built into MIA
that will just work off the bat
(14:00):
automatically without youhaving to deploy your own tool servers.
We have nice dashboards and dashboardsfor the tools you attach to your models.
And we’re alsoproviding a lot of really cool
features relating to MCP.
MCP is an open-source protocol.
It stands for Model Context Protocol,
(14:20):
and it basically helps definehow AI applications should interact
with tools and databases and resourcesin like a nice standardized way.
And we’re betting pretty heavy on MCP,and I’m very happy that we’re doing that.
So, a lot of our features arebuilt around making sure
deploying your own MCPservers to Heroku is really easy
(14:41):
and adding in yetmore special sauce around that.
Jon (14:44):
Great.
I like MIA because it’s as easyto add AI to your application
as adding a database or adding Redis,and it’s going to be first class.
When I was originally looking atthe API that the team designed,
I loved the simplicity of itand the extensibility of it.
It’s really great work.
Hillary (15:03):
Yay!
Jon (15:04):
[Laughs] Yeah, yay indeed.
So, what’s the biggest problem
MIA solves for Heroku customersand where do you think developers
should absolutely consideradding MIA to their applications?
Hillary (15:15):
Yeah. Okay. So, the biggest problem MIA solves,
MIA solves like a lot oftiny annoying problems.
Jon (15:22):
Right.
Hillary (15:23):
Similar to what Heroku does really well.
Like, we solve a lot of tinyannoying problems for you,
so it’s not a frustratingexperience to deploy apps.
So, if I think about the biggest problemMIA solves, it’s probably pretty boring.
You don’t have to go make anexternal account with like OpenAI
and give them your credit card and worry
that your data is beingtransferred to them.
(15:44):
That’s pretty simple, but it is nice.
And then you have a very easy-to-set-upLLM or image model or embedding model
that can easily connect to your otherHeroku apps or databases if you so choose.
So maybe that’s the biggestproblem solved. It’s perhaps
not the most interesting one,but it’s relevant to everyone who uses it.
(16:05):
So, it’s kind of high impact.
Jon (16:07):
Mm hmm. Yeah, absolutely.
Hillary (16:09):
What was the other question?
It was where do you thinkdevelopers should be adding it?
If they have an applicationwhere they were going to call out
to an external third party,large foundational model,
I would strongly considerusing MIA because you’re
getting often the same thing,but it’s just going to work
really nicely on Heroku because you getall of these features built in for free.
(16:33):
Like dashboards and token consumptionand all of those really nice features
relating to MCP.
And if you want like an MCP server,oh, all of a sudden it’s pretty easy
to add OAuth authenticationand do server registration
that works really well with MIA.
And that kind of stuff is just going tobe totally doable, but harder
if you use an external third party LLM.
(16:56):
So, I would say if you want touse large foundational models
like these APIs in your apps, that isawesome and consider using MIA.
Jon (17:05):
So, this is really, really important.
In the Star Wars filmAttack of the Clones, Padmé and Anakin
travel to Naboo via the H-typeNubian yacht, which is a luxury vessel
which is part of Naboo’s fleet
and known for its sleek designand strong deflector shields.
Obviously.
According to a Tumblr blog post that didthe math, the total travel time to Naboo
(17:26):
was 10 days and 10 hours,so my question for you, Hillary, is
how many movies or Disney Plus seasonsdo we need to cover this journey?
Ten days, 10 hours, really important.
Hillary (17:38):
Yeah. I think that’s a really excellent question.
Jon (17:42):
Oh, good.
Hillary (17:43):
And I have an answer, and that answer is it’s Andor season two.
Jon (17:47):
[Laughing] Oh my gosh, I’m watching that right now.
It’s so great.
Hillary (17:50):
It's a great show. Jon
Ten out of 10. Jon
I’m literally watching it for movie night tonight with friends.
So, no spoilers.
Jon (17:58):
Oh yeah, it’s fantastic.
Hillary (18:00):
Yeah. Jon
Yeah. If you’re listening now, like, what are you doing?
Go watch it. Oops! After.
Yeah, it’s really good.
Jon (18:08):
So, what kinds of technologies did you use and the team use
in the development of MIA?
Hillary (18:14):
Ooh, that’s a fun question!
So insofar as like MIA as it exists today,
we are mostly using technologiesthat Heroku provide.
So we’re dogfoodingHeroku, which I think is just
typically an excellent thing to do.
So, we’re using Heroku Dynos and
Heroku Private Spaces andHeroku Postgres and Heroku Redis.
And despite what I saidearlier about Python,
(18:36):
a lot of our code is written in Gobecause we wanted routing to be fast.
Jon (18:40):
Right.
Hillary (18:40):
And we’re doing less neural network development.
We want API routing to work really well.
The actual models are hosted byAmazon and secure AWS accounts.
We might add more models in thefuture without the might part actually.
So that’s sort of like the roughtech stack we’re using today.
I will say that a while back we wereplaying with the idea of hosting
(19:02):
our own models,
which I think it was definitelythe right decision to move away from that.
But we got to play with areally fun set of technologies.
So like Triton and TensorRTand vLLM and PyTorch
and doing a bunch of cost-performanceanalysis on different EC2 instances
like Inferentia and Trainium,classic GPU powered EC2s.
(19:22):
That was super fun.
I think that was just fun.
But those pieces of technology we’renot using in the current MIA.
We are though using a bit of Pythonlike we’re publishing various,
you know, open-source MCP reposso people can use our first-party tools.
If that floats their boat,they can deploy their own MCP servers
and have that work with MIA,and they can also kind of just clone
(19:44):
some of our getting-started reposto do that kind of thing even more easily.
Jon (19:48):
Can you walk me through what you think is the coolest feature of MIA?
Hillary (19:53):
Yeah, I don’t know if this feature
will be for general audienceby the time this podcast comes out.
If it’s not…Jon: Right.
It will be very soon.
Probably the coolest featurefor MIA, for me is what we’re
calling First-PartyAutomatic Tool Execution.
So, MIA offers like an agents endpoint,
(20:15):
which allows you to tell your model like,hey, you can call X, Y, Z tools
and normally with an inference provider,the model would select a tool
and then call back to you and you,the client would have to be like, oh,
the model wants to call this tool.
Now I have to like handle thatand call it to some server
I’ve deployed or do somethingand then give it back a response,
but if you use the agents endpointyou have the option of us
(20:38):
just doing all of thatcontrol loop nonsense for you.
And what is extra cool and I thinkis maybe the coolest feature,
is we are also offering tools that just comeworking built-in natively with MIA.
So the idea is you can createan app on Heroku and attach it to MIA
and maybe attach Claude Sonnet3.7 and say like, hey Claude,
(21:00):
you have all these tools availablethat I know like Heroku makes available
and I want you to like write Python codeand it’s going to run on a one-off dyno
in my Heroku account.
Or like, I want you to like,be able to search Google
or I want you to be able
to look at one of my read-onlydatabases and tell me about it.
Or I want you to be able to like parsethis random PDF and talk about it.
There’s various tools that we’rejust thinking are very useful
(21:24):
and we want to justoffer natively for free.
And what I think is really cool
is that that’s something that peoplecan just use in the first three minutes
of using MIA, and it just doesn’ttake a lot of boilerplate
and a lot of work to get thatworking because again, like that’s
something you can buildyourself, but it is really nice
if you don’t have to build ityourself and it just works
(21:46):
beautifully in three minutesof writing a couple lines of code.
Jon (21:50):
That's my favorite kind of development.
Hillary (21:52):
It’s, it’s pretty nice. Jon
So, oftentimes when teams get togetherto mix up the stuff that comes out
in terms of products, oftentimeswe have technical disagreements.
It happens. People disagree.
I was wondering when it came to buildingMIA, were there any disagreements
the team had or did the productjust sort of naturally evolve without any?
(22:13):
I guess we’ll have to define disagreements, right?
Because I will say honestlythe Engineering Team
and the Heroku AI Team is the most
pro-social set of engineersI’ve ever worked with.
And I’ve been doing this like 12 years,so that should be a significant statement.
So, if I define disagreements like
high uncertainty, high impact,hot topic decisions that we waffle on
(22:36):
as a team, we’ve totally had thosebecause there are a lot of decisions that
there’s not a clear answer and we reallyneed to work through it together.
Like we’re still wafflingover some things like this.
If you want the hot gossip that mayor may not be deleted from this,
one really interesting problemthat we’re working through is
what is the best API schemafor like future models that we release,
(22:59):
or future end points that release.
Like what’s the best API format?
And there are just so many conflictingincentives here that are really important.
I don’t think there is a clear,perfect answer.
There’s just a lot of different tradeoffs.
You don’t want to overwhelmcustomers with choice.
That’s exhausting.
I don’t want to evaluate 250 types of jam.
(23:20):
I just want like a delicious jamand I want to eat a snack.
Like, that’s what I want. Same with Heroku.
On the other hand, though, there’stradeoffs between different API schemas.
Like you can take your underlyingmodel providers and use their API schema,
but then you have like adifferent one per model.
You can convert everything tolike a really popular schema, like OpenAI,
(23:41):
but then you’ll kind offall short on the edge cases
and you can do a lot of workto fix that in many situations,
but at the end of the day,you’ll still have little edge cases
that might make theexperience less elegant.
When you want to use a featurethat our model supports
but OpenAI doesn’t,or vice versa, or want to be explicit
(24:03):
that we’re actually ignoring a featurebut you also don’t want to break things
when people are running thecode through OpenAI SDKs.
It’s like there’s so manybenefits with doing that too.
Like, it’s so nice to just useall of the example code online
and all the packages onlinethat really know OpenAI really well.
And then to add to that party,you also have all of the custom stuff
we’re building that needs to belike a superset on whatever API format
(24:26):
we decide on that relates to the agenticcapabilities that we’re offering
with like the automatic toolexecution and that kind of thing.
And so that’s a really fun maybequote/unquote disagreement.
But I think more just like superinteresting problem that we will continue
to think about solving really wellas we release more and more things.
(24:48):
And honestly, if you’re listeningand have a strong opinion,
email me because I’mcurious to hear what you think
because it’s a super tough problemand people have a lot of good opinions.
Jon (24:57):
Well, inbox flooded, I’m sure.
Hillary (24:59):
I hope.
Jon (25:00):
[Laughing] That would be great.
So finally, I wonder,to wrap things up here.
Thank you, Hillary,for talking to me today.
Hillary (25:07):
Oh, thank you.
Jon (25:08):
Yeah, you’re welcome.
What’s coming next with MIA?What can we look forward to?
I know it just came out.
I know.
And everyone’s looking to seewhat’s next, but I’m just curious.
What’s next for MIA?
So much. I mean, mostly
the things that we wanted to squeezeinto our initial release that
didn’t get squeezed in.
The biggest category of stuffis a lot of really cool features
(25:30):
relating to MCP, the ModelContext Protocol I talked about.
So, we’re offering different typesof models, like an embedding model
for RAG—Retrieval Augmented Generationimage model and a couple chat models,
but you can really superchargechat models with tools.
And so we’re making that bigbet on MCP and really seeing
(25:51):
a lot of super cool featuresrelating to that pretty soon.
So, I’m excited about that.
Me too.
And thank you again, Hillary,for talking to me today.
Hillary (26:01):
Thank you very much.
Rachel (26:04):
Thanks for joining us for this episode of the Code[ish] Podcast.
Code[ish] is produced by Heroku.
The easiest way to deploy, manage,and scale your applications in the cloud.
If you’d like to learn more aboutCode[ish] or any of Heroku’s podcasts,
please visit Heroku.com/podcasts.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
Las Culturistas with Matt Rogers and Bowen Yang
Ding dong! Join your culture consultants, Matt Rogers and Bowen Yang, on an unforgettable journey into the beating heart of CULTURE. Alongside sizzling special guests, they GET INTO the hottest pop-culture moments of the day and the formative cultural experiences that turned them into Culturistas. Produced by the Big Money Players Network and iHeartRadio.