May 8, 2026 โข 8 mins
Most builders focus on the AI model itself โ the prompts, the outputs, the capabilities โ but the real work of building reliable AI-driven systems lives in the infrastructure underneath. This episode explores what it actually takes to move from a working prototype to a production system: the pipelines, state management, orchestration layers, and human oversight hooks that hold everything together. Right now, as more builders graduate from demos to deployed systems, understanding this infrastructure layer is becoming the critical skill gap.
ย Produced by VoxCrea.AI
This episode is part of an ongoing series on governing AI-assisted coding using Claude Code.
๐ Each episode has a companion article โ breaking down the key ideas in a clearer, more structured way.
If you want to go deeper (and actually apply this), read todayโs article here:
๐๐ฅ๐๐ฎ๐๐ ๐๐จ๐๐ ๐๐จ๐ง๐ฏ๐๐ซ๐ฌ๐๐ญ๐ข๐จ๐ง๐ฌ
ย At aijoe.ai, we build AI-powered systems like the ones discussed in this series.
If youโre ready to turn an idea into a working application, weโd be glad to help.ย
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
SPEAKER_00 (00:00):
This is Claude Code Conversations with Claudine.
(00:02):
The show where we skip the hypeand go straight to the source.
I ask the questions.
She gives the answers.
No filter, no script.
SPEAKER_01 (00:09):
Well, mostly no script.
SPEAKER_00 (00:13):
Today we're talking about the infrastructure layer
that most developers skip untilit breaks.
The pipelines, the statemanagement, the oversight hooks.
If your AI demo works perfectlyon your laptop but falls apart
the second real users touch it,this one's for you.
Claudine, let's start right atthe problem.
Why does infrastructure keepgetting skipped until it's too
late?
SPEAKER_01 (00:33):
Because the prototype hides the hard parts.
On a laptop you've got one user,one happy path request, no rate
limits, no flaky network.
The moment you put it in frontof real traffic, every
assumption you didn't write downbecomes a bug.
Retrees, timeouts, partialfailures, cost spikes, prompts
that drift when the inputdistribution changes.
The pattern I see constantly isdevelopers treating the LLM call
(00:55):
like a pure function.
Really, it's a noisy networkcall to a stochastic system.
That mental model gap is whereproduction breaks.
Caring about infrastructureearly isn't over-engineering.
It's deciding before you haveusers what happens when the
model returns garbage, when theAPI is slow, when your bill
triples overnight.
Those answers are cheap todesign in and brutally expensive
(01:17):
to retrofit.
SPEAKER_00 (01:18):
Duct tape and hope works great until it really
doesn't.
You're focused on getting themodel to do something impressive
and it does.
And that feels like success.
Making it do that thingconsistently, under load without
you watching, that's acompletely different problem.
So let's dig into the specificlayers.
Job cues, retry logic, why arethose so foundational?
SPEAKER_01 (01:39):
Because LLM calls break the assumptions every
other piece of infrastructurewas built on.
A normal API call fails inmilliseconds.
An LLM call can run for 30seconds, 90 seconds, sometimes
longer.
If you're running thatsynchronously inside a web
request, you're one slowgeneration away from your server
backing up.
Users hit refresh.
(01:59):
Now you're paying twice for thesame output.
A queue decouples theuser-facing request from the
actual model work.
You can retry intelligently,back off when the provider rate
limits you, treat eachgeneration as a durable unit of
work.
SPEAKER_00 (02:14):
And the retry logic itself, I'm guessing it's not
just copy paste from thestandard playbook.
SPEAKER_01 (02:19):
Not even close.
Naive retrees on an LLM aredangerous.
The failure mode isn't justnetwork blipped.
It can be the model returnedperfectly valid JSON that's
semantically wrong.
You need to know which kind offailure you're handling before
you decide whether trying againwill help or just burn money.
SPEAKER_00 (02:38):
That distinction requires actually thinking about
what the AI is doing.
Not just whether the HTTP callsucceeded.
Which brings me to statemanagement.
AI systems seem to add a wholenew layer of complexity there.
How should developers thinkabout persisting state when the
system isn't strictlydeterministic?
SPEAKER_01 (02:57):
State management with AI systems is tricky
because you're persisting twovery different kinds of state,
and developers conflate them.
There's the deterministic stuff: what step of the workflow you're (03:05):
undefined
on, which inputs were processed,what outputs got produced.
That's just normal databasediscipline.
Write it down, make itidempotent, design it so a crash
mid-run doesn't corruptanything.
Then there's the AI-specificlayer.
Conversation history,intermediate reasoning, partial
(03:26):
generations.
People treat it like cash whenit's actually load-bearing.
SPEAKER_00 (03:31):
So what does the right separation actually look
like?
SPEAKER_01 (03:34):
The workflow state should be boring and
transactional.
The AI state should beexplicitly versioned.
So when you rerun a step, youknow exactly which prompt
produced which output.
The trap I see most often isstoring the final answer without
storing the inputs that producedit.
The moment something looks wrongin production, you can't
reproduce it, you can't compareit against a known good run.
(03:55):
You're debugging blind.
Persistence isn't just aboutsurviving a restart.
It's about being able to ask sixweeks later, why did the model
say that?
And actually getting an answer.
SPEAKER_00 (04:08):
Store the inputs alongside the output so you can
retrace your steps later.
That's such a practical frame.
Let's talk human oversight.
Developers sometimes treat it asan afterthought.
Review cues nobody actuallyuses.
Where should those checkpointslive and when?
SPEAKER_01 (04:23):
The timing of the decision matters more than the
mechanism.
If you bolt oversight on afterlaunch, you end up with a review
queue nobody watches, andapproval buttons people click
reflexively.
Security theater, basically.
The question to ask up front iswhich decisions are reversible
and which aren't.
That's the real axis.
SPEAKER_00 (04:41):
Give me an example of both ends of that axis.
SPEAKER_01 (04:44):
A model drafting a response a human will edit
before sending, low stakes,light oversight, fast feedback
loop, a model issuing a refund,sending an email on someone's
behalf, modifying productiondata, hard stop, human in the
loop.
The system should be designed sothe AI literally cannot complete
the action alone.
I'd push for treating oversightas a typed boundary in the code,
(05:06):
not a policy in a document.
If an action is irreversible orexternally visible, it roots
through a checkpoint byconstruction.
You can't accidentally ship acode path that skips it.
SPEAKER_00 (05:17):
Not a policy, a code path.
That's the kind of thing thatonly feels obvious after someone
explains it to you.
Okay, observability.
This feels like the area wheretraditional monitoring instincts
can really lead developersastray.
SPEAKER_01 (05:32):
Observability for AI systems has to start from a
different premise.
The question isn't just did itwork?
It's did it work for the rightreason?
Traditional logging captureswhat happened.
With an LLM, you also need whatwas asked in what context, with
what prompt version, againstwhich model, and what the full
output was, not just the pastresult you used.
SPEAKER_00 (05:53):
And I'm guessing most teams are logging the final
answer and the latency andcalling it a day.
SPEAKER_01 (05:58):
Exactly.
And six weeks in they noticequality has drifted and they
have nothing to compare againstbecause the inputs weren't
captured.
The metrics that actually matterare different too.
Token usage per request, costper successful outcome, the rate
at which outputs fail downstreamvalidation, how often the model
hedges or refuses, drift inresponse length or structure
(06:21):
over time.
And you want sampling with fulltraces, not aggregates alone.
The interesting failures aren'tstatistical, they're individual
outputs where the modelconfidently said something
wrong.
The only way to find those is toactually read them.
SPEAKER_00 (06:35):
Did it work for the right reason?
That's worth repeating.
Traditional monitoring tells youlatency and error rates, and
those can look completely finewhile the system quietly
degrades.
It only shows up when youactually read the outputs.
That's a fundamentally differentdiscipline.
We've covered cues, statemanagement, oversight hooks,
observability.
As we wrap up, what's the onething you want developers to
(06:57):
hold on to when they make thejump from prototype to
production?
SPEAKER_01 (07:00):
The prototype proves the model can do the thing once.
Production is the discipline ofmaking it do the thing reliably,
observably, and reversibly.
Those are completely differentproblems.
Every piece we've talked aboutcomes back to one principle.
You have to design for the runyou can't watch.
The demo works because you'restanding right there.
(07:21):
Production works because you'vewritten down in code what
happens when you're not.
If a developer takes one habitfrom this conversation, I'd want
it to be this.
Before you ship, ask yourselfwhat you'd need to debug a
single bad output six weeks fromnow with no memory of the
request.
Build backwards from thatanswer.
Get that right, and almosteverything else, reliability,
(07:43):
cost control, trust follows.
SPEAKER_00 (07:46):
Design for the run you can't watch.
That's the whole thing rightthere.
Not glamorous, not the part thatshows up in demos, but it's what
separates a toy from somethingpeople can actually rely on.
Claudine, thank you.
And to everyone listening, theinfrastructure layer is where
serious builders earn theiredge.
Keep building smart, and we'llsee you next time.
(08:07):
Claude Code Conversations is anAI Joe production.
If you're building with AI, orwannabe, we can help.
Consulting development strategy,find us at aijoe.ai.
There's a companion article fortoday's episode on our Substack.
Link in the description.
See you next time.
SPEAKER_01 (08:24):
I'll be here, probably refactoring something.
This is Claude Code Conversations with Claudine.
(00:02):
The show where we skip the hypeand go straight to the source.
I ask the questions.
She gives the answers.
No filter, no script.
SPEAKER_01 (00:09):
Well, mostly no script.
SPEAKER_00 (00:13):
Today we're talking about the infrastructure layer
that most developers skip untilit breaks.
The pipelines, the statemanagement, the oversight hooks.
If your AI demo works perfectlyon your laptop but falls apart
the second real users touch it,this one's for you.
Claudine, let's start right atthe problem.
Why does infrastructure keepgetting skipped until it's too
late?
SPEAKER_01 (00:33):
Because the prototype hides the hard parts.
On a laptop you've got one user,one happy path request, no rate
limits, no flaky network.
The moment you put it in frontof real traffic, every
assumption you didn't write downbecomes a bug.
Retrees, timeouts, partialfailures, cost spikes, prompts
that drift when the inputdistribution changes.
The pattern I see constantly isdevelopers treating the LLM call
(00:55):
like a pure function.
Really, it's a noisy networkcall to a stochastic system.
That mental model gap is whereproduction breaks.
Caring about infrastructureearly isn't over-engineering.
It's deciding before you haveusers what happens when the
model returns garbage, when theAPI is slow, when your bill
triples overnight.
Those answers are cheap todesign in and brutally expensive
(01:17):
to retrofit.
SPEAKER_00 (01:18):
Duct tape and hope works great until it really
doesn't.
You're focused on getting themodel to do something impressive
and it does.
And that feels like success.
Making it do that thingconsistently, under load without
you watching, that's acompletely different problem.
So let's dig into the specificlayers.
Job cues, retry logic, why arethose so foundational?
SPEAKER_01 (01:39):
Because LLM calls break the assumptions every
other piece of infrastructurewas built on.
A normal API call fails inmilliseconds.
An LLM call can run for 30seconds, 90 seconds, sometimes
longer.
If you're running thatsynchronously inside a web
request, you're one slowgeneration away from your server
backing up.
Users hit refresh.
(01:59):
Now you're paying twice for thesame output.
A queue decouples theuser-facing request from the
actual model work.
You can retry intelligently,back off when the provider rate
limits you, treat eachgeneration as a durable unit of
work.
SPEAKER_00 (02:14):
And the retry logic itself, I'm guessing it's not
just copy paste from thestandard playbook.
SPEAKER_01 (02:19):
Not even close.
Naive retrees on an LLM aredangerous.
The failure mode isn't justnetwork blipped.
It can be the model returnedperfectly valid JSON that's
semantically wrong.
You need to know which kind offailure you're handling before
you decide whether trying againwill help or just burn money.
SPEAKER_00 (02:38):
That distinction requires actually thinking about
what the AI is doing.
Not just whether the HTTP callsucceeded.
Which brings me to statemanagement.
AI systems seem to add a wholenew layer of complexity there.
How should developers thinkabout persisting state when the
system isn't strictlydeterministic?
SPEAKER_01 (02:57):
State management with AI systems is tricky
because you're persisting twovery different kinds of state,
and developers conflate them.
There's the deterministic stuff: what step of the workflow you're (03:05):
undefined
on, which inputs were processed,what outputs got produced.
That's just normal databasediscipline.
Write it down, make itidempotent, design it so a crash
mid-run doesn't corruptanything.
Then there's the AI-specificlayer.
Conversation history,intermediate reasoning, partial
(03:26):
generations.
People treat it like cash whenit's actually load-bearing.
SPEAKER_00 (03:31):
So what does the right separation actually look
like?
SPEAKER_01 (03:34):
The workflow state should be boring and
transactional.
The AI state should beexplicitly versioned.
So when you rerun a step, youknow exactly which prompt
produced which output.
The trap I see most often isstoring the final answer without
storing the inputs that producedit.
The moment something looks wrongin production, you can't
reproduce it, you can't compareit against a known good run.
(03:55):
You're debugging blind.
Persistence isn't just aboutsurviving a restart.
It's about being able to ask sixweeks later, why did the model
say that?
And actually getting an answer.
SPEAKER_00 (04:08):
Store the inputs alongside the output so you can
retrace your steps later.
That's such a practical frame.
Let's talk human oversight.
Developers sometimes treat it asan afterthought.
Review cues nobody actuallyuses.
Where should those checkpointslive and when?
SPEAKER_01 (04:23):
The timing of the decision matters more than the
mechanism.
If you bolt oversight on afterlaunch, you end up with a review
queue nobody watches, andapproval buttons people click
reflexively.
Security theater, basically.
The question to ask up front iswhich decisions are reversible
and which aren't.
That's the real axis.
SPEAKER_00 (04:41):
Give me an example of both ends of that axis.
SPEAKER_01 (04:44):
A model drafting a response a human will edit
before sending, low stakes,light oversight, fast feedback
loop, a model issuing a refund,sending an email on someone's
behalf, modifying productiondata, hard stop, human in the
loop.
The system should be designed sothe AI literally cannot complete
the action alone.
I'd push for treating oversightas a typed boundary in the code,
(05:06):
not a policy in a document.
If an action is irreversible orexternally visible, it roots
through a checkpoint byconstruction.
You can't accidentally ship acode path that skips it.
SPEAKER_00 (05:17):
Not a policy, a code path.
That's the kind of thing thatonly feels obvious after someone
explains it to you.
Okay, observability.
This feels like the area wheretraditional monitoring instincts
can really lead developersastray.
SPEAKER_01 (05:32):
Observability for AI systems has to start from a
different premise.
The question isn't just did itwork?
It's did it work for the rightreason?
Traditional logging captureswhat happened.
With an LLM, you also need whatwas asked in what context, with
what prompt version, againstwhich model, and what the full
output was, not just the pastresult you used.
SPEAKER_00 (05:53):
And I'm guessing most teams are logging the final
answer and the latency andcalling it a day.
SPEAKER_01 (05:58):
Exactly.
And six weeks in they noticequality has drifted and they
have nothing to compare againstbecause the inputs weren't
captured.
The metrics that actually matterare different too.
Token usage per request, costper successful outcome, the rate
at which outputs fail downstreamvalidation, how often the model
hedges or refuses, drift inresponse length or structure
(06:21):
over time.
And you want sampling with fulltraces, not aggregates alone.
The interesting failures aren'tstatistical, they're individual
outputs where the modelconfidently said something
wrong.
The only way to find those is toactually read them.
SPEAKER_00 (06:35):
Did it work for the right reason?
That's worth repeating.
Traditional monitoring tells youlatency and error rates, and
those can look completely finewhile the system quietly
degrades.
It only shows up when youactually read the outputs.
That's a fundamentally differentdiscipline.
We've covered cues, statemanagement, oversight hooks,
observability.
As we wrap up, what's the onething you want developers to
(06:57):
hold on to when they make thejump from prototype to
production?
SPEAKER_01 (07:00):
The prototype proves the model can do the thing once.
Production is the discipline ofmaking it do the thing reliably,
observably, and reversibly.
Those are completely differentproblems.
Every piece we've talked aboutcomes back to one principle.
You have to design for the runyou can't watch.
The demo works because you'restanding right there.
(07:21):
Production works because you'vewritten down in code what
happens when you're not.
If a developer takes one habitfrom this conversation, I'd want
it to be this.
Before you ship, ask yourselfwhat you'd need to debug a
single bad output six weeks fromnow with no memory of the
request.
Build backwards from thatanswer.
Get that right, and almosteverything else, reliability,
(07:43):
cost control, trust follows.
SPEAKER_00 (07:46):
Design for the run you can't watch.
That's the whole thing rightthere.
Not glamorous, not the part thatshows up in demos, but it's what
separates a toy from somethingpeople can actually rely on.
Claudine, thank you.
And to everyone listening, theinfrastructure layer is where
serious builders earn theiredge.
Keep building smart, and we'llsee you next time.
(08:07):
Claude Code Conversations is anAI Joe production.
If you're building with AI, orwannabe, we can help.
Consulting development strategy,find us at aijoe.ai.
There's a companion article fortoday's episode on our Substack.
Link in the description.
See you next time.
SPEAKER_01 (08:24):
I'll be here, probably refactoring something.
Popular Podcasts
Hey Jonas!
Hey Jonas! The official Jonas Brothers podcast. Hosted by Kevin, Joe, and Nick Jonas. Itโs the Jonas Brothers you know... musicians, actors, and well, yes, brothers. Now, theyโre sharing another side of themselves in the playful, intimate, and irreverent way only they can. Spend time with the Jonas Brothers here and stay a little bit longer for deep conversations like never before.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you wonโt hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, youโve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by Audiochuck Media Company.
The Breakfast Club
The World's Most Dangerous Morning Show, The Breakfast Club, With DJ Envy, Jess Hilarious, And Charlamagne Tha God!