October 6, 2026 • 21 mins
A comprehensive guide to building production-grade predictive applications using the reactive systems paradigm. Through the narrative of a fictional startup, the author illustrates common pitfalls of naive designs, such as system instability, lack of scalability, and cascading failures. The text advocates for a modular architecture that is responsive, resilient, elastic, and message-driven to ensure reliability under heavy loads. Key technologies highlighted include Scala, Akka, and Spark, which facilitate functional programming and distributed data processing. Readers learn to manage the entire lifecycle of a model, from data collection and feature generation to model publishing and real-world response. Ultimately, the source serves as a blueprint for transforming machine learning from a mere experimental technique into a robust, scalable software system.
You can listen and download our episodes for free on more than 10 different platforms:
https://linktr.ee/cyber_security_summary
Get the Book now from Amazon:
https://www.amazon.com/Reactive-Machine-Learning-Systems-Smith/dp/1617293334?&linkCode=ll2&tag=cvthunderx-20&linkId=a0dd0c59bfe912a7e0614a154c6a7454&language=en_US&gaOptInStatus=true&ref_=as_li_ss_tl
Discover our free courses in tech and cybersecurity, Start learning today:
https://linktr.ee/cybercode_academy
You can listen and download our episodes for free on more than 10 different platforms:
https://linktr.ee/cyber_security_summary
Get the Book now from Amazon:
https://www.amazon.com/Reactive-Machine-Learning-Systems-Smith/dp/1617293334?&linkCode=ll2&tag=cvthunderx-20&linkId=a0dd0c59bfe912a7e0614a154c6a7454&language=en_US&gaOptInStatus=true&ref_=as_li_ss_tl
Discover our free courses in tech and cybersecurity, Start learning today:
https://linktr.ee/cybercode_academy
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:00):
So imagine for a moment that you're nurturing this rare
exotic plant.
Speaker 2 (00:06):
Okay. I'm picturing it.
Speaker 1 (00:07):
You've got it in this perfectly climate-controlled greenhouse, right? Like
you control the soil pH, you measure the water down
to the exact milliliter, and the temperature is just always
hovering at exactly 72 degrees.
Speaker 2 (00:20):
And naturally it thrives in there. Right. It thrives.
Speaker 1 (00:22):
But then, you know, you take that delicate little plant,
you carry it outside, and you just toss it directly
into a Category 5 hurricane. Right.
Speaker 2 (00:29):
I mean- it is going to get absolutely shredded.
Speaker 1 (00:32):
Completely shredded because a greenhouse has, you know, static conditions
and a single caretaker. The real world has unpredictable wind shear,
flying debris, and just total chaos.
Speaker 2 (00:42):
Right, yeah.
Speaker 1 (00:43):
And that immediate catastrophic destruction is exactly what it feels
like when you take a pristine, mathematically beautiful machine learning
model one you built in a vacuum on a laptop
and you blindly deploy it out into the wild.
Speaker 2 (00:56):
Out into the reality of, you know, Millions of concurrent users.
You've got dropping network connections, data corruption, endless edge cases.
It's a mess. It really is. I mean, the math
itself doesn't break. But the plumbing surrounding the math absolutely shatters.
Speaker 1 (01:09):
Which brings us to today's deep dive. We are looking
at Jeff Smith's incredible book, Machine Learning Systems Designs That Scale.
And we are exploring why mathematically perfect models just constantly
fail in the real world. Right. Our mission for you
today is to really bridge that massive gap between theoretical
data science and real-world deployment. We're going to explore how
(01:33):
to stop fighting server fires, basically.
Speaker 2 (01:35):
Yeah, and how to start building predictive systems that can
actually survive contact with live traffic. Exactly. And what really
sets the stage for this is the foreword of the
book by Sean Owen. He highlights this fundamental historical shift
in how we actually build these systems.
Speaker 1 (01:50):
Oh, right, the shift from the old tools.
Speaker 2 (01:52):
Yeah, if you look back, We used to rely on
these incredibly clunky JVM-based Hadoop tools to do the heavy
data lifting.
Speaker 1 (01:58):
And Hadoop relied on MapReduce.
Speaker 2 (02:00):
Exactly. Which fundamentally meant writing intermediate steps back to physical
hard disks. I mean, it was agonizingly slow.
Speaker 1 (02:07):
It's writing to disk over and over. Yeah.
Speaker 2 (02:09):
But recently, we transitioned to modern functional programming paradigms. Tools
like Spark, which handled data and memory, and just completely
changed our assumptions about scale and speed.
Speaker 1 (02:20):
So we're going to dive into a completely different architectural
mindset today called reactive design, right? Just to take advantage
of these new tools. But before we learn how to
build a scalable system, I think we have to look
at a relatable, spectacular failure to understand the mechanics of
what actually goes wrong.
Speaker 2 (02:36):
Oh, absolutely. Let me introduce you to a fictional startup
from our source material called Sniffable.
Speaker 1 (02:42):
Sniffable?
Speaker 2 (02:43):
Yeah, Sniffable. The business model is simple. It is Facebook.
But for dogs.
Speaker 1 (02:48):
I mean, that's a... Completely untapped goldmine.
Speaker 2 (02:51):
Naturally. So business is booming, but the product team wants
a new feature. They realize their most highly engaged users
are den mothers.
Speaker 1 (03:00):
Denom mothers.
Speaker 2 (03:01):
Yeah, like the canine equivalent of intense stage moms who
just desperately want their pets to be famous.
Speaker 1 (03:07):
Oh, wow. Okay, I know the type.
Speaker 2 (03:09):
Right. So Snivable decides to build the Pooch Predictor. It's
a tool where a den mother can draft a post,
they call it a bupdate, and before she hits send,
the system... analyzes her hashtags and predicts exactly how many
likes it will get.
Speaker 1 (03:23):
Okay, let's unpack this. They have a very clear business
use case here. So they hand this off to their
lone data scientist, right? Yes.
Speaker 2 (03:31):
And he does what a lot of folks would do
in a totally isolated environment.
Speaker 1 (03:34):
Right. He writes a script. He pulls the raw interaction
data from their database into like a few flat CSV files.
And he runs a model learning algorithm over those files.
Speaker 2 (03:45):
Classic approach.
Speaker 1 (03:46):
Yeah. Yeah. He creates a predictive model, declares victory, and
then hands the finalized model over to a junior backend engineer.
Speaker 2 (03:53):
And the junior engineer, who, you know, isn't a data scientist,
just embeds that finalized model directly into the main application's code.
And they ship it to production.
Speaker 1 (04:03):
Which just triggers an immediate chain reaction of disasters. I mean,
first there was the stale data problem.
Speaker 2 (04:08):
But yeah, the predictions basically never changed over time.
Speaker 1 (04:11):
Right. The data scientists just assumed the model was constantly
updating itself. But because it was hard-coded into the app,
nobody realized it needed an automated pipeline to retrain it
daily with new data.
Speaker 2 (04:22):
And then came the international incident.
Speaker 1 (04:26):
Oh, this was bad.
Speaker 2 (04:27):
Suddenly, for users outside the United States, Pooch Predictor started
predicting negative likes on their pup dates.
Speaker 1 (04:34):
Which absolutely enraged the community. I mean, people felt the
app was actively insulting the adorableness of their dogs.
Speaker 2 (04:40):
Yeah, it was this brutal bug in how they processed
location-based features that But it really highlights the danger of
untested edge cases. Definitely. But the final nail in the
coffin was the total system crash. Because to improve the model,
the data scientists required the app to send massive amounts
of telemetry data back to the database every single time
someone posted.
Speaker 1 (05:00):
Wait, so they took a heavy telemetry requirement and just
shoved it into the main app thread. What did that
actually do to the servers mechanically?
Speaker 2 (05:07):
It locked them up completely. I mean, every time a
user hit post, the main thread of the mobile app
had to pause and and wait for a heavy, synchronous
database write to finish before the user interface could update.
Speaker 1 (05:19):
Oh, no.
Speaker 2 (05:20):
Yeah. So when 10,000 denmothers posted it once, Sniffable literally
ran out of database connections. The threads backed up, the
servers threw out of memory exceptions, and the apps slowed
to an absolute crawl before just dropping pupdates entirely.
Speaker 1 (05:37):
So they basically built this brilliant custom Ferrari engine, which
is the Algorithm, And they duct taped it to a
golf cart chassis.
Speaker 2 (05:44):
Yeah, exactly.
Speaker 1 (05:45):
The second they hit the gas, the torque just ripped
the chassis apart. Right.
Speaker 2 (05:49):
And what's fascinating here is the underlying mindset issue. The
team viewed machine learning merely as a technique.
Speaker 1 (05:55):
Right, like a math problem.
Speaker 2 (05:56):
Exactly. To them, it was just an isolated math problem
to solve on a laptop. They fundamentally failed to view
it as an application. A real machine learning system has
five distinct phases, and you really cannot skip any of them.
Speaker 1 (06:09):
Walk us through those phases really quick. Sure.
Speaker 2 (06:11):
First is data collection, getting the data from the real
world safely without crashing your database. Second is feature and
concept generation, which is translating that raw data into a
format the model actually understands. Third is model learning, which,
by the way, is the only phase Sniffable actually focused on.
Speaker 1 (06:30):
Of course.
Speaker 2 (06:31):
Fourth is model publishing, moving the model out of the
lab and wrapping it in an API or a container.
And finally, responding, actually serving that prediction safely, to a
live user at scale.
Speaker 1 (06:43):
So Sniffable completely ignored four out of the five phases?
Speaker 2 (06:46):
Completely.
Speaker 1 (06:47):
Wow. So if tight coupling and shared databases are what
brought Sniffable to its knees, I guess decoupling those pieces
is the only way forward, which brings us to the
reactive approach. Right.
Speaker 2 (06:57):
The reactive approach is a fundamentally different architectural mindset, and
it's built on four core traits.
Speaker 1 (07:01):
Okay.
Speaker 2 (07:02):
First, a system must be responsive. It has to consistently
return timely responses no matter what. Second, it must be resilient.
It has to maintain that responsiveness even when hardware fails
or networks drop. Third, it needs to be elastic, so
effortlessly scaling up when traffic spikes and scaling down to
(07:22):
save money when things are quiet.
Speaker 1 (07:24):
It makes sense.
Speaker 2 (07:25):
And fourth, it must be message-driven. This means components communicate
via asynchronous, non-blocking message passing. It ensures loose coupling.
Speaker 1 (07:34):
Okay, I am going to challenge you on this because responsive, resilient...
I mean, those sound like empty buzzwords to me.
Speaker 2 (07:42):
Fair enough.
Speaker 1 (07:42):
Every company claims their software is resilient. So how do
you physically force a system to behave this way at
a code level?
Speaker 2 (07:49):
It is a vital question. You do not just wish
these traits into existence. You engineer them using three highly
specific strategies.
Speaker 1 (07:56):
Okay, what's the first one?
Speaker 2 (07:57):
First is replication. You redundantly process data. So let's look
at Spark, which we mentioned earlier. Spark uses something called
a directed acyclic graph or a DAG.
Speaker 1 (08:05):
A BAG-GAG. Right.
Speaker 2 (08:06):
It doesn't just pass raw data around. It keeps a
meticulous step-by-step log of the transformations required to process the data.
Speaker 1 (08:13):
So wait, if a server node physically catches fire in
the middle of training a model?
Speaker 2 (08:19):
The system doesn't panic. The master node simply looks at
the DAG, assigns a new healthy node, and that new
node grabs the original data slice and replays the exact steps.
Speaker 1 (08:31):
Oh, wow.
Speaker 2 (08:31):
Yeah, it doesn't even need to ask the dead node
where it left off. That provides absolute resilience without a
single point of failure.
Speaker 1 (08:38):
Okay, that covers infrastructure failure. But what about logic failures?
Like in the sniffable disaster, a bad hashtag threw an
exception in the model and it took down the entire
mobile app.
Speaker 2 (08:49):
That requires a second strategy, which is containment. You have
to isolate failures so they cannot cascade across the network.
Speaker 1 (08:55):
Oh, I see it. Like watertight doors on a submarine.
Speaker 2 (08:58):
Yes.
Speaker 1 (08:59):
Like if one compartment breaches and floods, you just seal
the bulkhead door. I mean, you might lose the use
of that specific room, but the submarine itself doesn't sink.
Speaker 2 (09:08):
That is the perfect analogy. If the machine learning model
throws a null pointer exception, that error must be trapped
inside its own isolated container. The user might not get
their pooch predictor estimate. But the main app doesn't crash.
They can still post their photo.
Speaker 1 (09:23):
So if we connect this to the bigger picture, containing
an error is great, but we still have a broken
model sitting in a flooded compartment, basically. How does the
system actually fix itself?
Speaker 2 (09:34):
This leads to the third strategy, supervision. You build an
architectural component called a model supervisor.
Speaker 1 (09:40):
But how does a supervisor sit outside the application to
watch it without, you know, getting dragged down if the
app crashes?
Speaker 2 (09:47):
It utilizes an actor model. So the supervisor is an
entirely separate, isolated process.
Speaker 1 (09:52):
Okay.
Speaker 2 (09:53):
It isn't running the model. It's simply pinging the model's
container and watching the health metrics.
Speaker 1 (09:58):
Oh, so it's just checking in.
Speaker 2 (10:00):
Exactly. If the model starts acting up, say it starts
spitting out those negative likes, the supervisor intercepts the traffic.
And because it sits outside the blast radius, it can
literally issue a kill command to destroy the rogue model's
container and seamlessly spin up a fresh backup version.
Speaker 1 (10:15):
That's wild. It makes the system completely self-healing.
Speaker 2 (10:18):
It really does.
Speaker 1 (10:19):
Here's where it gets really interesting, though. Standard software deals
with finite transactions, right? Like a bank transfer has a beginning,
a middle, and an end. True. But our sources point
out that machine learning systems have this terrifying quirk that
threatens to break all of this reactive design. The data
you are dealing with is effectively infinite.
Speaker 2 (10:37):
Yes. Think about the problem of scale with a social
network like Sniffable. You cannot possibly know, categorize, or store data.
Every single variation of a future pup date. Right. You
have infinite variations in dog breeds, lighting, filters, typos. I mean,
is a user typing hashtag adorable with you or hashtag
(10:57):
adorable with any? The stream of inputs from the real
world has no start and no end.
Speaker 1 (11:02):
Let me stop you there. My hard drive is a
finite piece of metal. RAM is finite. How does a
physical computer hold an infinite stream of data and memory
without immediately crashing?
Speaker 2 (11:11):
It sounds paradoxical, I know, but functional programming solves this
using a concept called laziness.
Speaker 1 (11:16):
Laziness, which is usually a terrible trait in an engineer.
Speaker 2 (11:20):
Right, but a brilliant one in a compiler. In functional programming,
laziness means a delay of execution. Okay. So instead of
trying to load a million rows of data into RAM
at once, which is impossible if the data never ends,
the system creates a pointer to an unevaluated expression. We
call it a thunk.
Speaker 1 (11:38):
A thunk. So it's not actually holding the data. It's
holding like an IOU.
Speaker 2 (11:41):
Precisely. The program basically says, I know the mathematical formula
to calculate this data whenever you actually need it. Right.
It conceives the data as an infinite flowing stream, but
it only evaluates the expression and pulls the data into
memory at the exact fraction of a second that the
final output is demanded by the user.
Speaker 1 (11:59):
Wow.
Speaker 2 (11:59):
Yeah. It does absolutely no work until it is forced to.
Speaker 1 (12:02):
It just sips from the fire hose on demand. But
doing that safely seems incredible. incredibly fragile.
Speaker 2 (12:08):
It is, which is why laziness only works if you
strictly enforce something called pure functions. This is a core
tenet of languages like Scala, which Jeff Smith relies heavily
on in the text.
Speaker 1 (12:18):
So what exactly makes a function pure?
Speaker 2 (12:20):
A pure function has two unbreakable rules. First, it always
returns the exact same output for the exact same input,
which is called referential transparency. Second, It has zero side effects. Okay.
Speaker 1 (12:34):
Explain what a side effect is in this context, because
getting rid of them is a massive takeaway for anyone
writing code.
Speaker 2 (12:40):
Sure. A side effect is when a function modifies a variable,
updates a database, or changes a state outside of its
own local scope. When you evaluate a lazy function hours
after you wrote the IOU, if evaluating it suddenly changes
a global variable that another part of your app relies on,
your system will just collapse in unpredictable ways.
Speaker 1 (13:00):
Ah, it's the sneaky state change bug. So a pure
function just takes an input, returns an output, and touches
absolutely nothing else in the world.
Speaker 2 (13:08):
Exactly. But even if you master pure functions and lazy streams,
you still have to face the ultimate boss of machine learning,
which is the real world.
Speaker 1 (13:16):
Right. The data coming in isn't just infinite. It is messy,
it's delayed, and wildly uncertain.
Speaker 2 (13:21):
Very much so.
Speaker 1 (13:22):
Let's talk about that uncertainty. Say a front-end developer at
Sniffable decides to change how timestamps are formatted on the
mobile app.
Speaker 2 (13:29):
Oh boy.
Speaker 1 (13:30):
They silently move from a standard Unix integer to an
ISO string, and they just don't tell the data team.
Speaker 2 (13:36):
A classic breakdown in communication. And if your database simply
accepts that new string and overwrites the old format, your
downstream machine learning model, which is mathematically expecting an integer,
will suddenly panic and crash.
Speaker 1 (13:49):
So how does a reactive system handle the reality of
bad or changing data?
Speaker 2 (13:55):
You have to rethink storage completely. In a reactive system,
you do not overwrite data. Ever. You record immutable facts.
Speaker 1 (14:02):
Immutable facts. Yes.
Speaker 2 (14:04):
When a payload of data comes in from the real world,
no matter how messy or weird it is, you save
the raw, exact payload. You write that complex truth in stone,
you append it to a log, and you never, ever
modify it.
Speaker 1 (14:15):
So you're just capturing history exactly as it happened.
Speaker 2 (14:18):
Right. and you let the machine learning model figure out
how to parse that history later. This leads to a
fascinating prediction strategy called possible worlds.
Speaker 1 (14:25):
That sounds incredibly theoretical.
Speaker 2 (14:27):
I know, but it's actually highly practical. In machine learning,
you rarely have one definitive truth about the future.
Speaker 1 (14:34):
Okay.
Speaker 2 (14:34):
Say Sniffable wants to allocate server capacity for the next hour,
and they need to know how many likes French Bulldogs
are going to get. You don't calculate one rigid number.
You calculate multiple realities.
Speaker 1 (14:44):
Walk me through the math on that.
Speaker 2 (14:46):
Okay, so... World A is the historical baseline. Based on
past behavior, we expect 6,500 pup dates.
Speaker 1 (14:52):
Right.
Speaker 2 (14:53):
We multiply that by our historical average of 23 likes
per post. That gives us an expectation of 149,500 likes.
Speaker 1 (15:01):
Straightforward enough.
Speaker 2 (15:02):
But World B is the ML-predicted reality. Our model is
analyzing real-time trending hashtags and projects, a massive surge, let's
say 7,250 pup dates. Wow. Okay. And because Frenchies are
currently viral... It predicts 28 likes each. That gives us 203,300 likes.
Speaker 1 (15:19):
So two very different numbers.
Speaker 2 (15:21):
Exactly. By querying across these different possible worlds simultaneously, the
system understands the range of uncertainty and spins up enough
server capacity to handle the absolute worst-case scenario.
Speaker 1 (15:31):
That's brilliant. I want to raise the stakes here with
an example from the book that perfectly illustrates the danger
of scale. Forget standard social media posting. Let's talk about
Hollywood Star.
Speaker 2 (15:41):
Oh, right. Their reality singing competition for dogs.
Speaker 1 (15:44):
Yes. Dogs barking to music on live television being judged.
And millions of viewers at home are voting on the
sniffable app at the exact same millisecond. I mean, the
traffic spikes are instantaneous and massive.
Speaker 2 (15:58):
This introduces a terrifying concept in distributed systems called tail latency.
Speaker 1 (16:03):
Tail latency. Yeah.
Speaker 2 (16:05):
When millions of users are voting... A single vote doesn't
just hit one server. It might touch dozens of microservices.
Speaker 1 (16:11):
Like authentication, fraud detection, the vote tally database.
Speaker 2 (16:15):
Exactly.
Speaker 1 (16:16):
And the wait time for the user isn't the average
speed of those services combined.
Speaker 2 (16:20):
No. The total wait time is entirely dictated by the
single slowest request in the chain. That is the tail latency. Well, man.
If you have 99 microservices operating in one millisecond, but
one remote server handling fraud detection hangs for five seconds.
Speaker 1 (16:35):
The user's mobile app freezes for five full seconds. Exactly.
This raises an important question, though. How do you physically
prevent one struggling backend server from completely freezing the user
interface for millions of people?
Speaker 2 (16:49):
This is where functional programming languages like Scala really earn
their keep. Scala has built in primitives to handle this asynchronously. First,
it uses something called futures.
Speaker 1 (17:00):
Futures. Right.
Speaker 2 (17:01):
Instead of the main application thread stopping and waiting for
that fraud server to respond, which is called blocking IO,
it fires off a future.
Speaker 1 (17:09):
So it essentially delegates the task. It says, you know,
go get this data on a background thread. Let me
know when you have it, but I'm going to keep
processing other things.
Speaker 2 (17:16):
Exactly. The main thread stays completely snappy. But Scala pairs
futures with something even more powerful called option types.
Speaker 1 (17:24):
OK, tell me about those.
Speaker 2 (17:25):
When the app asks the database for the vote count
for a dog named NomNum, It does not just get
an integer back. It gets an option.
Speaker 1 (17:33):
Explain how an option type mechanically differs from a regular variable.
Because this is the key to preventing crashes, right?
Speaker 2 (17:40):
It is. An option type acts like a highly secure
locked box. That box can only contain one of two things.
It either contains some value, like the number 42, or
it contains none, meaning the data is missing or just
hasn't arrived yet. Okay. The magic here is that the
compiler will physically refuse to compile your code unless you
(18:01):
use pattern matching to explicitly write logic for both scenarios.
Speaker 1 (18:06):
So it forces the programmer to handle the uncertainty up front.
It physically prevents a null pointer exception because you can
never accidentally call a variable that doesn't exist.
Speaker 2 (18:16):
Exactly. You have to tell the system, what do we
display if the database says sum 42, and what do
we display if the database returns? None, because the network
is congested.
Speaker 1 (18:26):
So what does this all mean for us? To me,
it means we have to actively design for failure. It's
like a world-class restaurant. If the kitchen runs out of
the signature dish, the head chef doesn't run to the
front door, lock it, shut off the power, and scream
in panic.
Speaker 2 (18:39):
Which is the architectural equivalent of what Sniffable did, honestly.
Speaker 1 (18:42):
Right. Instead, the waiter gracefully comes to your table, apologizes,
and offers you a complimentary appetizer.
Speaker 2 (18:48):
It is called graceful degradation.
Speaker 1 (18:51):
Exactly. In our Hollywood Star app, because we are using
futures and option types, we can program a strict timeout.
We can say, if that remote database takes longer than
500 milliseconds to fetch the live vote count, abandon the request.
Speaker 2 (19:05):
Just abandon it.
Speaker 1 (19:06):
Yeah. Just return none and use pattern matching to display
the historical average to the user's screen instead.
Speaker 2 (19:12):
And is displaying the historical average perfectly mathematically accurate in
that exact millisecond?
Speaker 1 (19:18):
No.
Speaker 2 (19:19):
But the app stays incredibly fast, the main thread doesn't
lock up, and the user remains completely engaged.
Speaker 1 (19:26):
Because the eventual consistency of the database catches up a
second later anyway.
Speaker 2 (19:30):
Precisely.
Speaker 1 (19:31):
It is a brilliant shift in perspective. So let's look
at the journey we've taken today. We moved away from
the naive, firefighting, duct-taped architecture of the original Pooch predictor.
Speaker 2 (19:41):
We really did.
Speaker 1 (19:42):
We explored how to build a robust, reactive machine learning system.
We learned how to use pure functions and lazy evaluation
to handle infinite difficulties. data streams without running out of memory.
We learned how to store immutable facts to honestly capture
reality and how to use futures and option types to
force that graceful degradation.
Speaker 2 (20:00):
And if there is one overarching takeaway for you today,
whether you are building, managing or even just interacting with
these massive machine learning systems. It is realizing that machine
learning is an application.
Speaker 1 (20:12):
It really is.
Speaker 2 (20:13):
It is an entire ecosystem of data pipelines, containment strategies,
and supervisors. It is not just an isolated algorithm. And
when you treat it like a holistic application, that is
your shortcut. to creating software that actually survives contact with
the real world.
Speaker 1 (20:30):
Which leaves us with a final, slightly provocative thought for
you to mull over. We have spent this entire deep
dive establishing that the most resilient machine learning systems are
fundamentally built on accepting uncertainty. Right. They are explicitly designed
to handle maybe, you know, to process multiple possible worlds
and to degrade gracefully when they don't know the exact truth.
(20:50):
But think about the world these models are being deployed into.
Our legal systems, our compliance frameworks, and our regulatory laws
demand absolute black and white deterministic certainty.
Speaker 2 (21:02):
They absolutely do.
Speaker 1 (21:03):
So it makes you wonder, when a reactive system is
perfectly comfortable with intentionally being a little bit wrong to survive,
how long will it take for our rigid legal structures
to catch up to the reality of how this software
actually works?
Speaker 2 (21:17):
That's a great question.
Speaker 1 (21:18):
Because it really is the only way that delicate plant
survives the hurricane. Thanks for taking a deep dive with us.
So imagine for a moment that you're nurturing this rare
exotic plant.
Speaker 2 (00:06):
Okay. I'm picturing it.
Speaker 1 (00:07):
You've got it in this perfectly climate-controlled greenhouse, right? Like
you control the soil pH, you measure the water down
to the exact milliliter, and the temperature is just always
hovering at exactly 72 degrees.
Speaker 2 (00:20):
And naturally it thrives in there. Right. It thrives.
Speaker 1 (00:22):
But then, you know, you take that delicate little plant,
you carry it outside, and you just toss it directly
into a Category 5 hurricane. Right.
Speaker 2 (00:29):
I mean- it is going to get absolutely shredded.
Speaker 1 (00:32):
Completely shredded because a greenhouse has, you know, static conditions
and a single caretaker. The real world has unpredictable wind shear,
flying debris, and just total chaos.
Speaker 2 (00:42):
Right, yeah.
Speaker 1 (00:43):
And that immediate catastrophic destruction is exactly what it feels
like when you take a pristine, mathematically beautiful machine learning
model one you built in a vacuum on a laptop
and you blindly deploy it out into the wild.
Speaker 2 (00:56):
Out into the reality of, you know, Millions of concurrent users.
You've got dropping network connections, data corruption, endless edge cases.
It's a mess. It really is. I mean, the math
itself doesn't break. But the plumbing surrounding the math absolutely shatters.
Speaker 1 (01:09):
Which brings us to today's deep dive. We are looking
at Jeff Smith's incredible book, Machine Learning Systems Designs That Scale.
And we are exploring why mathematically perfect models just constantly
fail in the real world. Right. Our mission for you
today is to really bridge that massive gap between theoretical
data science and real-world deployment. We're going to explore how
(01:33):
to stop fighting server fires, basically.
Speaker 2 (01:35):
Yeah, and how to start building predictive systems that can
actually survive contact with live traffic. Exactly. And what really
sets the stage for this is the foreword of the
book by Sean Owen. He highlights this fundamental historical shift
in how we actually build these systems.
Speaker 1 (01:50):
Oh, right, the shift from the old tools.
Speaker 2 (01:52):
Yeah, if you look back, We used to rely on
these incredibly clunky JVM-based Hadoop tools to do the heavy
data lifting.
Speaker 1 (01:58):
And Hadoop relied on MapReduce.
Speaker 2 (02:00):
Exactly. Which fundamentally meant writing intermediate steps back to physical
hard disks. I mean, it was agonizingly slow.
Speaker 1 (02:07):
It's writing to disk over and over. Yeah.
Speaker 2 (02:09):
But recently, we transitioned to modern functional programming paradigms. Tools
like Spark, which handled data and memory, and just completely
changed our assumptions about scale and speed.
Speaker 1 (02:20):
So we're going to dive into a completely different architectural
mindset today called reactive design, right? Just to take advantage
of these new tools. But before we learn how to
build a scalable system, I think we have to look
at a relatable, spectacular failure to understand the mechanics of
what actually goes wrong.
Speaker 2 (02:36):
Oh, absolutely. Let me introduce you to a fictional startup
from our source material called Sniffable.
Speaker 1 (02:42):
Sniffable?
Speaker 2 (02:43):
Yeah, Sniffable. The business model is simple. It is Facebook.
But for dogs.
Speaker 1 (02:48):
I mean, that's a... Completely untapped goldmine.
Speaker 2 (02:51):
Naturally. So business is booming, but the product team wants
a new feature. They realize their most highly engaged users
are den mothers.
Speaker 1 (03:00):
Denom mothers.
Speaker 2 (03:01):
Yeah, like the canine equivalent of intense stage moms who
just desperately want their pets to be famous.
Speaker 1 (03:07):
Oh, wow. Okay, I know the type.
Speaker 2 (03:09):
Right. So Snivable decides to build the Pooch Predictor. It's
a tool where a den mother can draft a post,
they call it a bupdate, and before she hits send,
the system... analyzes her hashtags and predicts exactly how many
likes it will get.
Speaker 1 (03:23):
Okay, let's unpack this. They have a very clear business
use case here. So they hand this off to their
lone data scientist, right? Yes.
Speaker 2 (03:31):
And he does what a lot of folks would do
in a totally isolated environment.
Speaker 1 (03:34):
Right. He writes a script. He pulls the raw interaction
data from their database into like a few flat CSV files.
And he runs a model learning algorithm over those files.
Speaker 2 (03:45):
Classic approach.
Speaker 1 (03:46):
Yeah. Yeah. He creates a predictive model, declares victory, and
then hands the finalized model over to a junior backend engineer.
Speaker 2 (03:53):
And the junior engineer, who, you know, isn't a data scientist,
just embeds that finalized model directly into the main application's code.
And they ship it to production.
Speaker 1 (04:03):
Which just triggers an immediate chain reaction of disasters. I mean,
first there was the stale data problem.
Speaker 2 (04:08):
But yeah, the predictions basically never changed over time.
Speaker 1 (04:11):
Right. The data scientists just assumed the model was constantly
updating itself. But because it was hard-coded into the app,
nobody realized it needed an automated pipeline to retrain it
daily with new data.
Speaker 2 (04:22):
And then came the international incident.
Speaker 1 (04:26):
Oh, this was bad.
Speaker 2 (04:27):
Suddenly, for users outside the United States, Pooch Predictor started
predicting negative likes on their pup dates.
Speaker 1 (04:34):
Which absolutely enraged the community. I mean, people felt the
app was actively insulting the adorableness of their dogs.
Speaker 2 (04:40):
Yeah, it was this brutal bug in how they processed
location-based features that But it really highlights the danger of
untested edge cases. Definitely. But the final nail in the
coffin was the total system crash. Because to improve the model,
the data scientists required the app to send massive amounts
of telemetry data back to the database every single time
someone posted.
Speaker 1 (05:00):
Wait, so they took a heavy telemetry requirement and just
shoved it into the main app thread. What did that
actually do to the servers mechanically?
Speaker 2 (05:07):
It locked them up completely. I mean, every time a
user hit post, the main thread of the mobile app
had to pause and and wait for a heavy, synchronous
database write to finish before the user interface could update.
Speaker 1 (05:19):
Oh, no.
Speaker 2 (05:20):
Yeah. So when 10,000 denmothers posted it once, Sniffable literally
ran out of database connections. The threads backed up, the
servers threw out of memory exceptions, and the apps slowed
to an absolute crawl before just dropping pupdates entirely.
Speaker 1 (05:37):
So they basically built this brilliant custom Ferrari engine, which
is the Algorithm, And they duct taped it to a
golf cart chassis.
Speaker 2 (05:44):
Yeah, exactly.
Speaker 1 (05:45):
The second they hit the gas, the torque just ripped
the chassis apart. Right.
Speaker 2 (05:49):
And what's fascinating here is the underlying mindset issue. The
team viewed machine learning merely as a technique.
Speaker 1 (05:55):
Right, like a math problem.
Speaker 2 (05:56):
Exactly. To them, it was just an isolated math problem
to solve on a laptop. They fundamentally failed to view
it as an application. A real machine learning system has
five distinct phases, and you really cannot skip any of them.
Speaker 1 (06:09):
Walk us through those phases really quick. Sure.
Speaker 2 (06:11):
First is data collection, getting the data from the real
world safely without crashing your database. Second is feature and
concept generation, which is translating that raw data into a
format the model actually understands. Third is model learning, which,
by the way, is the only phase Sniffable actually focused on.
Speaker 1 (06:30):
Of course.
Speaker 2 (06:31):
Fourth is model publishing, moving the model out of the
lab and wrapping it in an API or a container.
And finally, responding, actually serving that prediction safely, to a
live user at scale.
Speaker 1 (06:43):
So Sniffable completely ignored four out of the five phases?
Speaker 2 (06:46):
Completely.
Speaker 1 (06:47):
Wow. So if tight coupling and shared databases are what
brought Sniffable to its knees, I guess decoupling those pieces
is the only way forward, which brings us to the
reactive approach. Right.
Speaker 2 (06:57):
The reactive approach is a fundamentally different architectural mindset, and
it's built on four core traits.
Speaker 1 (07:01):
Okay.
Speaker 2 (07:02):
First, a system must be responsive. It has to consistently
return timely responses no matter what. Second, it must be resilient.
It has to maintain that responsiveness even when hardware fails
or networks drop. Third, it needs to be elastic, so
effortlessly scaling up when traffic spikes and scaling down to
(07:22):
save money when things are quiet.
Speaker 1 (07:24):
It makes sense.
Speaker 2 (07:25):
And fourth, it must be message-driven. This means components communicate
via asynchronous, non-blocking message passing. It ensures loose coupling.
Speaker 1 (07:34):
Okay, I am going to challenge you on this because responsive, resilient...
I mean, those sound like empty buzzwords to me.
Speaker 2 (07:42):
Fair enough.
Speaker 1 (07:42):
Every company claims their software is resilient. So how do
you physically force a system to behave this way at
a code level?
Speaker 2 (07:49):
It is a vital question. You do not just wish
these traits into existence. You engineer them using three highly
specific strategies.
Speaker 1 (07:56):
Okay, what's the first one?
Speaker 2 (07:57):
First is replication. You redundantly process data. So let's look
at Spark, which we mentioned earlier. Spark uses something called
a directed acyclic graph or a DAG.
Speaker 1 (08:05):
A BAG-GAG. Right.
Speaker 2 (08:06):
It doesn't just pass raw data around. It keeps a
meticulous step-by-step log of the transformations required to process the data.
Speaker 1 (08:13):
So wait, if a server node physically catches fire in
the middle of training a model?
Speaker 2 (08:19):
The system doesn't panic. The master node simply looks at
the DAG, assigns a new healthy node, and that new
node grabs the original data slice and replays the exact steps.
Speaker 1 (08:31):
Oh, wow.
Speaker 2 (08:31):
Yeah, it doesn't even need to ask the dead node
where it left off. That provides absolute resilience without a
single point of failure.
Speaker 1 (08:38):
Okay, that covers infrastructure failure. But what about logic failures?
Like in the sniffable disaster, a bad hashtag threw an
exception in the model and it took down the entire
mobile app.
Speaker 2 (08:49):
That requires a second strategy, which is containment. You have
to isolate failures so they cannot cascade across the network.
Speaker 1 (08:55):
Oh, I see it. Like watertight doors on a submarine.
Speaker 2 (08:58):
Yes.
Speaker 1 (08:59):
Like if one compartment breaches and floods, you just seal
the bulkhead door. I mean, you might lose the use
of that specific room, but the submarine itself doesn't sink.
Speaker 2 (09:08):
That is the perfect analogy. If the machine learning model
throws a null pointer exception, that error must be trapped
inside its own isolated container. The user might not get
their pooch predictor estimate. But the main app doesn't crash.
They can still post their photo.
Speaker 1 (09:23):
So if we connect this to the bigger picture, containing
an error is great, but we still have a broken
model sitting in a flooded compartment, basically. How does the
system actually fix itself?
Speaker 2 (09:34):
This leads to the third strategy, supervision. You build an
architectural component called a model supervisor.
Speaker 1 (09:40):
But how does a supervisor sit outside the application to
watch it without, you know, getting dragged down if the
app crashes?
Speaker 2 (09:47):
It utilizes an actor model. So the supervisor is an
entirely separate, isolated process.
Speaker 1 (09:52):
Okay.
Speaker 2 (09:53):
It isn't running the model. It's simply pinging the model's
container and watching the health metrics.
Speaker 1 (09:58):
Oh, so it's just checking in.
Speaker 2 (10:00):
Exactly. If the model starts acting up, say it starts
spitting out those negative likes, the supervisor intercepts the traffic.
And because it sits outside the blast radius, it can
literally issue a kill command to destroy the rogue model's
container and seamlessly spin up a fresh backup version.
Speaker 1 (10:15):
That's wild. It makes the system completely self-healing.
Speaker 2 (10:18):
It really does.
Speaker 1 (10:19):
Here's where it gets really interesting, though. Standard software deals
with finite transactions, right? Like a bank transfer has a beginning,
a middle, and an end. True. But our sources point
out that machine learning systems have this terrifying quirk that
threatens to break all of this reactive design. The data
you are dealing with is effectively infinite.
Speaker 2 (10:37):
Yes. Think about the problem of scale with a social
network like Sniffable. You cannot possibly know, categorize, or store data.
Every single variation of a future pup date. Right. You
have infinite variations in dog breeds, lighting, filters, typos. I mean,
is a user typing hashtag adorable with you or hashtag
(10:57):
adorable with any? The stream of inputs from the real
world has no start and no end.
Speaker 1 (11:02):
Let me stop you there. My hard drive is a
finite piece of metal. RAM is finite. How does a
physical computer hold an infinite stream of data and memory
without immediately crashing?
Speaker 2 (11:11):
It sounds paradoxical, I know, but functional programming solves this
using a concept called laziness.
Speaker 1 (11:16):
Laziness, which is usually a terrible trait in an engineer.
Speaker 2 (11:20):
Right, but a brilliant one in a compiler. In functional programming,
laziness means a delay of execution. Okay. So instead of
trying to load a million rows of data into RAM
at once, which is impossible if the data never ends,
the system creates a pointer to an unevaluated expression. We
call it a thunk.
Speaker 1 (11:38):
A thunk. So it's not actually holding the data. It's
holding like an IOU.
Speaker 2 (11:41):
Precisely. The program basically says, I know the mathematical formula
to calculate this data whenever you actually need it. Right.
It conceives the data as an infinite flowing stream, but
it only evaluates the expression and pulls the data into
memory at the exact fraction of a second that the
final output is demanded by the user.
Speaker 1 (11:59):
Wow.
Speaker 2 (11:59):
Yeah. It does absolutely no work until it is forced to.
Speaker 1 (12:02):
It just sips from the fire hose on demand. But
doing that safely seems incredible. incredibly fragile.
Speaker 2 (12:08):
It is, which is why laziness only works if you
strictly enforce something called pure functions. This is a core
tenet of languages like Scala, which Jeff Smith relies heavily
on in the text.
Speaker 1 (12:18):
So what exactly makes a function pure?
Speaker 2 (12:20):
A pure function has two unbreakable rules. First, it always
returns the exact same output for the exact same input,
which is called referential transparency. Second, It has zero side effects. Okay.
Speaker 1 (12:34):
Explain what a side effect is in this context, because
getting rid of them is a massive takeaway for anyone
writing code.
Speaker 2 (12:40):
Sure. A side effect is when a function modifies a variable,
updates a database, or changes a state outside of its
own local scope. When you evaluate a lazy function hours
after you wrote the IOU, if evaluating it suddenly changes
a global variable that another part of your app relies on,
your system will just collapse in unpredictable ways.
Speaker 1 (13:00):
Ah, it's the sneaky state change bug. So a pure
function just takes an input, returns an output, and touches
absolutely nothing else in the world.
Speaker 2 (13:08):
Exactly. But even if you master pure functions and lazy streams,
you still have to face the ultimate boss of machine learning,
which is the real world.
Speaker 1 (13:16):
Right. The data coming in isn't just infinite. It is messy,
it's delayed, and wildly uncertain.
Speaker 2 (13:21):
Very much so.
Speaker 1 (13:22):
Let's talk about that uncertainty. Say a front-end developer at
Sniffable decides to change how timestamps are formatted on the
mobile app.
Speaker 2 (13:29):
Oh boy.
Speaker 1 (13:30):
They silently move from a standard Unix integer to an
ISO string, and they just don't tell the data team.
Speaker 2 (13:36):
A classic breakdown in communication. And if your database simply
accepts that new string and overwrites the old format, your
downstream machine learning model, which is mathematically expecting an integer,
will suddenly panic and crash.
Speaker 1 (13:49):
So how does a reactive system handle the reality of
bad or changing data?
Speaker 2 (13:55):
You have to rethink storage completely. In a reactive system,
you do not overwrite data. Ever. You record immutable facts.
Speaker 1 (14:02):
Immutable facts. Yes.
Speaker 2 (14:04):
When a payload of data comes in from the real world,
no matter how messy or weird it is, you save
the raw, exact payload. You write that complex truth in stone,
you append it to a log, and you never, ever
modify it.
Speaker 1 (14:15):
So you're just capturing history exactly as it happened.
Speaker 2 (14:18):
Right. and you let the machine learning model figure out
how to parse that history later. This leads to a
fascinating prediction strategy called possible worlds.
Speaker 1 (14:25):
That sounds incredibly theoretical.
Speaker 2 (14:27):
I know, but it's actually highly practical. In machine learning,
you rarely have one definitive truth about the future.
Speaker 1 (14:34):
Okay.
Speaker 2 (14:34):
Say Sniffable wants to allocate server capacity for the next hour,
and they need to know how many likes French Bulldogs
are going to get. You don't calculate one rigid number.
You calculate multiple realities.
Speaker 1 (14:44):
Walk me through the math on that.
Speaker 2 (14:46):
Okay, so... World A is the historical baseline. Based on
past behavior, we expect 6,500 pup dates.
Speaker 1 (14:52):
Right.
Speaker 2 (14:53):
We multiply that by our historical average of 23 likes
per post. That gives us an expectation of 149,500 likes.
Speaker 1 (15:01):
Straightforward enough.
Speaker 2 (15:02):
But World B is the ML-predicted reality. Our model is
analyzing real-time trending hashtags and projects, a massive surge, let's
say 7,250 pup dates. Wow. Okay. And because Frenchies are
currently viral... It predicts 28 likes each. That gives us 203,300 likes.
Speaker 1 (15:19):
So two very different numbers.
Speaker 2 (15:21):
Exactly. By querying across these different possible worlds simultaneously, the
system understands the range of uncertainty and spins up enough
server capacity to handle the absolute worst-case scenario.
Speaker 1 (15:31):
That's brilliant. I want to raise the stakes here with
an example from the book that perfectly illustrates the danger
of scale. Forget standard social media posting. Let's talk about
Hollywood Star.
Speaker 2 (15:41):
Oh, right. Their reality singing competition for dogs.
Speaker 1 (15:44):
Yes. Dogs barking to music on live television being judged.
And millions of viewers at home are voting on the
sniffable app at the exact same millisecond. I mean, the
traffic spikes are instantaneous and massive.
Speaker 2 (15:58):
This introduces a terrifying concept in distributed systems called tail latency.
Speaker 1 (16:03):
Tail latency. Yeah.
Speaker 2 (16:05):
When millions of users are voting... A single vote doesn't
just hit one server. It might touch dozens of microservices.
Speaker 1 (16:11):
Like authentication, fraud detection, the vote tally database.
Speaker 2 (16:15):
Exactly.
Speaker 1 (16:16):
And the wait time for the user isn't the average
speed of those services combined.
Speaker 2 (16:20):
No. The total wait time is entirely dictated by the
single slowest request in the chain. That is the tail latency. Well, man.
If you have 99 microservices operating in one millisecond, but
one remote server handling fraud detection hangs for five seconds.
Speaker 1 (16:35):
The user's mobile app freezes for five full seconds. Exactly.
This raises an important question, though. How do you physically
prevent one struggling backend server from completely freezing the user
interface for millions of people?
Speaker 2 (16:49):
This is where functional programming languages like Scala really earn
their keep. Scala has built in primitives to handle this asynchronously. First,
it uses something called futures.
Speaker 1 (17:00):
Futures. Right.
Speaker 2 (17:01):
Instead of the main application thread stopping and waiting for
that fraud server to respond, which is called blocking IO,
it fires off a future.
Speaker 1 (17:09):
So it essentially delegates the task. It says, you know,
go get this data on a background thread. Let me
know when you have it, but I'm going to keep
processing other things.
Speaker 2 (17:16):
Exactly. The main thread stays completely snappy. But Scala pairs
futures with something even more powerful called option types.
Speaker 1 (17:24):
OK, tell me about those.
Speaker 2 (17:25):
When the app asks the database for the vote count
for a dog named NomNum, It does not just get
an integer back. It gets an option.
Speaker 1 (17:33):
Explain how an option type mechanically differs from a regular variable.
Because this is the key to preventing crashes, right?
Speaker 2 (17:40):
It is. An option type acts like a highly secure
locked box. That box can only contain one of two things.
It either contains some value, like the number 42, or
it contains none, meaning the data is missing or just
hasn't arrived yet. Okay. The magic here is that the
compiler will physically refuse to compile your code unless you
(18:01):
use pattern matching to explicitly write logic for both scenarios.
Speaker 1 (18:06):
So it forces the programmer to handle the uncertainty up front.
It physically prevents a null pointer exception because you can
never accidentally call a variable that doesn't exist.
Speaker 2 (18:16):
Exactly. You have to tell the system, what do we
display if the database says sum 42, and what do
we display if the database returns? None, because the network
is congested.
Speaker 1 (18:26):
So what does this all mean for us? To me,
it means we have to actively design for failure. It's
like a world-class restaurant. If the kitchen runs out of
the signature dish, the head chef doesn't run to the
front door, lock it, shut off the power, and scream
in panic.
Speaker 2 (18:39):
Which is the architectural equivalent of what Sniffable did, honestly.
Speaker 1 (18:42):
Right. Instead, the waiter gracefully comes to your table, apologizes,
and offers you a complimentary appetizer.
Speaker 2 (18:48):
It is called graceful degradation.
Speaker 1 (18:51):
Exactly. In our Hollywood Star app, because we are using
futures and option types, we can program a strict timeout.
We can say, if that remote database takes longer than
500 milliseconds to fetch the live vote count, abandon the request.
Speaker 2 (19:05):
Just abandon it.
Speaker 1 (19:06):
Yeah. Just return none and use pattern matching to display
the historical average to the user's screen instead.
Speaker 2 (19:12):
And is displaying the historical average perfectly mathematically accurate in
that exact millisecond?
Speaker 1 (19:18):
No.
Speaker 2 (19:19):
But the app stays incredibly fast, the main thread doesn't
lock up, and the user remains completely engaged.
Speaker 1 (19:26):
Because the eventual consistency of the database catches up a
second later anyway.
Speaker 2 (19:30):
Precisely.
Speaker 1 (19:31):
It is a brilliant shift in perspective. So let's look
at the journey we've taken today. We moved away from
the naive, firefighting, duct-taped architecture of the original Pooch predictor.
Speaker 2 (19:41):
We really did.
Speaker 1 (19:42):
We explored how to build a robust, reactive machine learning system.
We learned how to use pure functions and lazy evaluation
to handle infinite difficulties. data streams without running out of memory.
We learned how to store immutable facts to honestly capture
reality and how to use futures and option types to
force that graceful degradation.
Speaker 2 (20:00):
And if there is one overarching takeaway for you today,
whether you are building, managing or even just interacting with
these massive machine learning systems. It is realizing that machine
learning is an application.
Speaker 1 (20:12):
It really is.
Speaker 2 (20:13):
It is an entire ecosystem of data pipelines, containment strategies,
and supervisors. It is not just an isolated algorithm. And
when you treat it like a holistic application, that is
your shortcut. to creating software that actually survives contact with
the real world.
Speaker 1 (20:30):
Which leaves us with a final, slightly provocative thought for
you to mull over. We have spent this entire deep
dive establishing that the most resilient machine learning systems are
fundamentally built on accepting uncertainty. Right. They are explicitly designed
to handle maybe, you know, to process multiple possible worlds
and to degrade gracefully when they don't know the exact truth.
(20:50):
But think about the world these models are being deployed into.
Our legal systems, our compliance frameworks, and our regulatory laws
demand absolute black and white deterministic certainty.
Speaker 2 (21:02):
They absolutely do.
Speaker 1 (21:03):
So it makes you wonder, when a reactive system is
perfectly comfortable with intentionally being a little bit wrong to survive,
how long will it take for our rigid legal structures
to catch up to the reality of how this software
actually works?
Speaker 2 (21:17):
That's a great question.
Speaker 1 (21:18):
Because it really is the only way that delicate plant
survives the hurricane. Thanks for taking a deep dive with us.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Bobby Bones Show
Listen to 'The Bobby Bones Show' by downloading the daily full replay.