October 1, 2025 • 29 mins
Episode Sponsor: Attribute - https://dev0ps.fyi/attribute
We're joined by 20 year industry veteran and DevOps advocate, Adam Korga, celebrating the release of his book IT Dictionary. In this episode we quickly get down to the inspiration behind postmortems as we review some cornerstone cases both in software and in general technology.
Adam shares how he started in the industry, long before DevOps was a coined term, focused on making systems safer and avoiding mistakes like accidentally dropping a production database. we review the infamous incidents of accidental database deletion, by LLMs and human's alike.
And of course we touch on the quintessential postmortems in civil engineering, flight, and survivorship bias from World War II through analyzing bullet holes on returning planes.
Notable Facts
We're joined by 20 year industry veteran and DevOps advocate, Adam Korga, celebrating the release of his book IT Dictionary. In this episode we quickly get down to the inspiration behind postmortems as we review some cornerstone cases both in software and in general technology.
Adam shares how he started in the industry, long before DevOps was a coined term, focused on making systems safer and avoiding mistakes like accidentally dropping a production database. we review the infamous incidents of accidental database deletion, by LLMs and human's alike.
And of course we touch on the quintessential postmortems in civil engineering, flight, and survivorship bias from World War II through analyzing bullet holes on returning planes.
Notable Facts
- Adam's book: IT Dictionary
- Knight Capital: the 45 minute nightmare
- Work Chronicles Comic: Will my architecture work for 1 Million users?
- Warren - Cuitisan CANDL storage containers
- Adam - FUBAR
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:07):
And welcome back to Adventures in DevOps. I almost can't
believe it. Attribute has returned to sponsor the podcast for
a second week.
Speaker 2 (00:16):
In a row.
Speaker 1 (00:16):
What they are doing in the finnoffspaces genius they call
it finn Ops without Tagging and it's the first runtime
technology that analyzes your infrastructure instead of relying on billing reports,
exports and tagging. It's for architecture ops and platform teams
that need visibility into product customer attribution or spend or
insight into cost anomalies without wasting hours. They capture the
(00:37):
cost based on actual application usage generated from Kubernetes, databases,
storage and over thirty five multi cloud services. And they
even break it down by micro service all the way
to the database query level. And I don't think I've
ever seen another tool that does quite that. So you
want to check them out. That's Attribute and I'll drop
a link in the episode description And now back to
the show and our guest today. They actually may have
(01:00):
a lot to say about that currently, em of a
platform engineering at an undisclosed large company. The interesting thing
about how long you've been in DevOps is you probably
started with the mindset long before it was even coined
as a term, and I like to introduce Adam Korga.
Speaker 3 (01:15):
So, like you say, I did actually start in this
before the develops, before became a term for full disclosure,
not all the time as develops. So I started as
a beacond engineer. Then at some point I figured out
that it was called release.
Speaker 4 (01:28):
Engineering back then, and it wasn't even on virtual machines
I experienced staff.
Speaker 1 (01:33):
When did you figure out what you were doing was
actually embodying the devlops mindset and not just doing what
someone else told you to do.
Speaker 3 (01:41):
I always wanted to do this safer or better though,
That was one thing. And second, how we can avoid
mistakes like accidental dropping of production database spoiler alert.
Speaker 4 (01:55):
I did that.
Speaker 1 (01:56):
First I came to mind was the incident that one
of the quote unquote AI providers had where they dropped
databases programmatically through their LM accidentally of their customers because
the LM decided to go and do this. So I
think we come full circle inexperienced engineers who didn't have
a lot of infrastructure experience accidentally dropping databases. I think
(02:18):
in twenty sixteen was the major event from one of
the well known GIT servers and now we're two having lms.
Speaker 3 (02:25):
Do it illustrate the famous joke about there are two
kinds of people that people who do backups and people
who will do backups issue work in develops, you know
that there is also a third category, people who test
if their backups are working.
Speaker 1 (02:41):
Thinking about get lab where they had an issue with
their QL database and they I think it was postgrass,
it may have been my sequel, and they accidentally dropped
the production database instead of the replicated one because replicating
what replication was failing.
Speaker 3 (02:55):
And this was like, that's a different story, because I
thought about the GitHub situation where they ran RAF by accident.
They had five backups. One wasn't working for a couple months,
second was on the same servers, was fine to delete
as well, and so on and so on and so on.
It was fun day, well, I say funniest probably for
(03:18):
people involved, it wasn't so fun. Pixar almost bankrupted because
of m REF because they during production of Toy Story two,
they accidentally deleted entire movie. Entire movie was saved because
one of the director was on the due to maternity leave,
(03:39):
was working often from home, and she had the files
on her private computer which was violating of course the
processes in the company, but it was the only survived
copy after this incident, so they moved her own home
PC to the office to restore the movie.
Speaker 4 (03:59):
And that's why we have studied tool.
Speaker 1 (04:01):
Actually, that's not the first time I've heard of that
saving move. There was a while ago, I think it
was almost fifteen years ago now where there was an
issue with I believe KDE like the UI for Linux distribution,
where there was some corruption that got into I think
this was like before they were using get got into
the source control and because of how they were replicating
(04:21):
all of it to all the mirrors where all the
source was, there was like automatic process to clone from
the source. Like as soon as the corruption got in there,
every single copy of it in existence also got that
was cloned and got that corruption in it, and they
were all worthless. And so they had to go around
like individual engineers who like hadn't set up or specifically
disabled the automatic replication or to rebuild the source code
(04:44):
index there. Now, I really want to talk about post
mortems with you and like how they actually reviewed the situation.
But before we get to that, one of the reasons
that we grabbed you for this episode. Is you released
quite the satirical book, The Guide to the Industry IT Dictionary,
So this saying is absolutely ridiculous. This book, I mean
it really is a dictionary. You've got a bunch of
(05:05):
i'll say euphemisms with their implied real real world interpretation
of them, and it just goes on and on. For
these I feel like, you know, I had to pick
it up and you know, read something then put it down.
Speaker 3 (05:18):
Like every single entry is separate punchline, so it could
be repetitive and exhausting if you try to book read
it as a typical linear book. I don't recommend it.
I won't judge where you hold this book. It could
be on your desk, it could be in other places.
And find with that as long as you have found
it good. But there is second layer behind that, because
(05:42):
it is divided into five parts and each describes different
areas like quoriti agil, rituals and corporations, startups.
Speaker 4 (05:53):
And of course we have to mention AI nowadays.
Speaker 1 (05:57):
There is something that reminds me of random monrose X Casey,
Like if I just pull out some of the jokes,
it's like, I mean, what he takes three or four
panels to get across. You have like a single core
message that wraps it up, like just as an example
one like ticket types. One of your ticket types is
like quick bug, easy to reproduce, and then like if
you the actual translation is two weeks later your knee
(06:19):
deep in system calls questioning your life choices. I think
a lot of us have probably seen seen ourselves there
where you trying to fix something. It's usually on our
own computers, like not even from the company, And the
longer you spend trying to fix the problem, the worst
the issue gets.
Speaker 4 (06:35):
Well.
Speaker 3 (06:35):
I tries to really show how it looks from real life.
Sometimes I'm going also on the typical jargon that doesn't
necessarily touch the it or the core it itself.
Speaker 1 (06:47):
I mean, we should go through one of the sections.
I feel like that would be the most interesting. So
should we call it the agile one?
Speaker 4 (06:52):
I don't know.
Speaker 3 (06:52):
I have to admit that introductions in agile part are
my favorite one.
Speaker 1 (06:59):
So there's like this something for everyone here, no matter
what your flavor of engineering is that you're working in, or.
Speaker 2 (07:04):
Even if you're outside the field.
Speaker 1 (07:06):
I see a lot of companies saying they do agile
and not doing it. This one always it's quite close
to home for me, so I just I want to
read some of these out because I just think I
think they're quite funny. So there's like the term we're
doing agile, and what you have here it actually means
is we have a daily panic meeting and call it
a methodology. And then there's servant leadership booking meetings and
(07:31):
politely begging people to update JIRA. I mean obviously because
you don't no longer have the authority to tell people
what to do and you're just there to support them,
but they're not performing the necessary activities. Anyway, Let's see
if there's another particularly good one, shielding the team sitting
in meetings they weren't invited to either.
Speaker 3 (07:48):
There is even an entire section of a subsection about
the tools in agile, and my favorite there was Slack,
which is destruction.
Speaker 4 (07:58):
As I said, there are.
Speaker 1 (07:59):
Some actually good for organizations where understanding what the tempo
of the organization is by the Slack messages that you're
getting and being attentive to problems that show up and
responding to the team is incredibly valuable. But I find
that book really conveys here is that a lot of
times the notifications represent just total noise.
Speaker 3 (08:19):
Yeah, but that applies to everything. That's why the entire
part is called cargo cult. Cargo cult is actual term
from World War Two which evolved into mimicking rituals without
understanding reasons. And it's not like slack, gira, confluence or
whatever is actually wrong.
Speaker 4 (08:38):
It's not.
Speaker 3 (08:40):
It's just about people use it or or abuse it
hoping that it will solve their problems.
Speaker 1 (08:45):
I think this is actually really interesting to call out
because the origins actually were quite interesting for me. So
you're right, I believe it was Asia, and I mean realistically,
just the shortcut is care packages and goods were dropped
from like fighter jets on the ground, right, And so
of course with that there's like a lot of other
technology stack that you need to create there. And after
(09:07):
World War Two and the Allies left and stop dropping
care packages, they still wanted them to show up. So
if you see that a box of peers, if there
are landing strips and airplanes and control towers and those
are all gone now, well then you build the control
towers and the airplanes and you hope that the care
(09:28):
packages show up with it. Because you know, as you said,
and I think is really important that you see what
the practice is and you just replicate it and without
understanding what the reason was that was created or the
goal in mind, then you're really just doing something which
doesn't actually bring any value. And I think we see
this through the industry a lot, and it gets replicated
(09:48):
over and over one and I think I'm going to
get a lot of angry emails.
Speaker 2 (09:51):
As a result of this. I think Kubernetes is a
great one.
Speaker 1 (09:54):
Like you see companies building the architecture and platform to
support such a complicated technology and then they throw it
in and I think they hope that they're going to
be successful after that.
Speaker 5 (10:05):
Well, that's actually what picking up up.
Speaker 3 (10:08):
In startup, the product is not ready if it's not scalable,
and as a result, companies invest crazy amount of energy
and money into scalability micro service architecture, ensuring that it
would scale two thousands, despite the fact active user counts
is twelve including QA.
Speaker 1 (10:28):
Yeah, for sure, I think there's this there's this commat
that I'll end up linking after the episode for anyone
that's on our website Adventures and Develops, where the canonical
is some engineer goes to the platform architect, will this
infrastructure support millions of users and requests per second? And
(10:49):
the answer is another question, how many you have right now?
Speaker 4 (10:52):
Oh?
Speaker 2 (10:53):
We have five?
Speaker 1 (10:54):
Yes, that is the perfect architecture for what you're doing.
Speaker 3 (10:57):
For sure, I have this problem. I had to learn
to avoid this or notice this problem. But we tend
to go with the best practices.
Speaker 1 (11:07):
And I do want to ask what brought you to
the point where you felt like you had to get
this down in writing that you wanted to write the
book in the first place.
Speaker 3 (11:16):
Some time ago, I was dating some girl that was
working in legal department in IT company, and as a
small gift slash joke, I decided I offered to give
her a translation of terms that people in IT used
to say that others are stupid, without saying that openly,
(11:36):
like classics like Layer eight issue. Then I added some
translations of terms or phrases that people tend to use
to sound smart besides while saying that they do not
that they have no idea what's going on. And over
time it started growing I realized that this also could
be described.
Speaker 4 (11:56):
This could also be described.
Speaker 5 (11:58):
I continued adding, and when I when my.
Speaker 3 (12:05):
Table of contents, which was pretty much just a draft
of topics that I'd like to translate. Once I had that,
I looked at it and I found this underlying structure
that evolved into those parts. Somebody asked me if I
was looking at him from behind his back at his desk.
So it's leased and experienced, but I tried to make
(12:26):
it accessible. Nevertheless, once you start writing, you realize that, well,
I realized for sure probably others would have the same
as effect that there's more to say, there's more there,
and well, I we already started talking about the topic
of the what's come next?
Speaker 4 (12:44):
What comes next?
Speaker 3 (12:45):
Because I already started writing the next book, which would
be very friendly and not so easy to promote, the
title Fakapalmanac, which will be literally collection of disasters in
it and civil engineering. The focus on post mortem still
very cynical in most cases.
Speaker 1 (13:03):
I'm sure you got the forty five minute catastrophe of
Night Capital in there.
Speaker 4 (13:08):
Of course, that's.
Speaker 1 (13:09):
The cornerstone of post mortems. That's the one that you
see and you're like, you know what, I'm going to
keep a link to that.
Speaker 3 (13:14):
The Night Capital was when they wanted to automate trading
and they deployed new version. They deployed it on every
all the systems, except they didn't update one server that
still had the test mode enabled, which means that it
was creating enormous amount of trades with catastrophical consequences. And
(13:41):
I'm not sure, but they low in forty five minutes.
They lost something like eight hundred million dollars or something
like that. But there was also an interesting story, and
I think it was about Italium, and that's something I
don't remember in details, but basically there was a bag
in the contracts implementation that allowed with frequent enough trading,
(14:03):
you could get paid and before this was deducted from
the other account, return it.
Speaker 5 (14:10):
So if you did it properly, would you were.
Speaker 4 (14:13):
Pretty much creating this from scratch.
Speaker 1 (14:15):
You're talking about the forking hard fork of the Ethereum blockchain.
Before it was what it is today, it was Ethereum Classic.
There was this idea of programmatic governance of organizations using
a smart contracts, so unlike Bitcoin, Ethereum supports programmatic execution,
and someone abused that contract. Basically, the order of operations
in the contract didn't ensure that someone wasn't withdrawing more
(14:38):
money than another contract allowed, and you're abs aut, right.
Speaker 2 (14:42):
I mean, this is the thing with blockchain though.
Speaker 1 (14:43):
I think the post mortem here is in a way
like why did it happen? But also the path forward,
So in some way the value was what can we
do about it? And the thing with blockchain technologies, with
the public ledger is out of enough people decide consensus
right that what happened isn't okay and it should be different.
Then they go and they make it different, and that's
(15:03):
how we ended up with ethereum that.
Speaker 2 (15:06):
We have today. Actually it's not even the ethereum we have.
Speaker 1 (15:08):
Today because that was still when the consensus algorithm was
based off of proof of work, and now we have
ethereum two point zero, which is based off a proof
of steak which analyzes people with monetary punishment. So there's
been evolution here. So in some way, I guess my
question is going to be postmortems have value then, right?
Speaker 3 (15:28):
Yes, huge, at least my recent experience that that's another
area where those cargo cules appear. That, for instance, quite
common root cause analysis technique, which is five whys.
Speaker 5 (15:41):
I've seen multiple times when people were so.
Speaker 4 (15:44):
Stuck on these five whys that as.
Speaker 3 (15:47):
A result, some people were artificially bloating the reasoning and
suddenly one single answer was split into three points just
to hit the magic five at the end, or in
other cases, somebody hit five and stopped analysis right when
the analysis was starting to be interesting.
Speaker 1 (16:10):
Why do you think that a lot of root cause
analysis or postmodern meetings that happen after major events don't
push through that investigative phase and are able to get
to what the real insight is.
Speaker 2 (16:21):
Why are they stopping right before that threshold?
Speaker 5 (16:23):
If I had to ask, I would say there are
two reasons.
Speaker 3 (16:26):
One is, as soon as I can throw the ball
outside of my garden, it's enough, changing the purpose from
finding what happened and how we can avoid to proving
that it's not my fault.
Speaker 5 (16:40):
Second is just a typical lack of time.
Speaker 1 (16:44):
I do think we see the pendulum swaying back and
forth a lot in technology. And I don't know if
this is just my experience or technology itself, or because
I have a limited view and it happens other places
as well, but it does really seem like a lot
of organizations are somewhere they have a certain mindset, and
when things don't work right their strategy, and almost seems
(17:07):
like their first and only strategy is do the extreme
opposite of whatever that thing is. And then five years
later the organizations in a different place there and they
see problems with that extreme, and so then they swing
back to the other way, which by that point everyone
who had left the first extreme is now gone and
you're only left with new people who have the pain
of the decision that was from the past and swinging
(17:30):
the pendulum back and now are doing basically the wrong
thing again or something else incredibly wrong in a different way.
Speaker 2 (17:36):
I think you're really on point though there with.
Speaker 4 (17:39):
The it's not just in technology, the pendulument tendency to
extremes is just in human nature.
Speaker 1 (17:45):
I think you're absolutely right about why companies are stopping
their post mortem process too early. It goes back into
the fact that the incentives are aligned for short term
deliverables or focus, and I don't see any reason why
this wouldn't just continue snowball and causes bigger problems. So
maybe you know in your in your research that you've
(18:05):
done here, has there been any callouts for identifying how
companies or organizations can stay methodical and realize they need
to complete the process optimized for long term value opposed
to falling back on just delivering a short term.
Speaker 3 (18:19):
You can only say that it is a lot about
common sense. One example of value of post mortem is
how why airplanes have over circle windows? So it's not
it wasn't always like that. So basically, in pre World
War two airplanes had squares typical square rectangular windows and
(18:43):
it was fine. The windows were sometimes the glasses or
glass was sometimes crashing, but the planes were were not
flying that high. It had didn't have to be uh
pressurized and stuff. So when the planes that it was okay,
called the glacier, changed the window and move on, and
(19:05):
nobody investigated after World War two. Each actually coincidentally, it's
not that World War to change something. But in nineteen
fifties the first hypersonic airplane came to civil aviation.
Speaker 4 (19:20):
It was heavyland Comet, which.
Speaker 3 (19:24):
Was flying at the altitude of eleven or twelve kilometers,
and there the cabins had to be pressurized, so the
forces were much heavier and those tiny cracks that were
visible before started to be much more critical, and within
four or five years there were multiple catastrophes with casualties
(19:48):
and stuff, And that's when they did the full blown investigation.
We built special tanks to investigate, to analyze the pressures
and stuff.
Speaker 4 (20:00):
And then five and they.
Speaker 3 (20:01):
Found out that on the corners corners were pretty much
concentration points for the stress. The stress levels were even
for over four times, something like four times higher than
around the not on the corner. So and that was
around the corner of the windows and the antenna radio
(20:23):
antennas and stuff mountaints. So the solution was brilliantly simple.
If the corners have the problems, eliminate corners. That's why
the windows became round. If they investigation, the signs were
before that, but they weren't leading to.
Speaker 4 (20:40):
Catastrophes yet, so they were ignored.
Speaker 3 (20:43):
If you look at this from the IT perspective, it's
kind of the same. As long as you have an
issue when you deploy the new version and something crashes,
it's enough to restart pipeline or something like that. Okay,
let's not investigate, we need to focus, but that could
be an underlying issue that later could lead to the
(21:04):
catastrophe like night copy.
Speaker 1 (21:06):
First of all, the airline industry is quite the shining
example of diving into post mortems. I feel like, compared
to other industries, the idea of preventing not just loss
of life, but preventing all incidents now, current and future
is something that's really focused on and like really getting
to understand, like all the root causes, not just the
(21:28):
first one that pops up, so that these events in
any way can never happen again. And I feel like
World War Two in a lot of ways was a
turning point. I think there was this very specific interesting
conclusion insight that was had, and I believe it was
in the United States with the return of vehicles that
were damaged in some way.
Speaker 4 (21:46):
It was in US. You are talking about survivors.
Speaker 1 (21:49):
Bias, yes, yes, of course, but.
Speaker 3 (21:52):
The conclusion was from I think Hungarian statistician.
Speaker 1 (21:57):
Basically, the idea was that if you receive vehicle back,
then it survived its.
Speaker 3 (22:02):
R US Army was analyzing the bullet holes on the
returning bombers and their initial intuition was that if that's
where the planes were hit, they need to fortify those points.
And just this Hungarian mathematic statistician pointed out that hey, stop,
(22:22):
if you see those holes, you don't see the holes
on other parts.
Speaker 5 (22:26):
That means that the planes that were hit there did
not return.
Speaker 3 (22:30):
And that's also a very strong case for post mortems
because its very easy to focus on the success stories,
but the real learning, real they tell lies in the disaster.
Speaker 1 (22:41):
It's very counterintuitive. I think it's the conclusion that a
lot of times we come to here, and that it's
almost like if you aren't coming to a counterintuitive conclusion,
then maybe you're missing something critical about your post mortem process.
And I think this is where the survivorship is come
doesn't play you. You have a release or an incident,
and you sit down and a meeting and you discuss things,
and the conclusion invariably is like, we need more tests,
(23:05):
and well, anyone could have told you to have more tests.
Speaker 2 (23:08):
I think the clever trick.
Speaker 1 (23:10):
Is knowing which tests you should have, because you know,
if you just had the right test, you can avoid
every single production incident that would ever happen.
Speaker 4 (23:20):
For that.
Speaker 3 (23:20):
I kind of like that approach of do not write
more tests just because you want more tests?
Speaker 1 (23:27):
That conclusion is often a struggle for organizations, like there's
a belief that oh no, this this software doesn't change
that frequently, or we don't have the priority to actually
be able to implement there. But yeah, I mean, if
you're in the weeds and actually fixing something or changing
the way something works, I think a good metric that
at least my teams use is add the test before you.
Speaker 2 (23:48):
Make the change.
Speaker 1 (23:49):
Like, it doesn't matter if there are necessarily tests on code,
but if you're going to change something, then you can
have an opportunity to potentially add the test first to
ensure that the thing you're changing doesn't break in a
particular way. And I think the follow up here is
that often things break for the new functionality.
Speaker 3 (24:05):
But generally the most box most mistakes were when you
introduce the change.
Speaker 2 (24:11):
Yeah, yeah, for sure.
Speaker 1 (24:11):
Realistically here there's like a whole bunch of failure modes
where you're making assumptions about what the current thing is
doing rather than actually validating it in some capacity. Here,
there is one thing that I think is still worth
bringing up in the area of post mortems, especially when
it comes to the failure rate, and it's this idea
where you it's it is actually impossible to predict where
(24:36):
the future errors will will show up in some capacity,
and the mistake that's made frequently is tracking the number
of errors you have or the size of them in
the path, because nothing tells us that the errors that
we've encountered so far and potentially fixed or prevented, have
any impact on the future errors that we're going to
encounter in our organization. The criticality of the failure. Every
(24:58):
company works until the moment it doesn't. Every software works
until the moment it doesn't. And I think there is
this idea where, oh, yeah, we can just fix every
problem we've identified. We should tract the number of problems
over time and will just be good. And I think
I'm hoping the history and the research that you've done
leads leads, you know, anyone who reads the book to
(25:18):
the opposite conclusion, which is like you need a better
process or a mindset around how you're tackling what you're
tackling rather than just what potentially comes up.
Speaker 3 (25:27):
There was a technique of city editors. Microsoft was doing
this with Microsoft Betwindows, but it was totally for a
different purpose. But it was a technique to statistically evaluate
how many bugs they didn't find. Quite interesting, not necessarily
applicable to constantly evolving software nowadays. But the main lesson
(25:47):
is to not ignore the errors. On one hand, own
mistakes to try to analyze it. Second, don't ignore other
people's editors. That's free learning. Sometimes you can last, sometimes
you can be amazed, but for sure there is some
lessons there that you can avoid.
Speaker 1 (26:07):
Yeah, it's a very famous Russian submarine that I don't
recall that.
Speaker 3 (26:11):
That's what I was aiming at. It was Staniswa Pietrov
who was in Soviet air defense.
Speaker 4 (26:18):
And indeed they.
Speaker 3 (26:19):
Had one censorus reporting that US launched a missile, and
he ignored it until they got back up information from
different stations a couple of minutes later. According to Protesto.
To Proteto, he should be he should immediately counterfire. He
ignored it, and that that was nineteen eighty three. He
(26:42):
ignored that, and that's why the World War two did
not start. If he shot back, we wouldn't be talking
right now.
Speaker 2 (26:49):
For sure.
Speaker 4 (26:49):
Not.
Speaker 2 (26:50):
I think this could be a good stopping point.
Speaker 1 (26:52):
Let's move on to picks. My pick for today is
the quitos quetason. I don't know how to pronounce it.
Food stories containers. I really like these.
Speaker 2 (27:02):
They're really expensive, but they're highly durable. They go in
the oven, I don't know a microwave. They go on
the stove.
Speaker 1 (27:09):
I guess they go in the freezer deep freeze minus
twenty degrees celsius.
Speaker 2 (27:13):
They are absolutely fantastic.
Speaker 1 (27:15):
And what I like about this brand is they have
a lot of different sizes, so you can measure exactly
what fits in your refrigerators and your storage units and
focus on stacking them the best. I probably have like
fifteen or twenty of these, like almost this exact size.
Speaker 2 (27:28):
And honestly, they've been great about it.
Speaker 1 (27:30):
Five years ago I completely moved off of storing anything
in plastics in any way, and now everything I wanted
is in glass or stainless steel metal. Actually this is
a composite and these have been the best so far.
And I really like them because after cooking, I always
have leftovers.
Speaker 3 (27:48):
I need to check you and if you ask me what,
I would say, well, I recently watched the TV series
on Netflix called Fubar. It's u pical, well modern take
on nineteen nineties early two thousand action comedy movies with
(28:08):
Arnold Schwarzenegger, and it's super full of fun service jokes,
even quoting making fun of the most famous scenes with
Swarzenegger and his best movies. So if you grew up
in this time. It's very much enjoyable.
Speaker 1 (28:28):
I think I think you has actually been on my
watch list for a while, and I haven't. I haven't
gotten around here.
Speaker 3 (28:33):
It's really really good with really solid screenplay. Of course,
it's spy spy movies, so full of physical absorts, but
in the rules of the presented world, it stands on
its legs and you can really enjoy it.
Speaker 1 (28:54):
Okay, Well, then I guess that's going on someone's watch
lest that will be an episode notes. So at this
point I will just say thank you Adam for for
coming on to the to the show and talking about
your your book and your future upcoming book. I feel
like the postmortems will be really interesting and maybe we'll
get you back on it.
Speaker 3 (29:11):
At that point, well, thank you for having me, thank
you for a very nice conversation, and I would be
super happy to be back.
Speaker 1 (29:18):
And with that, I'll say thank you to Attribute one
more time for sponsoring today's episode, Thanks for everyone listening,
and we'll be back next
And welcome back to Adventures in DevOps. I almost can't
believe it. Attribute has returned to sponsor the podcast for
a second week.
Speaker 2 (00:16):
In a row.
Speaker 1 (00:16):
What they are doing in the finnoffspaces genius they call
it finn Ops without Tagging and it's the first runtime
technology that analyzes your infrastructure instead of relying on billing reports,
exports and tagging. It's for architecture ops and platform teams
that need visibility into product customer attribution or spend or
insight into cost anomalies without wasting hours. They capture the
(00:37):
cost based on actual application usage generated from Kubernetes, databases,
storage and over thirty five multi cloud services. And they
even break it down by micro service all the way
to the database query level. And I don't think I've
ever seen another tool that does quite that. So you
want to check them out. That's Attribute and I'll drop
a link in the episode description And now back to
the show and our guest today. They actually may have
(01:00):
a lot to say about that currently, em of a
platform engineering at an undisclosed large company. The interesting thing
about how long you've been in DevOps is you probably
started with the mindset long before it was even coined
as a term, and I like to introduce Adam Korga.
Speaker 3 (01:15):
So, like you say, I did actually start in this
before the develops, before became a term for full disclosure,
not all the time as develops. So I started as
a beacond engineer. Then at some point I figured out
that it was called release.
Speaker 4 (01:28):
Engineering back then, and it wasn't even on virtual machines
I experienced staff.
Speaker 1 (01:33):
When did you figure out what you were doing was
actually embodying the devlops mindset and not just doing what
someone else told you to do.
Speaker 3 (01:41):
I always wanted to do this safer or better though,
That was one thing. And second, how we can avoid
mistakes like accidental dropping of production database spoiler alert.
Speaker 4 (01:55):
I did that.
Speaker 1 (01:56):
First I came to mind was the incident that one
of the quote unquote AI providers had where they dropped
databases programmatically through their LM accidentally of their customers because
the LM decided to go and do this. So I
think we come full circle inexperienced engineers who didn't have
a lot of infrastructure experience accidentally dropping databases. I think
(02:18):
in twenty sixteen was the major event from one of
the well known GIT servers and now we're two having lms.
Speaker 3 (02:25):
Do it illustrate the famous joke about there are two
kinds of people that people who do backups and people
who will do backups issue work in develops, you know
that there is also a third category, people who test
if their backups are working.
Speaker 1 (02:41):
Thinking about get lab where they had an issue with
their QL database and they I think it was postgrass,
it may have been my sequel, and they accidentally dropped
the production database instead of the replicated one because replicating
what replication was failing.
Speaker 3 (02:55):
And this was like, that's a different story, because I
thought about the GitHub situation where they ran RAF by accident.
They had five backups. One wasn't working for a couple months,
second was on the same servers, was fine to delete
as well, and so on and so on and so on.
It was fun day, well, I say funniest probably for
(03:18):
people involved, it wasn't so fun. Pixar almost bankrupted because
of m REF because they during production of Toy Story two,
they accidentally deleted entire movie. Entire movie was saved because
one of the director was on the due to maternity leave,
(03:39):
was working often from home, and she had the files
on her private computer which was violating of course the
processes in the company, but it was the only survived
copy after this incident, so they moved her own home
PC to the office to restore the movie.
Speaker 4 (03:59):
And that's why we have studied tool.
Speaker 1 (04:01):
Actually, that's not the first time I've heard of that
saving move. There was a while ago, I think it
was almost fifteen years ago now where there was an
issue with I believe KDE like the UI for Linux distribution,
where there was some corruption that got into I think
this was like before they were using get got into
the source control and because of how they were replicating
(04:21):
all of it to all the mirrors where all the
source was, there was like automatic process to clone from
the source. Like as soon as the corruption got in there,
every single copy of it in existence also got that
was cloned and got that corruption in it, and they
were all worthless. And so they had to go around
like individual engineers who like hadn't set up or specifically
disabled the automatic replication or to rebuild the source code
(04:44):
index there. Now, I really want to talk about post
mortems with you and like how they actually reviewed the situation.
But before we get to that, one of the reasons
that we grabbed you for this episode. Is you released
quite the satirical book, The Guide to the Industry IT Dictionary,
So this saying is absolutely ridiculous. This book, I mean
it really is a dictionary. You've got a bunch of
(05:05):
i'll say euphemisms with their implied real real world interpretation
of them, and it just goes on and on. For
these I feel like, you know, I had to pick
it up and you know, read something then put it down.
Speaker 3 (05:18):
Like every single entry is separate punchline, so it could
be repetitive and exhausting if you try to book read
it as a typical linear book. I don't recommend it.
I won't judge where you hold this book. It could
be on your desk, it could be in other places.
And find with that as long as you have found
it good. But there is second layer behind that, because
(05:42):
it is divided into five parts and each describes different
areas like quoriti agil, rituals and corporations, startups.
Speaker 4 (05:53):
And of course we have to mention AI nowadays.
Speaker 1 (05:57):
There is something that reminds me of random monrose X Casey,
Like if I just pull out some of the jokes,
it's like, I mean, what he takes three or four
panels to get across. You have like a single core
message that wraps it up, like just as an example
one like ticket types. One of your ticket types is
like quick bug, easy to reproduce, and then like if
you the actual translation is two weeks later your knee
(06:19):
deep in system calls questioning your life choices. I think
a lot of us have probably seen seen ourselves there
where you trying to fix something. It's usually on our
own computers, like not even from the company, And the
longer you spend trying to fix the problem, the worst
the issue gets.
Speaker 4 (06:35):
Well.
Speaker 3 (06:35):
I tries to really show how it looks from real life.
Sometimes I'm going also on the typical jargon that doesn't
necessarily touch the it or the core it itself.
Speaker 1 (06:47):
I mean, we should go through one of the sections.
I feel like that would be the most interesting. So
should we call it the agile one?
Speaker 4 (06:52):
I don't know.
Speaker 3 (06:52):
I have to admit that introductions in agile part are
my favorite one.
Speaker 1 (06:59):
So there's like this something for everyone here, no matter
what your flavor of engineering is that you're working in, or.
Speaker 2 (07:04):
Even if you're outside the field.
Speaker 1 (07:06):
I see a lot of companies saying they do agile
and not doing it. This one always it's quite close
to home for me, so I just I want to
read some of these out because I just think I
think they're quite funny. So there's like the term we're
doing agile, and what you have here it actually means
is we have a daily panic meeting and call it
a methodology. And then there's servant leadership booking meetings and
(07:31):
politely begging people to update JIRA. I mean obviously because
you don't no longer have the authority to tell people
what to do and you're just there to support them,
but they're not performing the necessary activities. Anyway, Let's see
if there's another particularly good one, shielding the team sitting
in meetings they weren't invited to either.
Speaker 3 (07:48):
There is even an entire section of a subsection about
the tools in agile, and my favorite there was Slack,
which is destruction.
Speaker 4 (07:58):
As I said, there are.
Speaker 1 (07:59):
Some actually good for organizations where understanding what the tempo
of the organization is by the Slack messages that you're
getting and being attentive to problems that show up and
responding to the team is incredibly valuable. But I find
that book really conveys here is that a lot of
times the notifications represent just total noise.
Speaker 3 (08:19):
Yeah, but that applies to everything. That's why the entire
part is called cargo cult. Cargo cult is actual term
from World War Two which evolved into mimicking rituals without
understanding reasons. And it's not like slack, gira, confluence or
whatever is actually wrong.
Speaker 4 (08:38):
It's not.
Speaker 3 (08:40):
It's just about people use it or or abuse it
hoping that it will solve their problems.
Speaker 1 (08:45):
I think this is actually really interesting to call out
because the origins actually were quite interesting for me. So
you're right, I believe it was Asia, and I mean realistically,
just the shortcut is care packages and goods were dropped
from like fighter jets on the ground, right, And so
of course with that there's like a lot of other
technology stack that you need to create there. And after
(09:07):
World War Two and the Allies left and stop dropping
care packages, they still wanted them to show up. So
if you see that a box of peers, if there
are landing strips and airplanes and control towers and those
are all gone now, well then you build the control
towers and the airplanes and you hope that the care
(09:28):
packages show up with it. Because you know, as you said,
and I think is really important that you see what
the practice is and you just replicate it and without
understanding what the reason was that was created or the
goal in mind, then you're really just doing something which
doesn't actually bring any value. And I think we see
this through the industry a lot, and it gets replicated
(09:48):
over and over one and I think I'm going to
get a lot of angry emails.
Speaker 2 (09:51):
As a result of this. I think Kubernetes is a
great one.
Speaker 1 (09:54):
Like you see companies building the architecture and platform to
support such a complicated technology and then they throw it
in and I think they hope that they're going to
be successful after that.
Speaker 5 (10:05):
Well, that's actually what picking up up.
Speaker 3 (10:08):
In startup, the product is not ready if it's not scalable,
and as a result, companies invest crazy amount of energy
and money into scalability micro service architecture, ensuring that it
would scale two thousands, despite the fact active user counts
is twelve including QA.
Speaker 1 (10:28):
Yeah, for sure, I think there's this there's this commat
that I'll end up linking after the episode for anyone
that's on our website Adventures and Develops, where the canonical
is some engineer goes to the platform architect, will this
infrastructure support millions of users and requests per second? And
(10:49):
the answer is another question, how many you have right now?
Speaker 4 (10:52):
Oh?
Speaker 2 (10:53):
We have five?
Speaker 1 (10:54):
Yes, that is the perfect architecture for what you're doing.
Speaker 3 (10:57):
For sure, I have this problem. I had to learn
to avoid this or notice this problem. But we tend
to go with the best practices.
Speaker 1 (11:07):
And I do want to ask what brought you to
the point where you felt like you had to get
this down in writing that you wanted to write the
book in the first place.
Speaker 3 (11:16):
Some time ago, I was dating some girl that was
working in legal department in IT company, and as a
small gift slash joke, I decided I offered to give
her a translation of terms that people in IT used
to say that others are stupid, without saying that openly,
(11:36):
like classics like Layer eight issue. Then I added some
translations of terms or phrases that people tend to use
to sound smart besides while saying that they do not
that they have no idea what's going on. And over
time it started growing I realized that this also could
be described.
Speaker 4 (11:56):
This could also be described.
Speaker 5 (11:58):
I continued adding, and when I when my.
Speaker 3 (12:05):
Table of contents, which was pretty much just a draft
of topics that I'd like to translate. Once I had that,
I looked at it and I found this underlying structure
that evolved into those parts. Somebody asked me if I
was looking at him from behind his back at his desk.
So it's leased and experienced, but I tried to make
(12:26):
it accessible. Nevertheless, once you start writing, you realize that, well,
I realized for sure probably others would have the same
as effect that there's more to say, there's more there,
and well, I we already started talking about the topic
of the what's come next?
Speaker 4 (12:44):
What comes next?
Speaker 3 (12:45):
Because I already started writing the next book, which would
be very friendly and not so easy to promote, the
title Fakapalmanac, which will be literally collection of disasters in
it and civil engineering. The focus on post mortem still
very cynical in most cases.
Speaker 1 (13:03):
I'm sure you got the forty five minute catastrophe of
Night Capital in there.
Speaker 4 (13:08):
Of course, that's.
Speaker 1 (13:09):
The cornerstone of post mortems. That's the one that you
see and you're like, you know what, I'm going to
keep a link to that.
Speaker 3 (13:14):
The Night Capital was when they wanted to automate trading
and they deployed new version. They deployed it on every
all the systems, except they didn't update one server that
still had the test mode enabled, which means that it
was creating enormous amount of trades with catastrophical consequences. And
(13:41):
I'm not sure, but they low in forty five minutes.
They lost something like eight hundred million dollars or something
like that. But there was also an interesting story, and
I think it was about Italium, and that's something I
don't remember in details, but basically there was a bag
in the contracts implementation that allowed with frequent enough trading,
(14:03):
you could get paid and before this was deducted from
the other account, return it.
Speaker 5 (14:10):
So if you did it properly, would you were.
Speaker 4 (14:13):
Pretty much creating this from scratch.
Speaker 1 (14:15):
You're talking about the forking hard fork of the Ethereum blockchain.
Before it was what it is today, it was Ethereum Classic.
There was this idea of programmatic governance of organizations using
a smart contracts, so unlike Bitcoin, Ethereum supports programmatic execution,
and someone abused that contract. Basically, the order of operations
in the contract didn't ensure that someone wasn't withdrawing more
(14:38):
money than another contract allowed, and you're abs aut, right.
Speaker 2 (14:42):
I mean, this is the thing with blockchain though.
Speaker 1 (14:43):
I think the post mortem here is in a way
like why did it happen? But also the path forward,
So in some way the value was what can we
do about it? And the thing with blockchain technologies, with
the public ledger is out of enough people decide consensus
right that what happened isn't okay and it should be different.
Then they go and they make it different, and that's
(15:03):
how we ended up with ethereum that.
Speaker 2 (15:06):
We have today. Actually it's not even the ethereum we have.
Speaker 1 (15:08):
Today because that was still when the consensus algorithm was
based off of proof of work, and now we have
ethereum two point zero, which is based off a proof
of steak which analyzes people with monetary punishment. So there's
been evolution here. So in some way, I guess my
question is going to be postmortems have value then, right?
Speaker 3 (15:28):
Yes, huge, at least my recent experience that that's another
area where those cargo cules appear. That, for instance, quite
common root cause analysis technique, which is five whys.
Speaker 5 (15:41):
I've seen multiple times when people were so.
Speaker 4 (15:44):
Stuck on these five whys that as.
Speaker 3 (15:47):
A result, some people were artificially bloating the reasoning and
suddenly one single answer was split into three points just
to hit the magic five at the end, or in
other cases, somebody hit five and stopped analysis right when
the analysis was starting to be interesting.
Speaker 1 (16:10):
Why do you think that a lot of root cause
analysis or postmodern meetings that happen after major events don't
push through that investigative phase and are able to get
to what the real insight is.
Speaker 2 (16:21):
Why are they stopping right before that threshold?
Speaker 5 (16:23):
If I had to ask, I would say there are
two reasons.
Speaker 3 (16:26):
One is, as soon as I can throw the ball
outside of my garden, it's enough, changing the purpose from
finding what happened and how we can avoid to proving
that it's not my fault.
Speaker 5 (16:40):
Second is just a typical lack of time.
Speaker 1 (16:44):
I do think we see the pendulum swaying back and
forth a lot in technology. And I don't know if
this is just my experience or technology itself, or because
I have a limited view and it happens other places
as well, but it does really seem like a lot
of organizations are somewhere they have a certain mindset, and
when things don't work right their strategy, and almost seems
(17:07):
like their first and only strategy is do the extreme
opposite of whatever that thing is. And then five years
later the organizations in a different place there and they
see problems with that extreme, and so then they swing
back to the other way, which by that point everyone
who had left the first extreme is now gone and
you're only left with new people who have the pain
of the decision that was from the past and swinging
(17:30):
the pendulum back and now are doing basically the wrong
thing again or something else incredibly wrong in a different way.
Speaker 2 (17:36):
I think you're really on point though there with.
Speaker 4 (17:39):
The it's not just in technology, the pendulument tendency to
extremes is just in human nature.
Speaker 1 (17:45):
I think you're absolutely right about why companies are stopping
their post mortem process too early. It goes back into
the fact that the incentives are aligned for short term
deliverables or focus, and I don't see any reason why
this wouldn't just continue snowball and causes bigger problems. So
maybe you know in your in your research that you've
(18:05):
done here, has there been any callouts for identifying how
companies or organizations can stay methodical and realize they need
to complete the process optimized for long term value opposed
to falling back on just delivering a short term.
Speaker 3 (18:19):
You can only say that it is a lot about
common sense. One example of value of post mortem is
how why airplanes have over circle windows? So it's not
it wasn't always like that. So basically, in pre World
War two airplanes had squares typical square rectangular windows and
(18:43):
it was fine. The windows were sometimes the glasses or
glass was sometimes crashing, but the planes were were not
flying that high. It had didn't have to be uh
pressurized and stuff. So when the planes that it was okay,
called the glacier, changed the window and move on, and
(19:05):
nobody investigated after World War two. Each actually coincidentally, it's
not that World War to change something. But in nineteen
fifties the first hypersonic airplane came to civil aviation.
Speaker 4 (19:20):
It was heavyland Comet, which.
Speaker 3 (19:24):
Was flying at the altitude of eleven or twelve kilometers,
and there the cabins had to be pressurized, so the
forces were much heavier and those tiny cracks that were
visible before started to be much more critical, and within
four or five years there were multiple catastrophes with casualties
(19:48):
and stuff, And that's when they did the full blown investigation.
We built special tanks to investigate, to analyze the pressures
and stuff.
Speaker 4 (20:00):
And then five and they.
Speaker 3 (20:01):
Found out that on the corners corners were pretty much
concentration points for the stress. The stress levels were even
for over four times, something like four times higher than
around the not on the corner. So and that was
around the corner of the windows and the antenna radio
(20:23):
antennas and stuff mountaints. So the solution was brilliantly simple.
If the corners have the problems, eliminate corners. That's why
the windows became round. If they investigation, the signs were
before that, but they weren't leading to.
Speaker 4 (20:40):
Catastrophes yet, so they were ignored.
Speaker 3 (20:43):
If you look at this from the IT perspective, it's
kind of the same. As long as you have an
issue when you deploy the new version and something crashes,
it's enough to restart pipeline or something like that. Okay,
let's not investigate, we need to focus, but that could
be an underlying issue that later could lead to the
(21:04):
catastrophe like night copy.
Speaker 1 (21:06):
First of all, the airline industry is quite the shining
example of diving into post mortems. I feel like, compared
to other industries, the idea of preventing not just loss
of life, but preventing all incidents now, current and future
is something that's really focused on and like really getting
to understand, like all the root causes, not just the
(21:28):
first one that pops up, so that these events in
any way can never happen again. And I feel like
World War Two in a lot of ways was a
turning point. I think there was this very specific interesting
conclusion insight that was had, and I believe it was
in the United States with the return of vehicles that
were damaged in some way.
Speaker 4 (21:46):
It was in US. You are talking about survivors.
Speaker 1 (21:49):
Bias, yes, yes, of course, but.
Speaker 3 (21:52):
The conclusion was from I think Hungarian statistician.
Speaker 1 (21:57):
Basically, the idea was that if you receive vehicle back,
then it survived its.
Speaker 3 (22:02):
R US Army was analyzing the bullet holes on the
returning bombers and their initial intuition was that if that's
where the planes were hit, they need to fortify those points.
And just this Hungarian mathematic statistician pointed out that hey, stop,
(22:22):
if you see those holes, you don't see the holes
on other parts.
Speaker 5 (22:26):
That means that the planes that were hit there did
not return.
Speaker 3 (22:30):
And that's also a very strong case for post mortems
because its very easy to focus on the success stories,
but the real learning, real they tell lies in the disaster.
Speaker 1 (22:41):
It's very counterintuitive. I think it's the conclusion that a
lot of times we come to here, and that it's
almost like if you aren't coming to a counterintuitive conclusion,
then maybe you're missing something critical about your post mortem process.
And I think this is where the survivorship is come
doesn't play you. You have a release or an incident,
and you sit down and a meeting and you discuss things,
and the conclusion invariably is like, we need more tests,
(23:05):
and well, anyone could have told you to have more tests.
Speaker 2 (23:08):
I think the clever trick.
Speaker 1 (23:10):
Is knowing which tests you should have, because you know,
if you just had the right test, you can avoid
every single production incident that would ever happen.
Speaker 4 (23:20):
For that.
Speaker 3 (23:20):
I kind of like that approach of do not write
more tests just because you want more tests?
Speaker 1 (23:27):
That conclusion is often a struggle for organizations, like there's
a belief that oh no, this this software doesn't change
that frequently, or we don't have the priority to actually
be able to implement there. But yeah, I mean, if
you're in the weeds and actually fixing something or changing
the way something works, I think a good metric that
at least my teams use is add the test before you.
Speaker 2 (23:48):
Make the change.
Speaker 1 (23:49):
Like, it doesn't matter if there are necessarily tests on code,
but if you're going to change something, then you can
have an opportunity to potentially add the test first to
ensure that the thing you're changing doesn't break in a
particular way. And I think the follow up here is
that often things break for the new functionality.
Speaker 3 (24:05):
But generally the most box most mistakes were when you
introduce the change.
Speaker 2 (24:11):
Yeah, yeah, for sure.
Speaker 1 (24:11):
Realistically here there's like a whole bunch of failure modes
where you're making assumptions about what the current thing is
doing rather than actually validating it in some capacity. Here,
there is one thing that I think is still worth
bringing up in the area of post mortems, especially when
it comes to the failure rate, and it's this idea
where you it's it is actually impossible to predict where
(24:36):
the future errors will will show up in some capacity,
and the mistake that's made frequently is tracking the number
of errors you have or the size of them in
the path, because nothing tells us that the errors that
we've encountered so far and potentially fixed or prevented, have
any impact on the future errors that we're going to
encounter in our organization. The criticality of the failure. Every
(24:58):
company works until the moment it doesn't. Every software works
until the moment it doesn't. And I think there is
this idea where, oh, yeah, we can just fix every
problem we've identified. We should tract the number of problems
over time and will just be good. And I think
I'm hoping the history and the research that you've done
leads leads, you know, anyone who reads the book to
(25:18):
the opposite conclusion, which is like you need a better
process or a mindset around how you're tackling what you're
tackling rather than just what potentially comes up.
Speaker 3 (25:27):
There was a technique of city editors. Microsoft was doing
this with Microsoft Betwindows, but it was totally for a
different purpose. But it was a technique to statistically evaluate
how many bugs they didn't find. Quite interesting, not necessarily
applicable to constantly evolving software nowadays. But the main lesson
(25:47):
is to not ignore the errors. On one hand, own
mistakes to try to analyze it. Second, don't ignore other
people's editors. That's free learning. Sometimes you can last, sometimes
you can be amazed, but for sure there is some
lessons there that you can avoid.
Speaker 1 (26:07):
Yeah, it's a very famous Russian submarine that I don't
recall that.
Speaker 3 (26:11):
That's what I was aiming at. It was Staniswa Pietrov
who was in Soviet air defense.
Speaker 4 (26:18):
And indeed they.
Speaker 3 (26:19):
Had one censorus reporting that US launched a missile, and
he ignored it until they got back up information from
different stations a couple of minutes later. According to Protesto.
To Proteto, he should be he should immediately counterfire. He
ignored it, and that that was nineteen eighty three. He
(26:42):
ignored that, and that's why the World War two did
not start. If he shot back, we wouldn't be talking
right now.
Speaker 2 (26:49):
For sure.
Speaker 4 (26:49):
Not.
Speaker 2 (26:50):
I think this could be a good stopping point.
Speaker 1 (26:52):
Let's move on to picks. My pick for today is
the quitos quetason. I don't know how to pronounce it.
Food stories containers. I really like these.
Speaker 2 (27:02):
They're really expensive, but they're highly durable. They go in
the oven, I don't know a microwave. They go on
the stove.
Speaker 1 (27:09):
I guess they go in the freezer deep freeze minus
twenty degrees celsius.
Speaker 2 (27:13):
They are absolutely fantastic.
Speaker 1 (27:15):
And what I like about this brand is they have
a lot of different sizes, so you can measure exactly
what fits in your refrigerators and your storage units and
focus on stacking them the best. I probably have like
fifteen or twenty of these, like almost this exact size.
Speaker 2 (27:28):
And honestly, they've been great about it.
Speaker 1 (27:30):
Five years ago I completely moved off of storing anything
in plastics in any way, and now everything I wanted
is in glass or stainless steel metal. Actually this is
a composite and these have been the best so far.
And I really like them because after cooking, I always
have leftovers.
Speaker 3 (27:48):
I need to check you and if you ask me what,
I would say, well, I recently watched the TV series
on Netflix called Fubar. It's u pical, well modern take
on nineteen nineties early two thousand action comedy movies with
(28:08):
Arnold Schwarzenegger, and it's super full of fun service jokes,
even quoting making fun of the most famous scenes with
Swarzenegger and his best movies. So if you grew up
in this time. It's very much enjoyable.
Speaker 1 (28:28):
I think I think you has actually been on my
watch list for a while, and I haven't. I haven't
gotten around here.
Speaker 3 (28:33):
It's really really good with really solid screenplay. Of course,
it's spy spy movies, so full of physical absorts, but
in the rules of the presented world, it stands on
its legs and you can really enjoy it.
Speaker 1 (28:54):
Okay, Well, then I guess that's going on someone's watch
lest that will be an episode notes. So at this
point I will just say thank you Adam for for
coming on to the to the show and talking about
your your book and your future upcoming book. I feel
like the postmortems will be really interesting and maybe we'll
get you back on it.
Speaker 3 (29:11):
At that point, well, thank you for having me, thank
you for a very nice conversation, and I would be
super happy to be back.
Speaker 1 (29:18):
And with that, I'll say thank you to Attribute one
more time for sponsoring today's episode, Thanks for everyone listening,
and we'll be back next
Popular Podcasts
My Favorite Murder with Karen Kilgariff and Georgia Hardstark
My Favorite Murder is a true crime comedy podcast hosted by Karen Kilgariff and Georgia Hardstark. Each week, Karen and Georgia share compelling true crimes and hometown stories from friends and listeners. Since MFM launched in January of 2016, Karen and Georgia have shared their lifelong interest in true crime and have covered stories of infamous serial killers like the Night Stalker, mysterious cold cases, captivating cults, incredible survivor stories and important events from history like the Tulsa race massacre of 1921. My Favorite Murder is part of the Exactly Right podcast network that provides a platform for bold, creative voices to bring to life provocative, entertaining and relatable stories for audiences everywhere. The Exactly Right roster of podcasts covers a variety of topics including historic true crime, comedic interviews and news, science, pop culture and more. Podcasts on the network include Buried Bones with Kate Winkler Dawson and Paul Holes, That's Messed Up: An SVU Podcast, This Podcast Will Kill You, Bananas and more.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com