The Unspoken Challenges of Deploying to Customer Clouds

This episode we are joined by Andrew Moreland, co-founder of Chalk. Andrew explains how their company’s core business model is to deploy their software directly into their customers’ cloud environments. This decision was driven by the need to handle highly sensitive data, like PII and financial records, that customers don't want to hand over to a third-party startup.

The conversation delves into the surprising and complex challenges of this approach, which include managing granular IAM permissions and dealing with hidden global policies that can block their application. Andrew and Warren also discuss the real-world network congestion issues that affect cross-cloud traffic, a problem they've encountered multiple times. Andrew shares Chalk's mature philosophy on software releases, where they prioritize backwards compatibility to prevent customer churn, which is a key learning from a competitor.

Finally, the episode explores the advanced technical solutions Chalk has built, such as their unique approach to "bitemporal modeling" to prevent training bias in machine learning datasets. As well as, the decision to move from Python to C++ and Rust for performance, using a symbolic interpreter to execute customer code written in Python without a Python runtime. The episode concludes with picks, including a surprisingly popular hobby and a unique take on high-quality chocolate.

Notable Facts

• Fact - The $1M hidden Kubernetes spend
• Giraffe and Medical Ruler training data bias
• SOLID principles don't produce better code?
• Veritasium - The Hole at the Bottom of Math
• Episode: Auth Showdown on backwards compatible changes

Picks:

• Warren - Switzerland Grocery Store Chocolate
• Andrew - Trek E-Bikes

Episode Transcript

Speaker 1 (00:08):
And we're live. Welcome back, everyone to another episode of
Adventures in DevOps and I have today no co host,
so I'm flying solo for the what looks like foreseeable future.
So as consolation, I have a fact. There's a great
white paper by Portainer talking about the operational overhead of Kubernetes,

(00:28):
and so I'll link that in the talk description below
after the episode, so everyone get at it. Basically, the
TLDR is you're basically going to be spending an extra
million dollars a year just because you decided to pick
one orchestration platform over another one. More on that maybe later,
but let's get to the point of the episode. I've
got and I'm really excited today to have Andrew Morland,

(00:50):
the co founder of Chalk, with us. So welcome to
the show.

Speaker 2 (00:53):
Thanks Wan, it's great to be here. And I have
to say I do agree A million dollars sounds about right.

Speaker 1 (00:58):
It sounds like a lot of money, I think, But
if we realize that's pretty much just five engineers, Oh well,
I actually totally get that, but I think a lot
of people throw that under And it's interesting because if
I understand correctly, Chuck is a fairly recent company. Therefore,
maybe it's okay to call it a startup, and I
can wonder whether or not you've had to actually make
a decision on whether or not Kubernetes will be part
of the tech stackt that you've decided to run with.

Speaker 2 (01:19):
Yeah, so we do deploy on Kubernetes, and we can
maybe talk about some of the details later. We run
on Kubernet's clusters in our customer clouds. We did decide
that it made sense to use. I think one of
the things which pushed us in that direction is we
since we deploy into customer clou accounts, we have to
deal with multiple different kinds of clouds. So for us,
Kubernetes is kind of a normalizing layer across the specific

(01:40):
details of easy two or GCE.

Speaker 1 (01:42):
Seeah. I think that's a really intelligent way of putting
out because a lot of times I hear we need
to use Kubernetes because either we need to pad our resumes,
or more likely we want to be cloud agnostic, even
though there's no business reason to do so. But what
I heard is we actually have a business justification for
being cloud agnostic, and you're doing the ridiculous thing of
if I've got the right deploying to your customer's cloud

(02:02):
environments directly. That doesn't sound like something I ever want
to do.

Speaker 2 (02:07):
Yeah, it wasn't something that we wanted to do initially
either or that we intended to do. We started out
as like effectively trying to present a hosted SaaS, So
we wanted people to just come onto our website kind
of like for Seal or like you know, any of
the other platform the service companies click sign up and
then like deploy their software and move on with their lives.
But it turns out that when you work with more
mature companies and you ask them for all their most

(02:28):
sensitive data like their PII, their medical records, their financial records,
they don't really want to hand that to a random
seed stage staff startup. So it's kind of mandatory for
us to start processing data in our customer cloud accounts a.

Speaker 1 (02:41):
Pretty great justification, And we see the cost of paying
cloud provider storage and even transferring between one account to
another one increase over time as well, and so the
ability to go into any sort of account and utilize
the data where it is makes a lot of sense.
Can I ask, like, what is TALK doing where it
becomes the competitive advantage to be deploying into your customer's environment.

Speaker 2 (03:02):
Yeah. So, like, the key thing that we do for
our customers is help them process, transform serve data to
machine learning models. So we don't have specific opinions about
the exact transformations they want to do or what their
models are for. Oftentimes it's things like fraud or logistic
or logistics or recommendation systems or search or something like that.
But because these models tend to be really core to
their business or to their risk and fraud functions, they

(03:23):
process all the most sensitive things that they have. There's
a few different dimensions in which being in the customer
account matters. One is maybe the most obvious, which is
that their data doesn't leave their account. When we talk
to companies, we always talk to their infrastructure and compliance
and security teams, and it gives those teams a lot
of comfort to know that the data stay is resident
and that we don't actually have like we can't access

(03:44):
it if they don't want us to say. Another maybe
less obvious point is kind of what you touched on
with the network transit. Like a there's obviously no network transit,
but b it lowers the latency. So a lot of
our customers care a lot about super low latency for
these types of applications.

Speaker 1 (03:58):
My understanding was, and this is how zero experience and
either any sort of LM data modeling or any sort
of machine learning as far as it goes for the
actual data, say something like fraud, I wouldn't imagine necessarily
the PII would be relevant in the fraud detection. But again,
like I said, not the expert here, so I can
imagine the flip side, it may not be directly relevant,

(04:19):
but super difficult to separate it out to be able
to hand over only the relevant pieces.

Speaker 2 (04:24):
Yeah, it's kind of i'd saying both. So some of
the companies who work with are literally about the PII.
So like one customer, so cure, they're kind of their
whole business is is this particular set of PII Actually
the PII for a real person if you look at
their website, like their API is literally I mean not
literally but close enough is basically post US and social
security number, phone number, email address, and address and we'll

(04:46):
give you back a yep, that's a person or not score.
So as you can imagine, that's pretty sensitive.

Speaker 1 (04:51):
Yeah, it's always interesting being in that position where it's
almost like you don't know what the data is and
you're in a way praying that it's not sensitive. But
then you spin up an API where the answer is
is it sensitive data? And the answer like, you're getting
sensitive data and there for.

Speaker 2 (05:05):
Sure, yes, yeah, it's it's it's almost like nuclear waste.
You really don't want to have to process it. But
in this in particularly in fintech, you often can avoid it.

Speaker 1 (05:12):
Share with me here, if you've working with these fraud
detection systems, you must know all the secrets on how
to evade them.

Speaker 2 (05:18):
That well, I think part of the part of the
reason why we deploy into the cloud accounts is that
I also don't get to see the code the customers
are running oftentimes. I mean we talk about it. We
obviously very hands on with the people we work with.
One i'd say ideally there is no way to evade
these systems, and two I don't get to know all
the details of how they work. But I mean there's
some obvious things like probably anyone couldn't do it. Like

(05:40):
it turns out, identity theft does work if you're using
a real person's identity. It's a lot harder to you know,
detect that. You know, it's kind of all the obvious
things too. Like you if you look at some of
the other customers we work with, like Varasol, et cetera.
Just on the on the homepage of the website talk
about device fingerprinting, So you know, people who are trying
to commit fraud go to great lengths to try to
avoid these systems, and that mostly involves pretending to be

(06:02):
real people for very long periods of time so that
they can kind of later on flip and become malicious.

Speaker 1 (06:06):
There's a parallel to what we see in advanced persistent
threats on digital systems, where that attacker lives off the
resources for I think it's up to nine months is
usually the average time before actually deciding to actual trade resources.
And I think the other metric numbers like thirteen months,
where people start throwing away the log data at that

(06:29):
point and so you don't even know what systems they
had access to at that moment. I want to get
back to the deploying into the customer accounts though, because
it's something that I said that I've always tried to
avoid and never wanted to do, and you're doing it,
and so this must have come with either some risks
involved or some challenges as far as actually getting the
technology to work. Like, it's not just so simple even
in AWS, getting the access to the individual accounts hand

(06:53):
over with external account IDs or identifiers, i AM rules,
et cetera.

Speaker 2 (06:58):
I would say there's like a lot of things that
are hard about it. Some are obvious, some are not obvious.
The obvious ones are like, Okay, how do you actually
write down all the im permissions that your application needs?
Like who knows? Right? So the answer was kind of like, well,
what scope it to zero, build a tool to automatically
deploy it, and then just loop for days until we
finally find the minimal set. But there's more subtle things too,

(07:21):
like it turns out that in all these cloud platforms
there are global policies you can apply across all the accounts.
And of course, like really savvy infrastructure people are out
there secretly saying things like oh, you can't transitively assume
i AM rules, But we have no way of knowing
whether these policies are applied, and now do the people
we're working with these companies. So even if you control
all the configuration, right, you're still host. But there's all

(07:43):
sorts of other things that go wrong too. You mentioned
again earlier the network transit. It turns out that there's
like a known problem between GCP US East four or
Native US US East one, for example, like the network
pipe there is just full, so it is understood and
expected that you'll suffer packet loss over that pipe fairly often.
They actually have a whole product page which I didn't
expect to find, where you can buy a specific dedicated

(08:04):
link on that pipe to guarantee yourself bandwidth. But basically
the cross clouds defends up being super wacking in new scenarios.

Speaker 1 (08:11):
Also, that's actually really interesting. You just brought it up
because cloud Flair just had a downtime that affected a
whole bunch of customers, and I believe the same thing
actually happened with It was about network congestion a lot
from cloud Flare to their US East one in Ashburn,
And I'm like, I have no idea what's going on there,
And you've said that it's like the second time in
a month that I've I've heard about this, and now
I start getting really concerned.

Speaker 2 (08:32):
I think they're running out of internet in Virginia. It
seems like the problem.

Speaker 1 (08:34):
At this point, It's like does that data center still exist?
And I think the meme is for sure, like why
are you doing anything in US East one? And the
answer is because our customers are exactly how any choice?

Speaker 2 (08:45):
Yeah, I have no choice? Unfortunately.

Speaker 1 (08:48):
Wow, Okay, you know, the network going down was not
a top three of what I would have expected to
have to deal with, you know, as a startup. I
mean we're almost ten years old now, and I still
remember every single challenge we went through. Technically deploying the
customer accounts though, is definitely not one of the things
on the list. But yeah, crosscount im roles and service

(09:10):
control policies. It used to be much worse, right, used
to get to deny and you had no idea in AWS.
Now at least it says, yeah, denied due to service
control policy and your good good luck you know getting
that changed. But at least you know you know what
you're trying to avoid there. How do you deal with
expanding the footprint that you need in your customer environment?
So if additional permissions are required, is it a manual

(09:31):
process to get them to basically update something, or is
I can't imagine what the workflow would be.

Speaker 2 (09:36):
Yeah, it's not my favorite thing. Yeah, it ends up
being kind of a negotiation with every single customer. So
I think fortunately these days, most of the im permissions
are introspectable, like you can say do I have this
permission and before you go and use it. So we
try to write our software to do that sort of
thing and then present like warnings on the dashboard about oh,
we're missing some permissions. In general, it kind of means

(09:58):
we can't use a lot of stand tooling, Like terraform
doesn't work really well in this workflow because you know,
a terraform's kind of awkward if you want to automate
it as part of the sign up button flow. B
terraform really does not like not having permissions for things.
You'll end up in situations where the DAG can't totally resolve.
I can't even plan, So we end up having to
build a lot of the functionality that you would normally
get out of the box and terraform for free kind

(10:19):
of in our own go application which does the orchestration
all these resources, so that we can recover from missing permissions.

Speaker 1 (10:25):
I don't want to understate that It's just a simple
thing like you write in some terraform which are open
TOFU now to update a resource, and the first thing
you do is you try to go get that resource
and you get to deny there. So even if you
have an update like the ability to create that resource,
there is a hidden get there that you may not have.
And AWS or the other cloud providers do change over time,
and so that one resource can become two resources, sure can,

(10:48):
and the open TOFU terraform whatever bloomy provider will change
to make the two gets, and all of a sudden
you would start running into two issues there. So yeah,
I can. I can totally image. It's more of a
feature flag flow where it's like, well, in order to
get this feature, you have to first go through the
security steps and then we'll validate it on our side,

(11:08):
and once it's been validated, then we'll actually enable it
for you.

Speaker 2 (11:11):
But I'd say, like maybe the other answer too is
we've been trying to pull more things in house over time,
and in the beginning we're startup, we don't want to
build everything on top to bottom. We want to delegate
to the best in class solutions where we can as
we have more resources and more experience and more opinions.
We found that it's better to bring things like you know,
log aggregation in house rather than relying on like cloud
watch or cloud Trail whatever the Amazon products called, or

(11:33):
stack driver logging. We'd rather use like ClickHouse logs or
something like that that we control so that we can
make sure that it's a consistent experience and also kind
of lower our permission footprint.

Speaker 1 (11:43):
I can really understand that, especially if you're getting the
logs directly out from the customer account. It's one thing
if your abuse account or GCP account is creating logs
in the log solution itself, but if they're being generated
in somewhere that you don't control primarily, and then you
have to then write some sort of funnel to get
from one place to another one, you're going to immediately
start questioning where do we want this data to end

(12:04):
up and then pay the least amount for it and
have the most reasonable retention in that regard, Like what's
the onboarding process? Is it manual? Is it? You know,
do you take like a white glove approach each of
these customers you said, you know much larger they do
care about their data in a lot of ways, it's
not going to be probably self sign up in a
lot of scenarios.

Speaker 2 (12:21):
Yeah, we are marching towards self sign up honestly, just
to lower the operational burden of the White Glove version.
You know, I don't think anybody really likes having a
conversation about how to deplay software, so we would rather
have a sign up form people can go through. Whether
or not we actually make a publicly available as a
separate question, because in some ways we're a remote code
execution platform, so it is really challenging to make sure

(12:41):
that our software is secure in an untrusted environment. But
that aside. The way that we do it these days
is we do a little handshake where we exchange i
AM rolls or things like that, maybe some private keys
and public keys, and then we do work with people.
Probably for kind of cookie cutter deployments, it's usually a
couple hours. For more involved deployments, maybe across multiple cloud

(13:03):
regions or you know, or cloud accounts or cloud providers,
even that can be you know, two three weeks to
get stuff all the way world out and kind of
certified for production.

Speaker 1 (13:12):
When you ineavily get to the point where you've decided,
after many years of running your company that it's time
to deploy a second version of whatever you had. How
does that get to the customer environment.

Speaker 2 (13:23):
In the beginning, when we thought we were going to
be like the versell for data and to today we're
more enterprising. We used to just publish releases and people
will pick them up kind of automatically. But it turns
out again, you know, like large enterprises don't really like
surprise software updates for lots of reasons, you know, both.
You know, maybe we make a mistake, maybe we make
a change that they don't like, maybe we make an
improvement that they don't want. So we publish version releases

(13:45):
of our software now, and we do ask that customers,
you know, basically choose which version they'd like to execute.
You know, obviously that has all the downsides of version
software releases, i e. My mistakes live forever.

Speaker 1 (13:55):
Yeah, as you have like some HELM charts or something
that their customer is pulling in and running in their environment.

Speaker 2 (14:00):
Yeah, we version the HELM charts for like the full
metadata plane deployments when people host that themselves, we version
like the underlying software version or SDKs. All those things
are there.

Speaker 1 (14:09):
Unexpected downsides here of going down this approach rather than
you know, forced rollout or upgrades. Do you find that
you're getting support scenarios for the older versions that are
missing critical stuff.

Speaker 2 (14:21):
Yeah, it's definitely a problem. I mean I always what
I always tell people is, you know, I think my
latest version is my best version, and it should have
all of my my newest and best things in it.
So if people are not on it, then you know,
I think that they're missing some stuff I think they
should have. So we do run into issues where we
fix bugs, we make improvements, and they're important, like we
add better load shedding for instance. And if someone's runn

(14:43):
an old version that doesn't have the ability to like
properly do dynamic rate limiting, they may run into a
scenario where they get overwhelmed and end up having a
latency you know, espalation. We do keep track to what
versions people are running, and we do, like our kind
of solutions team works with them to make sure they're
updating regularly. We don't really want people to be stuck
a year old version forever.

Speaker 1 (15:01):
Are there though?

Speaker 2 (15:03):
I think we had one person who was running January
software until about four weeks ago but in general we
most people are within two or three months.

Speaker 1 (15:11):
I mean that's pretty recent, like even January, you've got
same year there, twenty twenty five, without any previous hiccups.
That's I would say. I'm a little surprised. I see
a lot of companies running older versions of stuff that like,
we have open source SDKs for every language for our product,
and I can guarantee you there are some that have

(15:31):
never upgraded since the moment they deployed their original version.
And those early customers of ours, you know, that's almost
ten years ago. So that's kind of scary in some regard.

Speaker 2 (15:40):
Yeah, No, I think that like the main way we
try to like engineer for that as we make sure
that we have never intentionally made a backwards incompatible change.
One day we may make a breaking change. We have
made some mistakes in API that we love to correct,
but we really want to prioritize that upgrade experience, so
we'd never ever intentionally break any time the customer can't

(16:01):
upgrade because of a regression, we make sure that's a
p zero issue. So that's top of the heap for
us in terms of triage.

Speaker 1 (16:06):
I think I really like this perspective I went into
it a little bit earlier in one of our episodes
in the Off Showdown where we were discussing multi tenant
versus single tenant solutions, and one of the things that
came up was like, can you manage backwards compatible like forever?
And because we have a SaaS, that's actually what we
really try to focus on doing. Because I'm totally with you,

(16:27):
it's so critical to be able to migrate from one
version to another one. And I feel like one of
the problems with like open source software or you know,
you choose the version as a customer is that there's
less discipline involved from the company. Right if you can
break things and because you say, well, people don't have
to upgrade, then you don't care about it. But you've
taken a like really smart, mature approach there, which is, yes,

(16:50):
it's not critical for any of our customers to do this,
but it's so critical from us, and we know the
pain that they're going to find at some point that
if they can't do the upgrade, it's something that we
have to have to fix. For sure.

Speaker 2 (17:01):
Early moments in the company's history that kind of shape
this philosophy. In addition just our opinions was we had
a company churned to us from a competitor because they said, well,
doing the upgrade for our competitors software is approximately as
hard as adopting a new solution, so we might as
well do a competitive email. I was like, well, okay,
we're never going to give anyone that opportunity.

Speaker 1 (17:19):
So many different things that we think about in our
product when it comes to our customers and trying to
avoid mental challenge of the churn, right, I mean, even
if they're not actually going to do it, you know,
having customers tell you the exact reason that they're churning
and be like, well, let's bake that into the future.
DNA is really is really something there. So before we
got started in today's episode, there was a lot of
technical problems, but many minutes of dealing with the current

(17:46):
state of microphones and software. But one of the things
you had mentioned that I completely glossed over, and I
have to be honest, I'd never heard anyone say this
before was we deal with time travel in data and
I honestly have no idea, like I know, the idea
of time travel like replaying events from some sort of
event stream that's not exactly the model that you have

(18:06):
to deal with.

Speaker 2 (18:07):
Yeah, it's not totally dissimilar. So one of the things
we think about a lot is bi temporal modeling, which
is a really fancy way of saying that there's you know,
at least two time stamps you actually care about in data.
One is you know, when did a thing happen, But
the other thing is when did you know that it happened.
And that's a really important distinction to make when you're
trying to train machine learning models. Thinking about a fraud model,

(18:28):
someone commits fraud on your platform, maybe on like January first,
but you don't actually find out that it was fraud
until maybe two months later when they issue an ach
return at the end of the sixty d expiration window.
So you did not know that it was fraud until
say March. So when you're building your training sets to
train your machine learning models, you need to be very
careful to make sure that no information which you learned

(18:49):
after the time of the decision leaks backwards into the
effectively row of data which represents that event, even though
you have been collecting data during that whole window and
learning new things. To make sure you filtered out and
that's not so hard if you have immutable data sources
in simple relationships, but it gets really nasty and you're
doing things like computing aggregations over transactions or applying complicated

(19:13):
filters on statuses which are evolving over time. So at
the core of our software is effectively a SQL engine,
but one of the ways in which it's different from
most equal engines is that it has this notion of
the effective timestamp of sell data in the system, so
we can kind of manage that filtering for you.

Speaker 1 (19:31):
If I'm doing some sort of machine learning, and in
the data set that I'm using to train the model,
I actually need to make sure I'm not including the
sort of output of the model, which in a lot
of ways is something you today with lllms. We actually
want in there because we want it to actually affect
the driver of the token selection on the output tokens.

(19:52):
But if it's included in there, then the model sort
of knows about that data, which will in effect defeat
the whole purpose of the model, which is to figure
out not knowing that. I mean, I can understand associations
or matching that where you would want it, but when
it comes to date based things. I totally get that.

Speaker 2 (20:07):
You can kind of apply the same logic to the
LM applications too. Imagine like you present a transaction to
GPT five and you say, like, is this fraud or not?
It'd be really unfortunate if part of the information you
gave it included the label is fraud or not. It
would hopefully pick up on that label and say, ah, well,
it's in fact not fraud, So we know that a priori.
That wouldn't be In general, when people go and swipe

(20:30):
credit cards at merchants, they don't actually tell the merchant
whether they're committing fraud, so it's not super useful to
train on that signal.

Speaker 1 (20:35):
This remind me of two scenarios in the recent history
for the creation of LMS and machine learning in general.
The first one is the presence of the medical ruler
in the detection of cancerous moles and termatology. Like, if
the ruler is there, there's a problem, right yeah, And
I think there's a generic base for this known as

(20:56):
like the giraffe problem early on, as if you gave
an image model a picture of a safari and said,
is there a giraffe? The answer is always yes. Because
there was, because it has not because there was ever
a giraft there. It was because if you ever asked
if there was a giraffe, it was highly correlated with
there actually being a giraffe. So therefore the presence of
the question eliminated the possibility of it being no. So

(21:18):
as soon as the question was asked, there was always
a draft, even if there wasn't one in the picture.

Speaker 2 (21:22):
Yeah, it's the same sort of problem that we're trying
to stop. It ends up being extremely painful and subtle
because you know, you have database records that have timestamps
on them, but it turns out in production systems the
timestamps are approximate, or maybe someone else came along and
updated the row and the original domain modelor didn't know
about it, or things like that, So I don't know.

(21:43):
Twenty percent of the time in the entire process of
doing machine learning is about actually figuring out when things
did happen and when you did really learn about them,
and would that information have been available.

Speaker 1 (21:53):
If someone had the ridiculous idea to try to, you know,
go build time travel themselves and not rely on chalk
to solve it for them, Like, is there some secret here.
I mean, don't don't tell us everything, but you know
something specific about how you're thinking about the problems in
order to evade this potential issue in the output.

Speaker 2 (22:10):
I think there's a couple of different dimensions we think
about it. One is like at the SDK layer, when
people are defining, you know, how data is integrated into
the system. We can kind of have things which encourage
them to think about the distinctions between these types of timestamps.
Then there's like the actual query execution layer, so inside
of the system, which is doing things that are like
seqal semantics. You know, in a normalst equal database you

(22:30):
track tuples. We're tracking tooples augmented with a lot of metadata,
so we know extra information, extra bits basically about every
value we're moving around, which tells us when did it
become effective, when did it become replaced? You know, is
this an actually valid value? Is it noll because it's missing,
and we're making sure that all the operations we do

(22:51):
kind of properly transform that metadata. In addition to the
source data. You would have to do something like that,
I think in order to build a replacement. And then
the third thing is down at the actual physical execute layer.
So we care a lot about making sure things want
run really fast. So we have to implement like custom
join algorithms and custom you know, actually ways of adding
numbers together so that we can respect all this metadata

(23:11):
and do these operations efficiently. It turns out that just
like if you just do straight standard SQL aggregations, you
run into enormous cardinality explosions and in the size of
the data you're processing when you have to think about
all the different time points.

Speaker 1 (23:23):
It occurs at just for clarity, the models, they're being
built on your customer side based off of the data
that you're providing, or are you helping to build the model.

Speaker 2 (23:32):
Yeah, So we don't usually get involved in the actual
model construction. We do oftentimes provide advice about how to
model the data that's flowing through the system, like what
schemas make sense? How should you think about time right now?
We don't do any model training, Like, we don't do
any like solutions stuff for model training for people.

Speaker 1 (23:50):
Think it's like immutable databases for machine learning.

Speaker 2 (23:53):
Yeah, and we call ourselves the data platform Free AI
machine learning. We don't think it's specifically fraud related like
we do recommendation systems search logistics models a lot of
content moderation, which is kind of fun. We basically think about, Okay,
what are all the workflows that are involved in our
productionizing AI machine learning? And can we build like an
end to end answer for that versus building horizontal infrastructure.

Speaker 1 (24:16):
You found one of the magical shovels in this AI bubble.

Speaker 2 (24:20):
Elliott and I both did effectively the job our customers
do at a bunch of different companies before. So we
saw a system like chalk get built a bunch of times,
and we've tried to, you know, make an actually good
version of it instead of the like, you know, four
engineers two quarters version of it.

Speaker 1 (24:33):
Having been in a similar circumstance, I find the challenge
there is you shift from being a technical person, which
it very much seems you are, into what I absolutely
hate doing, which is marketing and sales to try to
convince people to buy the product that you already know
they need. And you're nodding your head, which means you
know exactly what I'm talking about.

Speaker 2 (24:50):
I don't know if this is like something we should say,
but like, effectively we have one hundred percent pilot closed rate.

Speaker 1 (24:55):
I e.

Speaker 2 (24:56):
If you come and try our software. It's almost inevitable
you will buy it. Of course, Then the problem becomes,
how do you get the word out? How do you
get people to actually try it? So you know here
we are?

Speaker 1 (25:05):
It is.

Speaker 2 (25:06):
It is interesting because I think it's really difficult to
convey what we do and what problems we're solving. And
I don't think we've really found the two sentence elevator
pitch it despite thinking about it for three years.

Speaker 1 (25:15):
I'd maybe give you some more confidence in your ability here.
I do think you're you know, you're clear about talking
about what is going on there, and even at our point,
which is I don't know, I forget the exact year,
how long we've been around it. It's not even competitors.
It's other companies that have used similar language and basically
stolen it to mean something completely different. And now that

(25:36):
all that's left is either you compete with them and
say the same things they're saying, but mean something totally
different the right thing, I'll say, not the wrong thing
that they're they're peddling, of course, so you say something
completely different and your customers are like, we have no
idea what you're talking about, and they you know, churn
from your marketing page, or they don't even get there
because they don't understand what you're actually going at. I'll

(25:57):
say from experience, one of the secrets here, especially going
out to the larger enterprise customers, is it doesn't matter
because you don't care about your marketing page. You care
about you know, connecting directly with whoever is making the decision.

Speaker 2 (26:10):
Right, Yeah, FUAL high touch sales.

Speaker 1 (26:12):
Yeah, high touch sales, So you must love that.

Speaker 2 (26:14):
Fortunately, both of my co funders are much more charismatic
than I am, so they focus mostly on the pre
sales stuff before people actually sign to buy the software,
and I get to think about the hard, interesting problems
post sales. But it's definitely a big challenge, and I
think like the kind of reason we constructed this company
is because Elliot and I did a consumer fintech before,
and that was one hundred percent about marketing. If you
don't love changing the background color of ad creative on

(26:38):
Facebook Ads Console for your entire life, you shouldn't do
consumer fintech. And we basically spent the time during our
earn out at Credit Crema thinking about, Okay, how can
we make a company where we get paid money to
solve interesting systems problems for you know, big companies, and
this is what came up with you went.

Speaker 1 (26:54):
Down a third path. There, if I understood your LinkedIn
experience correctly, you made a great company and it got acquired.

Speaker 2 (27:01):
Yeah, we built a consumer fintech company. We basically stapled
a bank to an investment brokerage and then added like,
we call it self driving finance. But that's like you say,
Everyone says that now machine learning based wealth management, which
would do constraint solving to move money around automatically for you.
And it turns out that the set of people who
are really interested in personal finance such that that sounds

(27:22):
like a cool thing to them, and also who don't
want to be involved in the management of the personal
finance is approximately the nul SAT. Basically, everyone who cares
about personal finance wants to manage it, and everyone who
doesn't care about personal finance is not going to buy
a fancy financial planning product. So the tech was cool.
We grew the business. Eventually, Credit come I bought the
company because they wanted to offer that type of financial
management to the one hundred million some people that they serve.

(27:45):
I learned that I really don't enjoy doing consumer.

Speaker 1 (27:48):
Yeah, I have a slightly different takeaway there. I make
a lot of mistakes with what to do with my
personal finances, and that made me one time, say, you know,
doing some hypothesis and guessing it's going to be more
of a better strategy, And so I threw some money
out a product that automatically promises he returns by buying

(28:11):
and selling shares of different index funds. And it wasn't
the worst decision I ever made. I mean, I didn't
put all my money there, but I definitely put some there.
And so, like I do, if you think you're good
with money, like just dispel that notion. You're definitely worse
than just buying the index fund in in most scenarios.
But you can do a little bit better than that

(28:33):
if you happen to find an engine that is willing
to manage it. So being in that nulset, that magic
space where you know what you're doing but also want
to do the management, and then just stop doing the management.
It's like a form of delegation. Yeah, I like the
pivot because you're doing something that seems like way more
interesting than moving money around. I mean, I don't know.
I'm saying that I don't know anything about the technology

(28:53):
and either of these pieces, so maybe that's a bit unfair,
But I'm what you've shared so far is like, in
a way, really exciting because a you're not talking about
building AI. You're talking about your customers doing machine learning
on a complexity set of data that they just wouldn't
be able to achieve otherwise. And you're deploying your software
into their accounts so that you can get access or

(29:15):
integrate with their cloud provider where the data actually is,
not to do it yourself. I mean, you're avoiding a
lot of complexity that didn't bring value, and you figure
out a way to still, you know, solve the real
challenging problem, which you already experienced. I mean, I think
you're onto something. And you said the company's only three
years old.

Speaker 2 (29:33):
Now I would say that so far it has been.
It's been interesting because you see the challenges coming, but
maybe you choose to run into them anyways. So I
think we've dodged a lot of the dumb things like
messing up with fundraising, like how do you incorporate a company?
How do you business ops? Like accounting all those things like,
we're good at that because we've all been through this before.

(29:54):
But there are other challenges like that we kind of
signed up for. Like we started out roll the software
in Python because that was the fact thing to do
to get started, and we needed to run our customers
Python code. But we knew inevitably that that was not
going to scale, that it was not going to be
fast enough, and it has been a herculean project to
rebuild a lot of that software and C plus plus
and Rust, and it would have been simpler to start there,

(30:15):
but then we wouldn't have known whether anyone cared. Like basically,
if you plus for no reason, it doesn't help anyone.

Speaker 1 (30:22):
I had There was a great quote that one of
my early mentors had shared with me a couple of
times where it was it was all about it doesn't
matter if you do that in the in the perfect way,
because you have a need to have it actually deliver
value to your customers, which if you don't have is
normally a challenge. Yeah, I would have personally avoided Python,
but you know, if you're in the machine learning realm,

(30:44):
and you are, it's twenty twenty two, that's pretty much
on the interface around and what people are trying to
use to get running, And if you had started with
Rust at that point, it would have been way too
early on, I think, to actually have the infrastructure available,
but in the space going for performance, and we see
a lot of the database engines being written in rewritten
in RUSS now, which I can definitely see if you're

(31:06):
closer on the database side. That makes a lot of sense.

Speaker 2 (31:08):
Yeah, And so also it turns out that, to my surprise, honestly,
the interrupt between both RUSS, Dancy plus plus and Python
is really fantastic. So it's been fairly achievable to kind
of rip pieces out and rebuild them in the faster language.
Definitely would have been a lot simple to start there.

Speaker 1 (31:26):
Did you have Rust experience before going in here or
was that like we'd have to learn that's.

Speaker 2 (31:31):
Because right, yeah, very little. And I think that's actually
one of the things that's pushed us more towards C
plus plus. Although originally we thought we'd be a Rust
company because there's just so many things that are going
wrong every day that you don't want to add on
the extra layer if I don't even know my programming language.
So again, it's like if I had known Rust at
the very start of the company. Really well, I'm an expert.

(31:52):
I would have just started there, but for lots of reasons,
C plus plus was a better intermediate for us from Python, say,
aside from the complexity the language or the familiarity A lot.
Turns out a lot of database drivers are actually like native,
implent and C plus plus. So if you want like
the full suite of interfaces to things like, you know,
Dynamo dB or whatever, you don't get that through the

(32:14):
Python interface, and maybe the Rust driver doesn't even exist.

Speaker 1 (32:17):
You're integrating with the databases that exist through their APIs directly,
rather than utilizing say the AWSSDK for doing that.

Speaker 2 (32:26):
That's right, yeah, I mean it turns out like DATABASESDK
uses an old version of lib curl and it's really slow.
So if you want like really fast access to most
of the bos data stores, it makes sense to build
your own just a shapey connection to them.

Speaker 1 (32:38):
I think a lot of people either just cringed or
got really excited that you said that, because now they
have a new thing that they can spend all their
time doing. I can make my application faster by getting
rid of the a DOSSDK and replacing it with one
that I wrote myself.

Speaker 2 (32:52):
A lot of this company has been about that exact
kind of intuition breaking. Like I'm a library maximalist, like
I always start with the library, and I'm like, surely
this must work. But it turns out that if you
are trying to count microseconds, a lot of times the
library's written on different constraints, like maybe the author wanted
to autogenerate it, or maybe they wanted to have, you know,
really nice error messages or something. They have some other concerns.

(33:13):
But if literally all you care about is okay, I
have to make this happen in less than two milliseconds
and at a high, like a high through put, weird
stuff starts to creep in and you have to develop
a pains about a lot of layers in the stack.

Speaker 1 (33:25):
And I'm going to get this wrong because it's been
way too long. I'm starting to recall a video about
getting rid of all of the solid principles on software development.
So the set of solid which are like single responsibility principle,
open open closed, go substitution principle. I don't whatever YouTube
video creator basically goes thro each one of them will

(33:46):
be like why is it a mistake from a performance standpoint,
like we did we write better software by following these
and like it's like, well, no, actually we wrote worse software.
It's more confusing, but even worse if you look at
the performance, it's it's not a factor of two or
a fact or three or even ten where you're like, oh,
we'll just ignore it for the most part because we
have a set of ten items or even one hundred items,
but you have a set of one thousand or a million,

(34:07):
and it really makes an impact. There means a good
point because we saw this early on with especially with
JavaScript in AWS the library was managed manually, and then
I think it was like five years ago they switched
to AWS SDK version three where they were using Smithy
to generate all their SDKs. Now and I think it's
gotten significantly worse, not initially from a performance standpoint, but

(34:30):
definitely from a usability one. And it's okay if you're
a massive platform where people are like, we'll do whatever
we have to to make it work, and there is
consistency across languages, it's just not that great. Like if
you know your language really well, you know Python really well,
you know JavaScript, and you go to u ZSK it's
like nothing you've ever used before. But now you're saying also,
you know, security aside, because as is great at that

(34:51):
performance wise, it could be a concern like that didn't
really ever occur to me, Like unless you're interacting with
like val Key or rds, you know my sequel Aurora,
they're customer SDKs. Like you, we're not using ADUs one,
but if you're using one of the Servilus options, you
are probably using the SDK, So that's really surprising.

Speaker 2 (35:08):
Actually yeah, no, I mean actually valky is a great
example of where we had to spend a lot of
time thinking about stuff because like for one of our customers,
we're pushing back and forth gigabytes a second of floats basically,
so we had to think about like the encoding, the
connection pooling policies, the retry policies for intermittent failures, all
of those details. And when there's like two or three
layers of SDK and astraction in the way, it really

(35:31):
messes with your ability to reason about the system in
that in like the intense scenarios, it like works fine
on day one, but it's just you know, day ninety
regretting not understanding how this stuff works.

Speaker 1 (35:40):
You've mentioned that there's quite a few of those lessons
that you've learned, Like does this conversation just go on
for hours with all those lesson learners? Like you know
another one that just like immediately pops out at you.

Speaker 2 (35:50):
Well, I mean another one is probably like I originally
assumed that Python was not that slow, so you know,
I mean it has like a sin kio. You're like, cool,
I can use it to do parallelism. How slow could
adding numbers together be? Like, I have a lot of
experience writing JavaScript, and like it's an interpreted language sort
of like obviously the git's incredible, but if you do
a for loop in some integers, it will not be
that much slower than writing the equivalent program. And see right,

(36:12):
But those feelings don't really hold true for Python. It's
really easy to end up with function calls or the
trivial operations taking single digit or tens of microseconds because
of all the interaction and interpret overhead and the absence
of legit. And if you're trying to do something you
know billions or trillions of times and it takes tens
of micros, all of a sudden you're looking at like
real seconds adding up, which makes entire categories of operations impossible.

(36:36):
So obviously we wrote a lot of software to peoples
plus and rust, but we also try to transpile effectively
our customer software out of Python into an abstract expression representation.

Speaker 1 (36:48):
Wow.

Speaker 2 (36:49):
So that's the whole thing too. So we let people
write transformations over data and the maybe one of the
weird twists is that they want to write those transformations
in Python when they ran on our platform as opposed
to standard sequel. So what we do is we run
this thing which is effectively a symbolic interpreter. It has
roots in Google's type checker, which they just sunsets had,
but we basically evaluate all of their functions symbolically, so

(37:12):
instead actually executing them, we model the control flow and
try to do little proofs about nulity and non zeroness
and positive and negative veneagers and things like that to
prove we understand the semantics of it. And if we
can prove that we fully understand a function all the
associated functions of calls, then we don't need Python to
run it. We can basically interpret it into a different
DSL for executing expressions and then we can run A

(37:35):
plus B from Python or even for loops or conditionals
or like some library code entirely Python free.

Speaker 1 (37:40):
This is normally used for semantic validation of code, for
ensuring that the outputs, especially in the financial domain, are
like reasonable and I think good corollary here is if
you have an ad function, you're not literally going to
pass in every possible integer value or float value into
both parameters and then validate the output. Just not possible.

(38:01):
So instead you write your programming language or your tree
in a way that is very verifiable, and then you
verify it. But you're using it to actually then board
it over and do it not only interoperability, but actually
runtime execution and whatever engine you want. Are you then
running it in roster C plus plus at that point.

Speaker 2 (38:19):
Yeah. So the underlying execution engine for us is vlox,
which is what Facebook uses to accelerate Presto, which is
their internal analytical database. People are also using it to
accelerate Spark. It's called gluten that is implemented in C
plus plus. A lot of the operations are either GPU
accelerated or sin be accelerated. So like DOWNE in the
assembly code. But yeah, it's exactly what you said that.
The point is if you if you understand what a

(38:40):
function does, you don't need to literally run it with
the original programm language was written in. And then for
us the game becomes, you know, making sure we're perfectly sound,
because we're going to take your Python and like not
run Python. We better really know, we better be really
sure we understand exactly what that's doing. And that is
tricky in the presence of a lot of Python constructs
like if people are using you know, like if conditionals

(39:02):
to implicitly check nullity, but also like non falsiness. We
have to represent all the semantics. So it ends up
being kind of like a type checker, and we do
a lot of the stuff that like liquid hasple does
to trap not just like course categories of types, but
also the values within the types. So we can prove that,
like array indexes are inbounds for a list, so we

(39:23):
know that like indexing a list by some in some
integer value, I never throws the index error or something
like that because.

Speaker 1 (39:29):
You already know the length of the array fundamentally at
equivalently compile time of the code.

Speaker 2 (39:35):
Yeah, yeah, we track like we know it's a constant list,
and we know the indexes less than the constant length
the list, et cetera. That sort of stuff.

Speaker 1 (39:40):
You've got a future in so many technology directions here
because not just interacting with the database, but like unit testing,
our code validation, semantic validation. I think there's another one
here that I'm not thinking about the moment, but like
just you can just easily go down any of these
pass as added value for your customers in a way.

Speaker 2 (39:59):
Yeah, that's right. I mean I think like we have
an LSP plug in. I mean it's incredibly slow right now,
so no one uses it, but we're working on making
it faster.

Speaker 1 (40:07):
That's that's the reason no one's using it because it's slow.

Speaker 2 (40:10):
It's like takes forty five seconds to type check right now.
But basically, like the intent and I'm told that we'll
have this fixed in is that like as you're typing in,
your editor will be able to tell you whether or
not we can transpile the program or not. But also
we can we can tell you, oops, you've made a mistake,
because we know that you've like you know, done a
null point or error, so we can let you know
that right there in the editor, and then the kind
of the next thing we're going to start doing is

(40:32):
actually taking the historical values that this function would have
been computed on, Like we know all the values of
all the inputs to the functions ecstatically, so we can
tell you, like as you're typing, what the output distribution
of your function would be, which I think is an
interesting thing for data engineers, because you know, if your
function only produces zero, it's probably not a very useful function.

Speaker 1 (40:49):
Is this something that you've developed for your company that
is prorietary internal or something you've open sourced.

Speaker 2 (40:56):
Yeah, it's proprietary right now. We did a presentation on
how it works at vlock. There's an open question about
exactly how much of this will upstream, but probably not
none of it.

Speaker 1 (41:05):
The question came to me because I do remember someone
asking a number of years ago, like hey, I've got
I mean, it wasn't linear algebra like calculating the solution set,
but it was similar to like, hey, I'm just sort
of curious for these variables that are parameters in my function?
What values input and output? Makes sense? Yeah? And it
sounds a lot like that, like oh, yeah, you know
the output bounds are here's the distribution. You know, we

(41:27):
know it is always been between zero and one or
zero and you know maxin or whatever guaranteed never negative.
And I think there is something to be said about
the value there. I do worry that in today's world
there is more and more companies that are trying to
go faster and sacrificing quality and caring about that. Is
that just my network or you know, do you see
something similar? Yeah?

Speaker 2 (41:48):
Well, I mean it's actually a big support challenge for US.
I mean, every customer we have wants to use Cursor
cloud code to generate all the integration code of US.
And it turns out Curser and cloud Code have a
lot of great ideas for functionality we should build but
have not yet. And suddenly it becomes like a customer
relationship issue for US that you know, they're they're generating

(42:08):
code without verifying that it even matches anything they've everclaimed
to support. I feel that really acutely right now, people
are moving very fast.

Speaker 1 (42:15):
A GMT Polumi two episodes ago told me that they
use that to determine originally some features they should build. Yeah,
because at which which like I already don't like that
we're changing humanity to figure out how to talk to
the LMS rather than improving the LMS to understand you know,
us individually. But now we're also letting the LMS aside

(42:35):
which features we should have, just because they're continuing to
lie to our customers about having that feature, even if
it doesn't necessarily make sense.

Speaker 2 (42:43):
Yes, I think that we are starting to evaluate all
of the dsls and things like that that we expose
by like does this work with cursor or not? So, yeah,
it's it's definitely a new kind of dynamic in that
product management.

Speaker 1 (42:57):
Would it make sense to take your I mean, obviously
you're a small, small team in a small company. You know,
still startup three years old. You know, you have the
whole runway in front of you basically of whether or
not it's going to be successful. So this is probably
incredibly irresponsible of me to ask, But it does seem
like there's a there's another product here where you can
absolutely use your engine building the ASTs to validate the

(43:22):
semantic correctness of the programs that the alums are generating,
and I don't think anyone is doing this effectively right now.

Speaker 2 (43:28):
Yeah, that's definitely something we I mean, even just for
my own stanity, we need to start doing because I
can't keep answering questions about the hallucinated code for the
rest of my life. But I think that kind of
the original inspiration, part of the inspiration for the company
dates back to I can't remember the name of the editor,
but there was that kickstartered editor about ten years ago
where someone wanted to called light Table, where basically people

(43:51):
were like, let's reimagine what an ide could be. I
think a lot of the ideas in there were fascinating, like, Okay,
what if functions were entries in a database instead of
like lines in a file. What if we did track
all the inputs and outputs to functions and productions so
we could show you, like what the distributions were. But also,
if you make a bug, we can tell you immediately
that you're about to ship abug to production. I do
want to get back into the like the core thesis

(44:13):
of Chocolate that we could provide really good data developer
experience for data engineers. And I do want to get
back into the nuts and bolts of like working on
those sorts of problems rather than just you know, fixing
TCP congestion issues admirable.

Speaker 1 (44:24):
Definitely, I approve on solving the technical problems. I mean,
because I feel like in my career it definitely started
out as this lie. We tell ourselves like, we do
software development to solve challenging problems. And I think we
just say that because that was our education experience where
we were forced to solve challenging problems and then we
got credit for it and created this positive feedback loop,
and we thought, that's our personal identity. And then we

(44:46):
go out into the workforce and we keep on repeating that,
and then we learn after some time that we don't
get rewarded or promoted for solving technically challenging problems, and
so we get away from that, and then we have
this lie that transformed into it's all about talking about
our successes, et cetera. Although I think at this point
I'm trying to realize that the original thing isn't such

(45:07):
a lot. Like at a company level, if you build
a great technical solution that solves a real technical problem,
that it will get picked up and start being used,
and the better job you do at it, the more
people that will flock to it. Because, as you said,
you like you won't have any churn. You literally are
the only ones that build this thing and are the
best at building it. Everyone will come and start using it.

(45:28):
And that makes me a little bit happier than believing that, yeah,
some company out there is going to replicate every single
SaaS product that will ever exists. As if it's like
it's going to work. Yeah, sure it may look like
it works, but I don't think it's going to actually
work until they have chalks newly open sourced. I shouldn't
start it because it's not open source yet. Would be

(45:48):
open sourced some part of it ast validator.

Speaker 2 (45:52):
Yeah, one day we'll Bible to approve all software is
correct and solve the whole thing problem, but not today.

Speaker 1 (45:56):
Unfortunately that's not a solvable problem. Actually, and there's a
great Vertalitia video about the whole at the bottom of math,
but that's not going to be my pick for this episode.
So before we get to picks, I'll ask Andrew, is
there anything else that about chalk or about LMS, or
about immutable data structures Python interpretability that you want to

(46:17):
share with our listeners.

Speaker 2 (46:18):
Oh, mutable data structures, I mean I feel like they're
a constant siren you always want to use a mutable
data structure when you are, you know, working in your software,
because it's got all the properties you want right like,
you can't mutate it so you can avoid the thread
safety issues, etcetera, etcetera. But they're just always so slow.
They're just always so slow, so we've always had to

(46:39):
rip them out.

Speaker 1 (46:40):
I would never have guessed that. I mean, I could
imagine that you're a huge fan of like Haskell.

Speaker 2 (46:45):
Then I don't have my high schoolook on my desk
right now. I think way back in high school, like
senior or high school. I found learning you a high
school on the Internet or something like that and absolutely
blew my mind. But I mean, it's the same lie
that Haigh School has to tell, Like I'm presenting a
mutable Internferce Spender the hood as beautable data structures. And unfortunately, my.

Speaker 1 (47:03):
Problem with functional programming was that everyone who tried to
convince me that functional programming was the best thing ever
always had the same message, which is, functional programming is
the best thing ever. There's like monads and immutability, and
that never sold me, honestly, and I never got there.
And after years and years of software development, I came
to the realization that there's a much better marketing sales

(47:27):
pitch for these languages, especially things like Rust or Haskell,
where you can actually look at a function and know
that you've handled every edge case and you don't and
like that. For me, like this was always something I
was like, I hate Java, but I appreciate Java because
it captures the exceptions as part of the function signature.

(47:50):
And until I realized that functional programming is actually baked
this notion in by default into the language, it never
really connected with me. But if anyone had ever said,
aren't you worried that you're missing some edge cases, and
I'd be like, yeah, I am totally worried, then I
would have started with Rust much sooner. I would have
switched to f sharp, probably earlier in my career, and

(48:11):
I think I would be happier for it. So people
who say, yeah, oh Haskell is great for like rule
engines and stuff like that, you know whatever, But yeah,
this this edge case for and you know what we
talked to hear about the ASTs, I think is a
really good justific justification to spend time to switch out
and try some of these alternative solutions.

Speaker 2 (48:28):
Yeah, definitely, I mean checked errors and rust have made
so many categories of bugs and not happen. I sleep
a love better at night.

Speaker 1 (48:35):
I early on in our company started categorizing every single
possible bug that we ever saw in production or caught
and didn't expose in production, and like what caused that?
And there's a number of bugs that are type related
things which are just like, yes, we had the wrong
type or it was too loose or whatever, and we
should fix that by switching to something else. But then
there's a whole class of things of like, no, it's

(48:57):
actually not possible to write the validation in this way,
and switching to a different language would have just prevented
this from ever happening.

Speaker 2 (49:04):
Yeah, absolutely, I mean the then DNS is down and
it didn't matter anyways. It turns out that like DNS
in Kubernetes is not really a very reliable thing unless
you spend a lot of time thinking about it. So
that's been a recurring theme in my life this year.

Speaker 1 (49:18):
I'm horrified that you could even say those words, and
I'm even more concerned to ask for more details. We're
on the cusp of the end of the episode. So
if it's if it's if it's quick and concise.

Speaker 2 (49:29):
Uh oh, I don't. I don't know if networking debuggings
are that quick or concise, but I would everyone's using
cube DNS still, maybe stop, but also check your scaling policy.
It's probably underscaled and gk by.

Speaker 1 (49:40):
De fault and this affects DNS.

Speaker 2 (49:42):
How if you don't have enough replicas, dns will start
to time out when you try to resolve names. So
I mean the classic issues your auto scaling group says, cool,
tend to double the podcast. Everyone boots up and tries
to resolve you know, your database IP address, and then
they fail to resolve your datase hypea address and then
unfortunate things happened as a consequence.

Speaker 1 (49:58):
Absolutely amazing. Okay, So with that, let's switch over to
picks for the episode. Andrew, what did you bring for
us today?

Speaker 2 (50:06):
Yeah? So I had a serious answer, but I think
my silly answer is everyone should get an e bike.
It's a lot of fun. I would recommend getting one
of the e bikes, which looks exactly like a regular
road bike, so that when you're going up a hill,
all the you know, really extremely serious cyclists get jealous
and you can hear their gears change as you go by.
It just goes chunk ka chunk ka, chunk goes. They

(50:27):
all cry to catch up.

Speaker 1 (50:28):
You've had this experience multiple times?

Speaker 2 (50:30):
Yeah, of course, it's my favorite weekend activity.

Speaker 1 (50:33):
Is this a newfound hobby of yours or one that
you've been living with since the invention of e bikes?

Speaker 2 (50:38):
Yeah. So I've actually been really into road cycling for
a long time, but my wife bought an e bike
recently so that we could kind of keep pace better
on rides together. She doesn't want to get super beat
up and I do, but we've been doing that basically
every weekend for maybe the past six months.

Speaker 1 (50:53):
What sort of e bike do you have?

Speaker 2 (50:55):
She has a truck e bike. It's like a truck
to money or something like that. But it looks exactly
like a regular road bike, so it's it passes the
stealth check.

Speaker 1 (51:03):
Okay. I was racking my brain what to pick today,
but since I'm traveling to the US, I decided to
bring some Swiss chocolate. And I have to say that
something about Swiss chocolate. Since I'm originally for the United States,
I thought I just always hated chocolate. I have to
say that it's because the chocolate is bad, honestly. And
if you're ever in Switzerland and you come here and
you're like, I want to get some really great chocolate,

(51:24):
the number one thing you should not do is go
to a famous like chocolate manufacturer, or go to a
special store that's shrouded in controversy. Instead, just like go
to your local grocery store and just grab a random one.
It will be absolutely fantastic. So my pick today is
actually going to be the grocery store Migro, which is
like one of two major grocery stores in Switzerland, and

(51:44):
they have a special brand called Frey and this is
like the pistachio version, and it's like, honestly the best
chocolate I've ever had. And it's like you just go
to the store and you just buy it. It's there's
no like special dance you have to do. It's not
like a special company or anything. And the interesting thing
is like their brand that actually produces this is like
totally social conscious and concerned about environmental protection laws and

(52:06):
where the cocoa has grown and manufactured and assembled and
everything like that, so you can be sure that no
corners are being cut there.

Speaker 2 (52:12):
Well, I got a mood to Special Land. It's the dream.

Speaker 1 (52:14):
The second best time is now. All right, So that's
it for today's episode. Thank you Andrew for coming and
joining us and talking through all those really interesting technical
challenges that you had.

Speaker 2 (52:25):
Of course, thanks Warren, I really appreciate you having me.

Speaker 1 (52:27):
Yeah, I mean, this has been a great conversation and
we'll be back again next week.