December 3, 2025 • 43 mins
What if changing a single flag could save you from a failed migration, a broken API, or a late-night rollback?
Join us as we dive into how feature flags become a practical tool for changing application behavior at runtime, not just toggling UI elements. Cory talks Mike Zorn about real stories from LaunchDarkly and Rippling, covering how teams use flags to ship safely, debug faster, and simplify complex systems.
You’ll hear about:
- Using feature flags to avoid staging overload and ship directly to production
- Migrating critical systems and databases with minimal downtime and risk
- Controlling log levels and rate limits for specific customers on the fly
- Managing flag sprawl so teams do not drown in half-rolled-out features
- Experimenting with AI features, prompts, and models without fully committing
If you’re working on a platform, running critical infrastructure, or just trying to ship faster without breaking everything, this conversation offers concrete patterns you can start using right away.
Guest: Mike Zorn, Senior Software Engineer at Rippling
Mike’s software engineering journey began with an early interest in problem-solving and programming, starting with creating programs on a TI-83 calculator in middle school. After studying mathematics in college, he transitioned into software through an applied math project that required coding, which sparked his interest in engineering as a career.
Professionally, he has worked at several product and SaaS companies, including one that was an early LaunchDarkly customer, where they experienced firsthand the challenges of managing feature flags internally. That experience led him to appreciate the value of tools like LaunchDarkly, eventually joining the company himself. Since then, he has contributed across various areas, including focusing on how LaunchDarkly can best adopt its own platform internally to streamline releases and help engineers work more efficiently. His latest adventure has been joining Rippling as a Senior Staff Software Engineer.
Links to interesting things from this episode:
- SigNoz
- Signadot
- Open Container Initiative
- “Using Feature Flags to Avoid Downtime During Migrations”
- Apache Iceberg
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:13):
Welcome back to the PlatformEngineering Podcast. I'm your host,
Cory O'Daniel, and today Ihave with me Mike Zorn, senior staff
engineer at Rippling. Mike andI met, geez, maybe a month or two
ago at a Honeycomb conference.You were working at LaunchDarkly
at the time. A little bitabout Mike's journey - he started
out in middle school doingTI-83 calculator programming, that's
(00:35):
right?
That's right.
We've got some stuff to talkabout there because I used that calculator
to do a ton of cheating incollege. He's carried the love of
problemproblem-solvingsolvingindifferentrolesin SaaS
companies, from LaunchDarklingto Rippling. Now he's helping teams
build platforms. And so happyto have Mike on the show today. Mike,
thanks for coming on and sorryabout the reschedule.
(00:57):
Oh, no worries. I'm here now,so it works out great.
Awesome. So today I think it'sgoing to be a fun conversation. We're
going to talk about platforms,feature flags, I might try to bug
you a little bit about the AWSoutage recently. But before we hop
into it, I just want to take amoment to say thanks for Kelsey Hightower
(01:17):
hosting the last threeepisodes. Really appreciated that.
Had some really awesomesponsors, SigNoz and Signadot, working
on those episodes. So thanksagain, Kelsey, and apologies to everybody
who had to hear me talk aboutwhat I do during my day job for three
episodes. But it was a funcombo with Kelsey. So, Mike, tell
us a little bit about yourbackground. Like, how did you get
into software? Maybe talkabout these TI-83 shenanigans and
(01:41):
what led you to Rippling?
The first time I wrote acomputer program was definitely on
the TI-83. That's where itstarted. I was initially using...
it was like, "Oh, let me seeif I can make blackjack game."
And then I was able to dothat. That worked great. But then
there was one later, I triedto make like a poker game.
(02:02):
Yeah.
And, you know, I was in middleschool, I didn't know anything about
programming, so I didn't knowthat, like, arrays existed. So I
was like, "Oh, I need to,like, put all the cards in the deck
into variables. But I've onlygot 26 variables from, like, the
letters of the alphabet." AndI was like, "Shoot, I guess you can't
make this program." So thatwas, you know, my first of many programming
(02:26):
failures that I Learned fromlater. I kind of got more serious
into software in college. Istudied math. I was doing, you know,
some computer programs tosolve some math problems and was
like, "Oh, this is like apretty enjoyable thing - to write
(02:46):
a computer program that, like,does this specific task for me."
So when I finished that degreeprogram, I was like, "Oh, you know,
there's actually a lot morejobs in software than there are in
math, so maybe I should trthe software thing out." And it's
been a fun ride for the last,I don't know, like, more than 10
years. I don't really likethinking about how big that number
(03:08):
is now.
It's getting big. It's getting big.
Yeah, I've got a lot of grayhairs now. But in any case, yeah,
I got maybe more into platformtypes stuff as I was working at LaunchDarkly...
(03:30):
o I mean, the job rightbefore LaunchDarkly where I took
ownership of our internalfeature flagging system and then
replace it with LaunchDarkly.And saw how think,moreefficient.a
That's fun. I didn't realizethat you worked on a feature flagging
(03:52):
tool before you went toLaunchDarkly. That's extra fun for
me because I feel like whenyou talk to somebody about the product
that they work on day in, dayout, there's obviously passion for
the idea... but you'veactually experienced both sides of
it. Trying to build onein-house, which I've also experienced
that and was like, "Wow, I'mvery glad things like LaunchDarkly
exist so I don't have to dothat anymore." It is a bit more than
(04:16):
just a side project somebodycould work on at a company, right? There's
a lot to feature flagging,more than just is this thing true
or false, yes?
Yeah, yeah. It starts out asis this thing true or false? But
then you get Customer Successpeople that are like, "Hey, can I
(04:36):
turn this on just for onecustomer? They want to try it out."
And then it's like, "Oh, okay,well, now it's not just true or false.
It's true or false for somepeople, but not for others." And
then you build that for themand then you realize, like, you've
created a nightmare foryourself. Because now you have to
turn things on and off fordifferent customers because you don't
(04:58):
want to give Customer Successpeople - as great as they are - direct
access to one of your proddatabases to make these changes.
And you're like, "Okay, now Ineed like a UI or something to enable
that." And then you're like,"Okay, now I'm personally maintaining
like a minor product." At thatpoint it's like, "Well, maybe I just
(05:19):
buy this thing from somebodyelse." So that was my journey to
that company.
That's an interesting one too,because I feel like, you know, looking
at like the... we have a lotof classes of different tools in
the software engineering spaceand I think obviously, or maybe not
obviously... but the earlieryou adopt some of these tools, like
(05:42):
the better patterns you getin. But there's obviously weight
and time and effort toadopting tools like CI/CD, ephemeral
environments, feature flags.There's so many ways to build and
deliver and deploy goodquality software and I feel liklike,ehonestly,oneofthe
places where you can yield themost out of good quality software
(06:04):
early without a ton offriction is adopting feature flagging.
I feel like I've seen so manyteams that struggle around staging
swim lanes and all sorts ofstuff like that. And there's so much
pain and there's just so mucheffort and delivery of software that
can be resolved by starting tolean into feature flags early. And
(06:25):
I feel like it's one of thosethings you don't see enough teams
lean into until they starthitting that later stage, big teams.
And it's like the stuff youcan get out of day one, just like
MVP testing, being able to getsomething out there and test it just
a little bit in like a realproduction-like environment is something
that you can really yieldbenefits from as an early stage company.
At LaunchDarkly, what was thecustomer base like? Were you getting
(06:47):
a lot more mature customers?Or like, were you seeing earlier
stage companies start torealize like the benefits that they
could get out of featureflagging and start to lean into it?
Is it still something that isnot really realized until companies
are at like much later stages?
So I think it's prettyvariable. So, you know, LaunchDarkly
has a ton of really bigcustomers and working as an engineer
(07:10):
there, those are kind of yourfocus because you bring in the most
money. But there also are aton of, you know, smaller customers.
This company I was at beforeLaunchDarkly wa kind of one of those
customers. I was there fromlike Series A through B. And this
was back when that was not alot of... you know, series A was
(07:30):
like a few million dollars,not like $30 million.
Yeah.
And that company... the CTO,made a pretty good decision early
to, like, have feature flagsbe a core concept. The way we developed
software there was, like, justvery easy because we didn't have
staging environments oranything like this. It was like you
(07:53):
just shipped your new thingbehind a flag. It was in production
within, like, a minute afteryou merged, and then you turn it
on for yourself to test itout. If it didn't work, you'd like,
you know, make some changesand, like, that'd be your feedback.
You'd still have a reallytight feedback loop like that. You
know, that company didn't turnout for reasons that were not really
related to our engineeringefficiency, but, you know, we had
(08:18):
a really efficient engineeringteam when it came to, like, our feedback
loop and how we developedsoftware. I'm a little bit removed
from LaunchDarkly now, so Idon't know exactly what the customer
makeup, the real customermakeup is, versus the one I perceive
from the support requests thatgot escalated.
Yeah, no, that's fine. But Imean, you were seeing, like, earlier
stage companies start to adoptfeature flags though, there. Yeah?
(08:43):
Yeah, yeah, yeah. And I thinkin the early days of LaunchDarkly...
like, yo know, it's a companywhere you put the code, like, really
deep in your own code base, soit's really hard to take out. So
churn has alway been prettylow at LaunchDarkly. There was a
while where the leading sourceof churn were the smal companies
(09:05):
that adopted us that like hadjust ran out of runway and didn't
take off.
Oh, yeah, yeah. That happenswith startups. Bee there, done that.
Yeah, yeah, yeah.
They're hopefully not doingthat right now.
Yeah.
It's funny, I feel like it'sone of those places that is so easy
(09:27):
to get it in. Right? I mean,like, when you have an MVP out, I
think this is one of those,like, technologies also has, like,
staying power. Right. Like,you know, you don't meet a project
manager at Series C who'slike, man, I really wish we didn't
have feature flags. Right. AndI feel like it's one of those things
that's so easy to get inearly, and it just helps you ship
better quality product andalso, like, maintain those cloud
costs. Like you're saying,it's like not having a staging environment.
(09:49):
It's amazing. Right nowthere's a bit more rigor to like
using feature flags, likeresponsibly. I would love to like,
you know, hear some of yourthoughts on that. But like that same
rigor is used to set up andmaintain that staging environment
and that thing costs you moneyall day long. Right. So like being
able to get stuff out in theproduction I think is really, really
valuable. That being said, itis deeper in your code. What are
(10:11):
some of the ways that you'veseen platform teams start to embrace
feature flagging? Maybe whatyou're doing now at rippling and
how do you I guess instillgood practices in a team for where
to use them and how to usethem to cut out a feature and how
to clean them up efficiently?One of the things I've seen is some
(10:32):
teams just tend to use themvery much on the front end and they
say, hey, we built the featureout, you can actually hit it if you
hit our API, but we hide itfrom you using a feature flag. And
I've seen other teams whereit's like they feature flag down
the whole stack. You can't seeit in the UI and you can't access
the endpoint unless you're inthe flag. Right. And it's like kind
of different approaches tolike how you want to scope this stuff.
But like how are you seeingplatform teams use this internally?
(10:56):
Are they using it, are theysurfacing it to their end user customers?
And what are some of thepractices you're seeing that work
well with feature flags?
Yeah, okay, so there's a lot,sorry, those questions.
Okay, I like it.
So I'm probably going to missa couple things and we'll maybe revisit
them, we'll dig in. But Ithink yeah, there's a lot of different
(11:17):
ways to use flags and I thinkyou touched on a few of them and
all of them are prettyreasonable. It just kind of depends
on like what type of productyou have. Like, I think that a really
simple way to use flags is tojust use them in the front end and
like ship APIs that aren'tused unless you are calling them
from the front end. And thatworks great if you know, you don't
(11:39):
have any concerns about likeyour customers spelunking into your
APIs and trying just like todo random curl requests and you have
like a really UI drivenproduct. Like that's a really good
approach. I think that oneplace where the, you know, when you
start thinking about like fullstack flagging and like using flags
(11:59):
on the back end, a place wherethis comes in is a lot of value.
Is like whenever you're doingsome kind of like performance optimization,
you can now be really rigorousabout it because like if you're like
this query is reallyimportant, that the performance is
like uniformly better. Yeah.And I'm like spending a bunch of
time optimizing this querywith a flag. You can like really
(12:23):
do like an actual like a Btest type experiment and see how
the query latency is changedlike with live traffic. And if it
doesn't have the result youexpected, like you can turn it off
really quickly. So those typeof things are I think super powerful
for back end flagging. And ifyou're on a team that is like kind
of, you know, responsible forthe performance of some kind of like
(12:49):
shared database or somethinglike this, it's a really, really
effective tool because you notonly get really kind of like high
fidelity measurement of livetraffic, but you also have the ability
to react super quickly ifsomething isn't as you expected.
You don't have to do aredeploy or anything like this. So
(13:10):
yeah, I think that queryoptimization is like a good, it's
kind of a canonical example ofkind of like a backend flag.
You know what's funny aboutthat? I've used feature flagging
for years. I've never thoughtto use it around like optimizing
a query that you're not. It'sfunny because like my background
is in databases. I've got aton of postgres experience. I've
just never even thought aboutit along like the read level.
(13:34):
Right.
Like, besides like showinglike a new thing but like having
an alternate version of like aSQL query that's like, hey, we're
trying to optimize this, wedon't know how this is going to work
in producing and like beingable to flip between that is actually
pretty cool. And I've gotmultiple things I'm working on right
now in my day job that I'mlike, we're going to push this to
prod and hope it performswell, that I might have to take a
(13:55):
gander. One of the things thatwe've actually seen that works that
we've used is like we've had acouple of times where we've done
a lot of like consolidation asa startup. Like we have a lot of
changes happen to our softwareand we've had a couple of like subsystem
migrations. For example, weoriginally built our registry from
scratch. We're a registry thatstores like, you know, NPM modules,
(14:17):
stuff like that. We built oursfrom scratch. We later rebuilt it
on top of like the OCIstandard. And this is one of those
places where we got to usefeature flags. When we rolled out
our OCI stuff, there was asubset of the system that like, went
to OCI versus our oldregistrya,andndwehad issues with
ita,andndwewere able to justturn that flag off and everybody
just went back to the oldsystem. We were kind of like dual
(14:38):
writing, so we're puttingeverything in both places. But it
was one of those places whereit was like doing a migration. Right?
It's one thing to turn afeature on and off, but like, being
able to like, migrate a systemlive is also a very harrowing thing
to do. And to do that nowadayswithout a feature flag, like, I don't
know how I wouldn't go baldfrom the stress.
(14:58):
Oh, yeah, yeah, yeah, yeah.Even with the flag, you can still
get a little stressed. Severalyears ago, you know, we needed to
kind of like change the entiredata ingest layer of LaunchDarkly.
And you know, we used afeature flag to do the migration.
It was like, there's this sortof daily ritual I had where I was
like, okay, let's expose like5 more percent of traffic into th
(15:20):
new data ingest system thatlike has some scale issues that we're
working through. And it wouldbe like, okay, turn on 5% and then
like, stare at these graphsfor like 30 minutes and see if anything
goes up and to the rightreally quick, like abruptly, and
if so, then like, roll itback. But that was just like a ritual
I had for a while of that was,you know... it was a little stressful,
(15:43):
but it would have been way,way worse if it weren't that. I actually
wrote a talk about that andthen I compared it to like... in
this Bay Area, there's likethe San Francisco-Oakland Bay Bridgean,anddtheyreplaced
one of the spans like, like 10years ago now. And when they did
(16:04):
that, they had to like, theybuilt like a whole second bridge
next to the bridge and thenthey'd close the bridge for a long
weekend and reroute all thelanes between it. And if anything
was wrong, it was just like,"Oh, this crucial artery is just
out forever," or not forever,but for a significant amount of time.
So the amount of testing andstuff they had to do to get the confidence
(16:25):
that that would work, I mean,it's a bridge, so they were going
to do it anyways, I guess. Butit's Just interesting to compare
how things operate in realworld infrastructure and software
infrastructure and how muchmore control you get and how much
faster you're able to move asa result.
And so for platform teams, Ithink this is another place where
(16:48):
like I said earlier, I feellike feature flags are something
that is beneficial to any teamthat has the time to adopt them.
And I feel like there doesn'tget to a point where you're like,
"Man, we should get rid ofthis thing." Thinking of teams as
they kind of mature and scalelike up the DevOps maturity model
and they start moving towardsplatform engineering. I feel like
this is one of those places...like I feel like a platform team
in and of itself can be aforce multiplier, but this is one
(17:10):
of those technologies I feellike can yield a ton of values for
engineering velocity, not justfor getting stuff to prod, but I
mean like (A) surfacingfeature flags as a feature of your
platform, but (B) just usingfeature flags in the platform in
and of itself, whether it'slike releasing new features or pathways
to your engineers or evendoing things like load shedding some
(17:31):
of their services. Like whatare some of the interesting ways
that you've seen platformteams kind of adopt feature flags
in ways that aren't like thetraditional sense of like just hiding
a UI element or enabling afeature in a backend?
Rate limiting and logging arekind of, I think, two places. Logging
is may be a little bit moreuniversal where like it's pretty
(17:56):
often like if you've got asystem with some throughput, like
you're not logging at likeinfo level in your whole application.
And if you've got somecustomer that's doing something weird,
you often need like, "Oh, Ijust need to see those info and debug
logs for this one personbecause they're having a weird experience
(18:18):
and I don't have enoughinformation to figure it out." A
really interesting use offeature flags is to basically change
your log levels for a givenuser kind of in real time. So like,
you know, this person's havingan issue, you're talking to them
on support, you can increaseyour log level for them and then
now you get all the stuff thatyou needed to, to see what the problem
(18:40):
was.
That's pretty cool. We dosomething today, we lean pretty heavy
into OTel and so we actuallyhave an attribute in our OTel that
like what we can essentiallymark that like something has happened
in like the, just the fullOTel process that's gone wrong. And
we actually change our errorlevel if we see that, so we can actually
log some more information. Butit's interesting. There's been times
(19:00):
where we've run into issueswhere it's like, there is no error
- there's no error, but it'slike something weird is still going
on. No exception has happened,but something weird is going on.
And we're like, "Damn, we needsome extra information, but how do
we signal to the system thatdoesn't think anything is going wrong
that we need additionalinformation?" And how we do that
is we redeploy and change ourlog level - which sucks, it's so
(19:21):
primitive. That is tight. Thatis sweet. Okay, sorry, continue,
I'm bookmarking that one.
After working at LaunchDarklyfor so long, I've come to believe
that feature flags are notfeature flags, but they're really
just a tool for changing yourapplication behavior at runtime without
doing a deploy.
Yeah.
And when you think of them inthat general way, there's a lot of
(19:43):
weird stuff you could do. Someof it good, some of it bad. I think
the logging one is pretty justfine. If you have that philosophy
around feature flags, anotherthing that we used at LaunchDarkly
at least was rate limiting. Sosometimes you have a rate limit you
want to apply acrosseverybody, but every now and then
you have some importantcustomer that's like, "Oh, I'll give
(20:05):
you a little bit of moneyextra or something to get a higher
rate limit." And being able tochange that through feature flags
is pretty nice because if youalready are rolling out features
for a customer, you can justkind of use the same way to reconfigure
rate limits or something likethis for them.
Yeah. And for people thathaven't used a more advanced feature
flagging system likeLaunchDarkly, a lot of the feature
(20:27):
flagging systems nowadays,it's not just like a, "Hey, this
email address..." or "Hey, isthis thing true or false?" You can
actually push data into themas well. So you can say like, "Hey,
this is the email address andthey happen to be of this org that
has this SLA." So you canactually make the flags a bit more
dynamic because you canactually push data into them. Sorry,
just a little note there.
Oh, yeah. That's a goodcomment because I've been in the
(20:51):
game too long t remember thatthat isn't like the normal baseline.
I mean, they weren't like thatfive or six years ago... seven or
eight years ago, right? I thin I saw that feature... I can't
remember the first platform Isaw that and I was like, "Oh, this
makes so many things so muchmore easier." Because it's like,
"Oh, I can give you somecontext to make the decision versus,
like, have it being so Boolean."
(21:13):
Yeah.
Ops teams, you're probablyused to doing all the heavy lifting
when it comes toinfrastructure as code wrangling
root modules, CI/CD scriptsand Terraform, just to keep things
moving along. What if yourdevelopers could just diagram what
they want and you still gotall the control and visibility you
need? That'sexactly whatMassdriver does. Ops teams upload
your trusted infrastructure ascode modules to our registry.Your
(21:33):
developers, they don't have totouch Terraform, build root modules,
or even copy a single line ofCI/CD scripts. They just diagram
(21:55):
their cloud infrastructure.Massdriver pulls the modules and
deploys exactly what's ontheir canvas. The result? It'sstill
managed as code, but withcomplete audit trails, rollbacks,
preview environments and costcontrols. You'll see exactly who's
using what, where and whatresources they're producing, all
without the chaos. Stop doingtwice the work. Startmaking Infrastructure
as Code simpler withMassdriver. Learn more at Massdriver.cloud.
So now you're at Rippling andyou're actually working on a platform
(22:18):
team. Are you already startingto use feature flags? Is it something
that was already like apattern there, or is that something
you're introducing in your new role?
Oh, yeah. No, no, no. SoRippling actually has been a LaunchDarkly
customer for a while.
Ooh, spicy.
So, yeah, they've had featureflags adopted for a while there.
Rippling actually, like,has... I'm still kind of undecided
(22:40):
about what I like to thinkabout this pattern but... they have
an interesting way formanaging having too many flags. Because
there's a lot of engineeringteams at Rippling, a lot of people
are working in kind of likethis giant monolith code base, and
there's a ton of flags in thatcode base. That leads to sometimes
(23:03):
unpredictable and surprisingbehavior when there are flags left
kind of in a partially rolledout state and somebody forgets to
fully roll out a feature orsomething like this. So they're pretty
mature in their adoption offlags to the point where they have
so many flags it's causingsome problems.
Oh, wow.
It's like thousands andthousands of flags. I think, on average,
(23:25):
each developer has like fourflags or something like this... and
that's after implementing thisprocess that's like driven it down
pretty significantly. Theprocess is that each team gets a
cap on how many partiallyrolled out flags they can have.
Okay.
So, like, if you have too manyflags that are partially rolled out,
you can't create a new flag todo a new rollout. Which is the way
(23:48):
we've sold it. We've tried tomitigate the issue at Rippling, and
it's been pretty effective. Ifyou look at the charts of, like,
how many partially rolled outflags we have, th cap is being respected.
But yeah, they definitely areusing flags a lot. We're using flags
a lot there.
Oh, man, you got me. We had...geez, this was like two years ago.
(24:13):
We had redone a major, like,replatforming, and we were kind of
doing the thing where... Ithink people use feature flags a
lot for this... where you wantto opt into the new UI, right? And
it's like, "Oh, I don't likeit. I want to go back to the old
one." Well, we hadn't got tothat level of maturity yet where
we're ready for customers topick whether or not they wanted the
(24:34):
UI. And internally, we'reusing it. We're like, "This is great.
This is awesome." And we put afew of, like, our bigger customers
into it, and then we justforgot that there was a feature flag,
like, gating off this wholenew great experience of the old experience.
And we just had a while wherepeople were signing up for the platform
and they were like, "Thisthing's a real piece of shit." And
we're like, "What are youtalking about? We overhauled the
shit out of this thing for,like, months." And we just forgot,
(24:57):
like... all of our payingcustomers were in the flag, everybody
internally was in the flag,but no new customers were in the
flag. I think that is like oneof the two big things that I'm always
watching for with flags is (A)we have 100% flags that are rolled
out that we forgot to cleanup, and they're like, littered across
(25:17):
the code base. And (B), thisexact thing of, like, you have a
flag, you've rolled it out to,like, a high percent people, but
not to everybody, and you'vejust forgotten about it. Right? So
you said you kind of have apattern now where it's like, max
four flags per person. Howhave you seen people solve this in
the past when you're atLaunchDarkly? And how are you thinking
about it now with the teamthat has such a high number of flags
(25:38):
in flight?
Yeah. So I think that, andthis is maybe a place where platform
teams can come in... isintegrating flags with observability
is pretty important. What Imean by that is like, if you have
spans where like flags areevaluated, you probably want to know
what the value of that flagwas. And then you can like create
(26:02):
monitors to let you know,like, are, you know, people of this
segment, you know, seeing theexperience that you expect they are.
And the thing is like, gettingthat type of observability to be
standardized across your stackis something that is really, really
nice. That was something thatwe had set up really well at LaunchDarkly,
(26:23):
where you know, like prettymuch any, any flag being evaluated,
we would record kind of likethe context that we were evaluating
flags with and like the flagvalue served like, and it would just
be like annotated on, on thetrace. If you have an observability
tool that lets you, you know,run queries across traces and spans,
(26:43):
then you can figure out prettyeasily like what fraction of people
are seeing this feature. Andthen if you're rolling out something
new, you can kind of keepyourself honest by looking at such
a dashboard or something whenyou're doing the rollout and realizing
like, "Oh, wait a second,there's like 20% of people that are
still seeing the oldexperience. I probably need to go
(27:04):
finish this."
One thing that I've seenthat's kind of an interesting way
to use feature flags alongwith OTel that we do internally that
we've really grown to like.We're an early stage startup, so
our API is rapidly changing,adding functionality as we're kind
of stabilizing the business.And at the same time we don't want
(27:28):
to release a million versionsof our API. We also don't want to
break stuff for customers. Andso we kind of have this sort of fluid
live API. And so what we do iscurrently we mark essentially like
fields on our API. We useGraphQL, so fields, queries, mutations,
we'll mark them as deprecated,like when we're ready to start deprecating.
And that'll remove it from ourdocs, which is cool. So like nobody's
(27:51):
going to learn about it andstart using it. And then what we
do is we actually put thosedeprecated accesses on our OTel traces
so we can actually see in OTelwe have a dashboard of like who's
still using deprecatedfeatures. So this is obviously only
people that are actively usingit. But like our, our activity is
pretty high so it's likepeople are using it for deploying
their software every day. Soit's like we're not seeing customers
(28:12):
leave for a month and thencome back to use something. It's
like it's integratedconstantly, right? So we can see
people kind of using thesedeprecated fields. We'll email people,
say, hey, we got this betterexperience, like, start using this
or upgrade your CLI, etcetera. But one of the things that
we do like, as a part of ourdeprecation process is use feature
flags. So when we actually getto the point where we're ready to
fully deprecate something, weactually feature flag people out
(28:34):
of it so that if we do findsomebody, for whatever reason was
using a feature that wedeprecated and they're still dependent
on it, we can actually bringthat back because we see there start
to be some essentially SLOthat's being violated. Sorry, SLO.
Gosh, I mix up my SLA's, SLI'sand SLO's, I know that makes me a
(28:55):
bad person, but dyslexia,we'll start to see that there's errors
there and it's around thedeprecated field and we'll actually
bring it back in for thatperson so that we're not breaking
their software. And then we'llbe like, hey, like, you know, we
had a deprecation window forthis. Like, we enabled it back for
you, right? And so it allowsus to kind of like get people off
of that and make sure peoplearen't using something deprecated
(29:16):
and kind of like slowly likewean off the customer base. And then
we'll just have like a verysmall portion of people that are
able to use it. And then we,as we turn that off and like work
through like the deprecation,we're like, ah, we were able to fully
remove it, but like, if weever get to that point where it's
like a week later, it's thisweird feature and somebody's using
it again, it'll fail, but thenlike flip back on. So it's been a
really interesting way tolike, kind of control not just the
(29:38):
introduction of a feature, butlike the retirement of a feature.
Oh, yeah, definitely. And thatwas, yeah, we employed a pretty similar
pattern at LaunchDarkly whenit was like, whenever we were making
like an API change that wouldhave to be breaking in some way.
You know, you can use flagsfor this and then identify customers
(29:59):
where like, you think theremight be an issue and kind of turn
it off for everyone that isn'tin that group and then work with
the people in that group asit's practical to deprecate it further
for them and kind of winnow itdown until you can actually turn
off, delete the code. Yeah.
So at LaunchDarkly, so yousaid earlier you had a feature flag
on that ingest system. DoesLaunchDarkly dog food its own live
(30:23):
system? Is that how thatworks? Or do you have a separate
LaunchDarkly for LaunchDarkly?
We basically have a separate.Or LaunchDarkly basically has a separate
LaunchDarkly for Launchdarkly.And the reason for that is like,
it's like an exact copy of thecode and everything. It's just like
a second environment thatLaunchDarkly deploys to after the
(30:44):
normal production environment.
Okay.
Because, you know, if there islike, you know, you turn on a feature
flag that brings downLaunchDarkly's UI or something and
you can't turn it off, that'spretty bad. That's why there's the
second one. Yeah, that's cool.
(31:05):
I love seeing teams dog foodtheir own things. But there is certain
classes of software where itgets like. It's one thing to be like,
hey, we're an accountingcompany, we use our own software
to account. But it's like,it's the thing that manages releasing
it. It always gets tricky. Sothis is a full second clone sitting
there that you all areactually using internally. You got
to use the full suite offunctionality then, which is pretty
(31:27):
rad.
Well, it was rad and not radbecause I was able to use all the
functionality, but we also hadto keep healthy. A whole second instance
of LaunchDarkly. Yeah, a wholenother set of computers are gonna
have problems. So as being anon call person periodically, that
would be sometimes annoying.But it was, yeah, it was definitely
(31:50):
nice to have every... al the functionality.
That's not a trivial system tosupport either. Like LaunchDarkly
is like you're in the hot pathof a lot of very, very big companies. So
even a feature flagging systemoperating a feature flagging system
is still, I'd assume, handlinga ton of traffic.
Yeah, it's always like, kindof amazing to just like when I was
(32:11):
working there, see the numbersof like how many flags are being
evaluated and how many likeconcurrent clients are connected
to the service. But then itwould also be like, "Okay, you're
on call for this." And youknow, the impact of a LaunchDarkly
outage is pretty, prettyhigh. So it was always like, you
know, on call, I took itpretty seriously.
(32:32):
Yeah. So how are you seeingteams so like, you know, I know.
I think a lot of ops andplatform teams have different takes
on the extent in which AIshould be involved in our work. But
like, how, how are you seeingpeople start to use AI in and around
feature flags? And like, whatis, I guess the like, state of the
(32:54):
art, as it were? If ther canbe a state of the art, I feel like
the state of the art ischanging so quick.
But yeah, so I think there'slike a couple of different directions
here. Like, one is like, howare teams that are building kind
of like AI products usingflags? I think that there's some
variance there. And thenthere's also the question of like,
(33:16):
you know, teams are justbuilding normal software. How are
they using flags like asthey're using, like having AI help
them write code?
Yeah.
You know, for the, the secondone's maybe more generally applicable.
So we start there. But apattern that I have seen is where
people will just kind of likeput a flag around, you know, the
code you're writing with AIand you know, just develop the feature
kind of like, like normal, butthen you, you know, you have, you
(33:39):
know, you have that, that flagaround it. So if you. Something doesn't
work out, you're all good. Andsimilarly in that additional level
of control lets you kind ofdelegate more to an AI coding agent
or something like this to havethem produce a feature, like the
agent produce a feature andput it wrapped in a flag is definitely
(34:00):
something I've seen. And thenon the other side of people that
are building AI products,like, I think observability for AI
products has kind of been aninteresting place where it's hard
to always know, like there'snot like a, an error that's going
to be logged if you get like anot great response from your like
(34:23):
LLM call. So there's, youknow, a lot of products that are
made to help you figure outlike based, like offline, like, you
know, you had all theseevaluations and then there's a lot
of products that help youfigure out were these good or bad.
And one tool that it seemslike more mature teams in that space
use is experimentation to, asthey make a change to how they're
(34:46):
calling the model or whichmodel they're using, they'll use
just normal experimentation,A/B testin as kind of like the final
step of a release becausethat's kind of like the best signal
you'll get. Is did this changehave the effect on user behavior
(35:07):
that I anticipated? Andexperimentation is a really good
tool for that. So those are acouple of ways that people use it.
It's interesting too becausewhen the new models come out and
it's like everybody kind ofrecoils because it's like got so
much dumber, lost all mywhatever. But yeah, I feel like when
you've spent so much effortand I feel like it is a lot of effort
(35:29):
to get a good prompt, it'slike everybody's like, oh, why don't
you just AI it? It's like,well the prompt is so hard. Yeah,
but like once you, once you'vegot a good prompt and like a good
set of agents like configuredand like working together and then
a model changes like oof,that's disruptive. Right. And it's
hard. Kind of going back towhat I was saying earlier, like how
we use it, like migratesubsystems. I think that's a really
(35:50):
great place to be able to swapout like subsets of users. Say like,
hey, what is, what does itlook like for us with these prompts
and agents on the new model?Right? Just being able to see like
how much it has changed andlike slowly rolling your customers
onto that new model before youjust kind of like whole shebang,
like hey, let me just changethe model name in this config and
deploy to production and seehow different this is for everybody.
(36:13):
Right? Like we do have a bigbag of nondeterminon-determinismnismandthisisoneofthoseone
placeswhereit's likeyoucanexhibitsomecontrolover it. I think
that in the age of AI, thesefeature flags become even more important.
Another place I've seen itused a little bit is internally is
around the actual prompts,controlling the prompts themselves
(36:36):
through the feature flags. Andsomebody might be saying why. It's
like it's very easy to getyour rigging into prod, but this
one of those things that alsokind of sucks a bit. It's like your
dat that you have locally,like while you're testing a prompt
or whatever is very differentthan the data that you have in production.
Right? And like that wholelike test in Prod, it's like it's
very hard to get a good promptworking with like pseudo fake factory
(37:02):
eyes, like local data versuslike the stuff you have in prod.
And when you have supersensitive stuff, it's hard to take
super sensitive stuff fromPROD to local to test on it. And
like some of the things thatwe found is pretty Iinterestingnterestingislike,hey,
pushupourriggingfor,like,agentsandwhatnot,but we're
actually going to control theprompts until we're happy through
feature flags and essentiallyfeed them in.
Yeah, it's just like. It'sjust kind of like the. The idea of
(37:26):
going through your wholedeployment process to change, like,
some English sentences isjust, on the face of it, seems a
little crazy.
Yes.
So, yeah, that's. That'sdefinitely the. The way to. Way to
go is, you know, speed up yourfeedback loop for production by,
you know, using flags or asimilar tool to. To like, manage
(37:47):
your prompts. Because, yeah,like, there's no reason to buil
a new binary and all this justto change some strings. Right?
So I know we're coming up ontime here, and I know you got a busy,
busy new job to get back to,but, like, what are you most excited
about in the space right now?Like, what is some technology you've
(38:08):
seen or something that you'rereally excited to start working with?
Okay, so I've been, I don'tknow, maybe doing more, like, data
stuff lately, and I think thatthe data space is like, I don't know,
very rapidly moving towardslike, like these iceberg tables and
(38:28):
all this, like, a lot. So muchmore. So much more stuff is interoperating
at, like, that kind of cataloglevel. So I'm just excited to see
like dat processing systemsgo from these crazy, incoherent,
complicated things that areimpossible to interop with and have
(38:49):
big egress cost implicationsto, "Oh, maybe we actually can share
big amounts of data betweencompanies in a pretty easy way."
So that's something that I'mreally excited about.
What are these called? Did yousay iceberg?
Yeah, iceberg - Apache Iceberg.
Oh, okay.
Yeah, it's lik a data catalogover, like, kind of like object storage.
(39:10):
Interesting.
But it seems to become sortof. It's becoming more and more like
the lingua franca of datacatalog. So it's exciting to see.
Yeah.
Because that has been such amess. Like, the amount of time I've
spent building products thatjust, like, ferry data from one system
to another is more than I'dlike to consider.
Is this for... is it designedfor essentially moving data between
(39:34):
organizations? Like, is itmore internally... like internal
organizations or externalorganizations? Like, if I'm a big
company like Macy's and I'vegot a bunch of, like, catalog stuff
that I want to run through anAI system. Is it like a technology
for how do I manage giving mydata to another company to build
a model and have governanceover my data? Or is it like more
(39:54):
internal organization organizations?
So kind of both. Like acompany could very reasonably build
kin of like a data lakehousething using Apache Iceberg for just
their internal data. They'llhave data and other systems that
they want to query as well.And today that looks like, you know,
some combination of like useFivetran, build some of your own
(40:17):
ingest stuff and other likeETLs. And it seems like there's a
chance that the industry ismoving to something that is more
like just have your querylayer also talk to this like other
thing that's being exposedthrough Iceberg and a different object
storage system and use thatwhich, you know, makes these things,
(40:38):
these problems much easierbecause you don't have to like slurp
up all the data into your ownobject storage from somewhere else.
That is pretty cool. Is thissomething that you're just tinkering
with or is this like aninitiative at work?
It's something that's likevery closely related to my work.
One of the things, the thingI'm working on right now at work
is like this ETL system and itwould be really nice if in the medium
(41:01):
term the industry moves tosome kind of agreed upon standard
for how we can exchange datathat doesn't involve like ETL from
rest APIs.
Yeah, yeah. Is there a talk inyour future on this or...
Maybe. We'll see. Maybe in thenext year or two. It depends on if
(41:22):
it really becomes more of atrend or not.
Okay, I can wait. Awesome,dude. Well, thanks for coming on
the show. It was great tofinally catch back up with you after
meeting a few months back.Again, sorry about having to reschedule,
but really appreciate the time today.
Yeah, no, it's all good. Yeah,my pleasure. Thanks for having me
on. It's been fun. Yeah.
Where can people find you online?
(41:43):
Shoot. I don't know if I'vereally got a good online. I can talk
about LinkedIn, Mike Zorn. Youcan also send me an email, I'm Mike
at Zorn.Zone. Those areprobably the main two places. I'm
not really big on social mediaright now.
Dude, I envy you. I've peeledso much social media out of my life
(42:04):
and it's like the only socialmedia I do is for the Pod and for
my day job. And I'm so happyto be off, like, personally, but,
like, oh, man, I still log inand I just see, like, the nonsense
and I'm like, "I'm gettingpulled back into this."
Yeah. And it's like, I don'tknow, every now and then I'll go
onto, like, Instagram orsomething. Like, oh, my gosh, it
(42:26):
just like sucks you in. Yeah.It is like, oh, I wanted to do this
one task... like, look up ifthis beer garden has a food truck
today. And then it's like I'vebeen on Instagram for 30 minutes
doing completely unrelated things.
Yep.
Yep. Like, they've. They'vereally optimized it to. To get your
attention.
Yeah. Awesome, man. Well,thanks again for coming on the show.
(42:48):
I really appreciate it. AndI'm going to be up in San Francisco
soon, so I'm going to trackyou down.
All right, sounds good.Looking forward to it.
Awesome. Thanks so much. Havea good one.
Welcome back to the PlatformEngineering Podcast. I'm your host,
Cory O'Daniel, and today Ihave with me Mike Zorn, senior staff
engineer at Rippling. Mike andI met, geez, maybe a month or two
ago at a Honeycomb conference.You were working at LaunchDarkly
at the time. A little bitabout Mike's journey - he started
out in middle school doingTI-83 calculator programming, that's
(00:35):
right?
That's right.
We've got some stuff to talkabout there because I used that calculator
to do a ton of cheating incollege. He's carried the love of
problemproblem-solvingsolvingindifferentrolesin SaaS
companies, from LaunchDarklingto Rippling. Now he's helping teams
build platforms. And so happyto have Mike on the show today. Mike,
thanks for coming on and sorryabout the reschedule.
(00:57):
Oh, no worries. I'm here now,so it works out great.
Awesome. So today I think it'sgoing to be a fun conversation. We're
going to talk about platforms,feature flags, I might try to bug
you a little bit about the AWSoutage recently. But before we hop
into it, I just want to take amoment to say thanks for Kelsey Hightower
(01:17):
hosting the last threeepisodes. Really appreciated that.
Had some really awesomesponsors, SigNoz and Signadot, working
on those episodes. So thanksagain, Kelsey, and apologies to everybody
who had to hear me talk aboutwhat I do during my day job for three
episodes. But it was a funcombo with Kelsey. So, Mike, tell
us a little bit about yourbackground. Like, how did you get
into software? Maybe talkabout these TI-83 shenanigans and
(01:41):
what led you to Rippling?
The first time I wrote acomputer program was definitely on
the TI-83. That's where itstarted. I was initially using...
it was like, "Oh, let me seeif I can make blackjack game."
And then I was able to dothat. That worked great. But then
there was one later, I triedto make like a poker game.
(02:02):
Yeah.
And, you know, I was in middleschool, I didn't know anything about
programming, so I didn't knowthat, like, arrays existed. So I
was like, "Oh, I need to,like, put all the cards in the deck
into variables. But I've onlygot 26 variables from, like, the
letters of the alphabet." AndI was like, "Shoot, I guess you can't
make this program." So thatwas, you know, my first of many programming
(02:26):
failures that I Learned fromlater. I kind of got more serious
into software in college. Istudied math. I was doing, you know,
some computer programs tosolve some math problems and was
like, "Oh, this is like apretty enjoyable thing - to write
(02:46):
a computer program that, like,does this specific task for me."
So when I finished that degreeprogram, I was like, "Oh, you know,
there's actually a lot morejobs in software than there are in
math, so maybe I should trthe software thing out." And it's
been a fun ride for the last,I don't know, like, more than 10
years. I don't really likethinking about how big that number
(03:08):
is now.
It's getting big. It's getting big.
Yeah, I've got a lot of grayhairs now. But in any case, yeah,
I got maybe more into platformtypes stuff as I was working at LaunchDarkly...
(03:30):
o I mean, the job rightbefore LaunchDarkly where I took
ownership of our internalfeature flagging system and then
replace it with LaunchDarkly.And saw how think,moreefficient.a
That's fun. I didn't realizethat you worked on a feature flagging
(03:52):
tool before you went toLaunchDarkly. That's extra fun for
me because I feel like whenyou talk to somebody about the product
that they work on day in, dayout, there's obviously passion for
the idea... but you'veactually experienced both sides of
it. Trying to build onein-house, which I've also experienced
that and was like, "Wow, I'mvery glad things like LaunchDarkly
exist so I don't have to dothat anymore." It is a bit more than
(04:16):
just a side project somebodycould work on at a company, right? There's
a lot to feature flagging,more than just is this thing true
or false, yes?
Yeah, yeah. It starts out asis this thing true or false? But
then you get Customer Successpeople that are like, "Hey, can I
(04:36):
turn this on just for onecustomer? They want to try it out."
And then it's like, "Oh, okay,well, now it's not just true or false.
It's true or false for somepeople, but not for others." And
then you build that for themand then you realize, like, you've
created a nightmare foryourself. Because now you have to
turn things on and off fordifferent customers because you don't
(04:58):
want to give Customer Successpeople - as great as they are - direct
access to one of your proddatabases to make these changes.
And you're like, "Okay, now Ineed like a UI or something to enable
that." And then you're like,"Okay, now I'm personally maintaining
like a minor product." At thatpoint it's like, "Well, maybe I just
(05:19):
buy this thing from somebodyelse." So that was my journey to
that company.
That's an interesting one too,because I feel like, you know, looking
at like the... we have a lotof classes of different tools in
the software engineering spaceand I think obviously, or maybe not
obviously... but the earlieryou adopt some of these tools, like
(05:42):
the better patterns you getin. But there's obviously weight
and time and effort toadopting tools like CI/CD, ephemeral
environments, feature flags.There's so many ways to build and
deliver and deploy goodquality software and I feel liklike,ehonestly,oneofthe
places where you can yield themost out of good quality software
(06:04):
early without a ton offriction is adopting feature flagging.
I feel like I've seen so manyteams that struggle around staging
swim lanes and all sorts ofstuff like that. And there's so much
pain and there's just so mucheffort and delivery of software that
can be resolved by starting tolean into feature flags early. And
(06:25):
I feel like it's one of thosethings you don't see enough teams
lean into until they starthitting that later stage, big teams.
And it's like the stuff youcan get out of day one, just like
MVP testing, being able to getsomething out there and test it just
a little bit in like a realproduction-like environment is something
that you can really yieldbenefits from as an early stage company.
At LaunchDarkly, what was thecustomer base like? Were you getting
(06:47):
a lot more mature customers?Or like, were you seeing earlier
stage companies start torealize like the benefits that they
could get out of featureflagging and start to lean into it?
Is it still something that isnot really realized until companies
are at like much later stages?
So I think it's prettyvariable. So, you know, LaunchDarkly
has a ton of really bigcustomers and working as an engineer
(07:10):
there, those are kind of yourfocus because you bring in the most
money. But there also are aton of, you know, smaller customers.
This company I was at beforeLaunchDarkly wa kind of one of those
customers. I was there fromlike Series A through B. And this
was back when that was not alot of... you know, series A was
(07:30):
like a few million dollars,not like $30 million.
Yeah.
And that company... the CTO,made a pretty good decision early
to, like, have feature flagsbe a core concept. The way we developed
software there was, like, justvery easy because we didn't have
staging environments oranything like this. It was like you
(07:53):
just shipped your new thingbehind a flag. It was in production
within, like, a minute afteryou merged, and then you turn it
on for yourself to test itout. If it didn't work, you'd like,
you know, make some changesand, like, that'd be your feedback.
You'd still have a reallytight feedback loop like that. You
know, that company didn't turnout for reasons that were not really
related to our engineeringefficiency, but, you know, we had
(08:18):
a really efficient engineeringteam when it came to, like, our feedback
loop and how we developedsoftware. I'm a little bit removed
from LaunchDarkly now, so Idon't know exactly what the customer
makeup, the real customermakeup is, versus the one I perceive
from the support requests thatgot escalated.
Yeah, no, that's fine. But Imean, you were seeing, like, earlier
stage companies start to adoptfeature flags though, there. Yeah?
(08:43):
Yeah, yeah, yeah. And I thinkin the early days of LaunchDarkly...
like, yo know, it's a companywhere you put the code, like, really
deep in your own code base, soit's really hard to take out. So
churn has alway been prettylow at LaunchDarkly. There was a
while where the leading sourceof churn were the smal companies
(09:05):
that adopted us that like hadjust ran out of runway and didn't
take off.
Oh, yeah, yeah. That happenswith startups. Bee there, done that.
Yeah, yeah, yeah.
They're hopefully not doingthat right now.
Yeah.
It's funny, I feel like it'sone of those places that is so easy
(09:27):
to get it in. Right? I mean,like, when you have an MVP out, I
think this is one of those,like, technologies also has, like,
staying power. Right. Like,you know, you don't meet a project
manager at Series C who'slike, man, I really wish we didn't
have feature flags. Right. AndI feel like it's one of those things
that's so easy to get inearly, and it just helps you ship
better quality product andalso, like, maintain those cloud
costs. Like you're saying,it's like not having a staging environment.
(09:49):
It's amazing. Right nowthere's a bit more rigor to like
using feature flags, likeresponsibly. I would love to like,
you know, hear some of yourthoughts on that. But like that same
rigor is used to set up andmaintain that staging environment
and that thing costs you moneyall day long. Right. So like being
able to get stuff out in theproduction I think is really, really
valuable. That being said, itis deeper in your code. What are
(10:11):
some of the ways that you'veseen platform teams start to embrace
feature flagging? Maybe whatyou're doing now at rippling and
how do you I guess instillgood practices in a team for where
to use them and how to usethem to cut out a feature and how
to clean them up efficiently?One of the things I've seen is some
(10:32):
teams just tend to use themvery much on the front end and they
say, hey, we built the featureout, you can actually hit it if you
hit our API, but we hide itfrom you using a feature flag. And
I've seen other teams whereit's like they feature flag down
the whole stack. You can't seeit in the UI and you can't access
the endpoint unless you're inthe flag. Right. And it's like kind
of different approaches tolike how you want to scope this stuff.
But like how are you seeingplatform teams use this internally?
(10:56):
Are they using it, are theysurfacing it to their end user customers?
And what are some of thepractices you're seeing that work
well with feature flags?
Yeah, okay, so there's a lot,sorry, those questions.
Okay, I like it.
So I'm probably going to missa couple things and we'll maybe revisit
them, we'll dig in. But Ithink yeah, there's a lot of different
(11:17):
ways to use flags and I thinkyou touched on a few of them and
all of them are prettyreasonable. It just kind of depends
on like what type of productyou have. Like, I think that a really
simple way to use flags is tojust use them in the front end and
like ship APIs that aren'tused unless you are calling them
from the front end. And thatworks great if you know, you don't
(11:39):
have any concerns about likeyour customers spelunking into your
APIs and trying just like todo random curl requests and you have
like a really UI drivenproduct. Like that's a really good
approach. I think that oneplace where the, you know, when you
start thinking about like fullstack flagging and like using flags
(11:59):
on the back end, a place wherethis comes in is a lot of value.
Is like whenever you're doingsome kind of like performance optimization,
you can now be really rigorousabout it because like if you're like
this query is reallyimportant, that the performance is
like uniformly better. Yeah.And I'm like spending a bunch of
time optimizing this querywith a flag. You can like really
(12:23):
do like an actual like a Btest type experiment and see how
the query latency is changedlike with live traffic. And if it
doesn't have the result youexpected, like you can turn it off
really quickly. So those typeof things are I think super powerful
for back end flagging. And ifyou're on a team that is like kind
of, you know, responsible forthe performance of some kind of like
(12:49):
shared database or somethinglike this, it's a really, really
effective tool because you notonly get really kind of like high
fidelity measurement of livetraffic, but you also have the ability
to react super quickly ifsomething isn't as you expected.
You don't have to do aredeploy or anything like this. So
(13:10):
yeah, I think that queryoptimization is like a good, it's
kind of a canonical example ofkind of like a backend flag.
You know what's funny aboutthat? I've used feature flagging
for years. I've never thoughtto use it around like optimizing
a query that you're not. It'sfunny because like my background
is in databases. I've got aton of postgres experience. I've
just never even thought aboutit along like the read level.
(13:34):
Right.
Like, besides like showinglike a new thing but like having
an alternate version of like aSQL query that's like, hey, we're
trying to optimize this, wedon't know how this is going to work
in producing and like beingable to flip between that is actually
pretty cool. And I've gotmultiple things I'm working on right
now in my day job that I'mlike, we're going to push this to
prod and hope it performswell, that I might have to take a
(13:55):
gander. One of the things thatwe've actually seen that works that
we've used is like we've had acouple of times where we've done
a lot of like consolidation asa startup. Like we have a lot of
changes happen to our softwareand we've had a couple of like subsystem
migrations. For example, weoriginally built our registry from
scratch. We're a registry thatstores like, you know, NPM modules,
(14:17):
stuff like that. We built oursfrom scratch. We later rebuilt it
on top of like the OCIstandard. And this is one of those
places where we got to usefeature flags. When we rolled out
our OCI stuff, there was asubset of the system that like, went
to OCI versus our oldregistrya,andndwehad issues with
ita,andndwewere able to justturn that flag off and everybody
just went back to the oldsystem. We were kind of like dual
(14:38):
writing, so we're puttingeverything in both places. But it
was one of those places whereit was like doing a migration. Right?
It's one thing to turn afeature on and off, but like, being
able to like, migrate a systemlive is also a very harrowing thing
to do. And to do that nowadayswithout a feature flag, like, I don't
know how I wouldn't go baldfrom the stress.
(14:58):
Oh, yeah, yeah, yeah, yeah.Even with the flag, you can still
get a little stressed. Severalyears ago, you know, we needed to
kind of like change the entiredata ingest layer of LaunchDarkly.
And you know, we used afeature flag to do the migration.
It was like, there's this sortof daily ritual I had where I was
like, okay, let's expose like5 more percent of traffic into th
(15:20):
new data ingest system thatlike has some scale issues that we're
working through. And it wouldbe like, okay, turn on 5% and then
like, stare at these graphsfor like 30 minutes and see if anything
goes up and to the rightreally quick, like abruptly, and
if so, then like, roll itback. But that was just like a ritual
I had for a while of that was,you know... it was a little stressful,
(15:43):
but it would have been way,way worse if it weren't that. I actually
wrote a talk about that andthen I compared it to like... in
this Bay Area, there's likethe San Francisco-Oakland Bay Bridgean,anddtheyreplaced
one of the spans like, like 10years ago now. And when they did
(16:04):
that, they had to like, theybuilt like a whole second bridge
next to the bridge and thenthey'd close the bridge for a long
weekend and reroute all thelanes between it. And if anything
was wrong, it was just like,"Oh, this crucial artery is just
out forever," or not forever,but for a significant amount of time.
So the amount of testing andstuff they had to do to get the confidence
(16:25):
that that would work, I mean,it's a bridge, so they were going
to do it anyways, I guess. Butit's Just interesting to compare
how things operate in realworld infrastructure and software
infrastructure and how muchmore control you get and how much
faster you're able to move asa result.
And so for platform teams, Ithink this is another place where
(16:48):
like I said earlier, I feellike feature flags are something
that is beneficial to any teamthat has the time to adopt them.
And I feel like there doesn'tget to a point where you're like,
"Man, we should get rid ofthis thing." Thinking of teams as
they kind of mature and scalelike up the DevOps maturity model
and they start moving towardsplatform engineering. I feel like
this is one of those places...like I feel like a platform team
in and of itself can be aforce multiplier, but this is one
(17:10):
of those technologies I feellike can yield a ton of values for
engineering velocity, not justfor getting stuff to prod, but I
mean like (A) surfacingfeature flags as a feature of your
platform, but (B) just usingfeature flags in the platform in
and of itself, whether it'slike releasing new features or pathways
to your engineers or evendoing things like load shedding some
(17:31):
of their services. Like whatare some of the interesting ways
that you've seen platformteams kind of adopt feature flags
in ways that aren't like thetraditional sense of like just hiding
a UI element or enabling afeature in a backend?
Rate limiting and logging arekind of, I think, two places. Logging
is may be a little bit moreuniversal where like it's pretty
(17:56):
often like if you've got asystem with some throughput, like
you're not logging at likeinfo level in your whole application.
And if you've got somecustomer that's doing something weird,
you often need like, "Oh, Ijust need to see those info and debug
logs for this one personbecause they're having a weird experience
(18:18):
and I don't have enoughinformation to figure it out." A
really interesting use offeature flags is to basically change
your log levels for a givenuser kind of in real time. So like,
you know, this person's havingan issue, you're talking to them
on support, you can increaseyour log level for them and then
now you get all the stuff thatyou needed to, to see what the problem
(18:40):
was.
That's pretty cool. We dosomething today, we lean pretty heavy
into OTel and so we actuallyhave an attribute in our OTel that
like what we can essentiallymark that like something has happened
in like the, just the fullOTel process that's gone wrong. And
we actually change our errorlevel if we see that, so we can actually
log some more information. Butit's interesting. There's been times
(19:00):
where we've run into issueswhere it's like, there is no error
- there's no error, but it'slike something weird is still going
on. No exception has happened,but something weird is going on.
And we're like, "Damn, we needsome extra information, but how do
we signal to the system thatdoesn't think anything is going wrong
that we need additionalinformation?" And how we do that
is we redeploy and change ourlog level - which sucks, it's so
(19:21):
primitive. That is tight. Thatis sweet. Okay, sorry, continue,
I'm bookmarking that one.
After working at LaunchDarklyfor so long, I've come to believe
that feature flags are notfeature flags, but they're really
just a tool for changing yourapplication behavior at runtime without
doing a deploy.
Yeah.
And when you think of them inthat general way, there's a lot of
(19:43):
weird stuff you could do. Someof it good, some of it bad. I think
the logging one is pretty justfine. If you have that philosophy
around feature flags, anotherthing that we used at LaunchDarkly
at least was rate limiting. Sosometimes you have a rate limit you
want to apply acrosseverybody, but every now and then
you have some importantcustomer that's like, "Oh, I'll give
(20:05):
you a little bit of moneyextra or something to get a higher
rate limit." And being able tochange that through feature flags
is pretty nice because if youalready are rolling out features
for a customer, you can justkind of use the same way to reconfigure
rate limits or something likethis for them.
Yeah. And for people thathaven't used a more advanced feature
flagging system likeLaunchDarkly, a lot of the feature
(20:27):
flagging systems nowadays,it's not just like a, "Hey, this
email address..." or "Hey, isthis thing true or false?" You can
actually push data into themas well. So you can say like, "Hey,
this is the email address andthey happen to be of this org that
has this SLA." So you canactually make the flags a bit more
dynamic because you canactually push data into them. Sorry,
just a little note there.
Oh, yeah. That's a goodcomment because I've been in the
(20:51):
game too long t remember thatthat isn't like the normal baseline.
I mean, they weren't like thatfive or six years ago... seven or
eight years ago, right? I thin I saw that feature... I can't
remember the first platform Isaw that and I was like, "Oh, this
makes so many things so muchmore easier." Because it's like,
"Oh, I can give you somecontext to make the decision versus,
like, have it being so Boolean."
(21:13):
Yeah.
Ops teams, you're probablyused to doing all the heavy lifting
when it comes toinfrastructure as code wrangling
root modules, CI/CD scriptsand Terraform, just to keep things
moving along. What if yourdevelopers could just diagram what
they want and you still gotall the control and visibility you
need? That'sexactly whatMassdriver does. Ops teams upload
your trusted infrastructure ascode modules to our registry.Your
(21:33):
developers, they don't have totouch Terraform, build root modules,
or even copy a single line ofCI/CD scripts. They just diagram
(21:55):
their cloud infrastructure.Massdriver pulls the modules and
deploys exactly what's ontheir canvas. The result? It'sstill
managed as code, but withcomplete audit trails, rollbacks,
preview environments and costcontrols. You'll see exactly who's
using what, where and whatresources they're producing, all
without the chaos. Stop doingtwice the work. Startmaking Infrastructure
as Code simpler withMassdriver. Learn more at Massdriver.cloud.
So now you're at Rippling andyou're actually working on a platform
(22:18):
team. Are you already startingto use feature flags? Is it something
that was already like apattern there, or is that something
you're introducing in your new role?
Oh, yeah. No, no, no. SoRippling actually has been a LaunchDarkly
customer for a while.
Ooh, spicy.
So, yeah, they've had featureflags adopted for a while there.
Rippling actually, like,has... I'm still kind of undecided
(22:40):
about what I like to thinkabout this pattern but... they have
an interesting way formanaging having too many flags. Because
there's a lot of engineeringteams at Rippling, a lot of people
are working in kind of likethis giant monolith code base, and
there's a ton of flags in thatcode base. That leads to sometimes
(23:03):
unpredictable and surprisingbehavior when there are flags left
kind of in a partially rolledout state and somebody forgets to
fully roll out a feature orsomething like this. So they're pretty
mature in their adoption offlags to the point where they have
so many flags it's causingsome problems.
Oh, wow.
It's like thousands andthousands of flags. I think, on average,
(23:25):
each developer has like fourflags or something like this... and
that's after implementing thisprocess that's like driven it down
pretty significantly. Theprocess is that each team gets a
cap on how many partiallyrolled out flags they can have.
Okay.
So, like, if you have too manyflags that are partially rolled out,
you can't create a new flag todo a new rollout. Which is the way
(23:48):
we've sold it. We've tried tomitigate the issue at Rippling, and
it's been pretty effective. Ifyou look at the charts of, like,
how many partially rolled outflags we have, th cap is being respected.
But yeah, they definitely areusing flags a lot. We're using flags
a lot there.
Oh, man, you got me. We had...geez, this was like two years ago.
(24:13):
We had redone a major, like,replatforming, and we were kind of
doing the thing where... Ithink people use feature flags a
lot for this... where you wantto opt into the new UI, right? And
it's like, "Oh, I don't likeit. I want to go back to the old
one." Well, we hadn't got tothat level of maturity yet where
we're ready for customers topick whether or not they wanted the
(24:34):
UI. And internally, we'reusing it. We're like, "This is great.
This is awesome." And we put afew of, like, our bigger customers
into it, and then we justforgot that there was a feature flag,
like, gating off this wholenew great experience of the old experience.
And we just had a while wherepeople were signing up for the platform
and they were like, "Thisthing's a real piece of shit." And
we're like, "What are youtalking about? We overhauled the
shit out of this thing for,like, months." And we just forgot,
(24:57):
like... all of our payingcustomers were in the flag, everybody
internally was in the flag,but no new customers were in the
flag. I think that is like oneof the two big things that I'm always
watching for with flags is (A)we have 100% flags that are rolled
out that we forgot to cleanup, and they're like, littered across
(25:17):
the code base. And (B), thisexact thing of, like, you have a
flag, you've rolled it out to,like, a high percent people, but
not to everybody, and you'vejust forgotten about it. Right? So
you said you kind of have apattern now where it's like, max
four flags per person. Howhave you seen people solve this in
the past when you're atLaunchDarkly? And how are you thinking
about it now with the teamthat has such a high number of flags
(25:38):
in flight?
Yeah. So I think that, andthis is maybe a place where platform
teams can come in... isintegrating flags with observability
is pretty important. What Imean by that is like, if you have
spans where like flags areevaluated, you probably want to know
what the value of that flagwas. And then you can like create
(26:02):
monitors to let you know,like, are, you know, people of this
segment, you know, seeing theexperience that you expect they are.
And the thing is like, gettingthat type of observability to be
standardized across your stackis something that is really, really
nice. That was something thatwe had set up really well at LaunchDarkly,
(26:23):
where you know, like prettymuch any, any flag being evaluated,
we would record kind of likethe context that we were evaluating
flags with and like the flagvalue served like, and it would just
be like annotated on, on thetrace. If you have an observability
tool that lets you, you know,run queries across traces and spans,
(26:43):
then you can figure out prettyeasily like what fraction of people
are seeing this feature. Andthen if you're rolling out something
new, you can kind of keepyourself honest by looking at such
a dashboard or something whenyou're doing the rollout and realizing
like, "Oh, wait a second,there's like 20% of people that are
still seeing the oldexperience. I probably need to go
(27:04):
finish this."
One thing that I've seenthat's kind of an interesting way
to use feature flags alongwith OTel that we do internally that
we've really grown to like.We're an early stage startup, so
our API is rapidly changing,adding functionality as we're kind
of stabilizing the business.And at the same time we don't want
(27:28):
to release a million versionsof our API. We also don't want to
break stuff for customers. Andso we kind of have this sort of fluid
live API. And so what we do iscurrently we mark essentially like
fields on our API. We useGraphQL, so fields, queries, mutations,
we'll mark them as deprecated,like when we're ready to start deprecating.
And that'll remove it from ourdocs, which is cool. So like nobody's
(27:51):
going to learn about it andstart using it. And then what we
do is we actually put thosedeprecated accesses on our OTel traces
so we can actually see in OTelwe have a dashboard of like who's
still using deprecatedfeatures. So this is obviously only
people that are actively usingit. But like our, our activity is
pretty high so it's likepeople are using it for deploying
their software every day. Soit's like we're not seeing customers
(28:12):
leave for a month and thencome back to use something. It's
like it's integratedconstantly, right? So we can see
people kind of using thesedeprecated fields. We'll email people,
say, hey, we got this betterexperience, like, start using this
or upgrade your CLI, etcetera. But one of the things that
we do like, as a part of ourdeprecation process is use feature
flags. So when we actually getto the point where we're ready to
fully deprecate something, weactually feature flag people out
(28:34):
of it so that if we do findsomebody, for whatever reason was
using a feature that wedeprecated and they're still dependent
on it, we can actually bringthat back because we see there start
to be some essentially SLOthat's being violated. Sorry, SLO.
Gosh, I mix up my SLA's, SLI'sand SLO's, I know that makes me a
(28:55):
bad person, but dyslexia,we'll start to see that there's errors
there and it's around thedeprecated field and we'll actually
bring it back in for thatperson so that we're not breaking
their software. And then we'llbe like, hey, like, you know, we
had a deprecation window forthis. Like, we enabled it back for
you, right? And so it allowsus to kind of like get people off
of that and make sure peoplearen't using something deprecated
(29:16):
and kind of like slowly likewean off the customer base. And then
we'll just have like a verysmall portion of people that are
able to use it. And then we,as we turn that off and like work
through like the deprecation,we're like, ah, we were able to fully
remove it, but like, if weever get to that point where it's
like a week later, it's thisweird feature and somebody's using
it again, it'll fail, but thenlike flip back on. So it's been a
really interesting way tolike, kind of control not just the
(29:38):
introduction of a feature, butlike the retirement of a feature.
Oh, yeah, definitely. And thatwas, yeah, we employed a pretty similar
pattern at LaunchDarkly whenit was like, whenever we were making
like an API change that wouldhave to be breaking in some way.
You know, you can use flagsfor this and then identify customers
(29:59):
where like, you think theremight be an issue and kind of turn
it off for everyone that isn'tin that group and then work with
the people in that group asit's practical to deprecate it further
for them and kind of winnow itdown until you can actually turn
off, delete the code. Yeah.
So at LaunchDarkly, so yousaid earlier you had a feature flag
on that ingest system. DoesLaunchDarkly dog food its own live
(30:23):
system? Is that how thatworks? Or do you have a separate
LaunchDarkly for LaunchDarkly?
We basically have a separate.Or LaunchDarkly basically has a separate
LaunchDarkly for Launchdarkly.And the reason for that is like,
it's like an exact copy of thecode and everything. It's just like
a second environment thatLaunchDarkly deploys to after the
(30:44):
normal production environment.
Okay.
Because, you know, if there islike, you know, you turn on a feature
flag that brings downLaunchDarkly's UI or something and
you can't turn it off, that'spretty bad. That's why there's the
second one. Yeah, that's cool.
(31:05):
I love seeing teams dog foodtheir own things. But there is certain
classes of software where itgets like. It's one thing to be like,
hey, we're an accountingcompany, we use our own software
to account. But it's like,it's the thing that manages releasing
it. It always gets tricky. Sothis is a full second clone sitting
there that you all areactually using internally. You got
to use the full suite offunctionality then, which is pretty
(31:27):
rad.
Well, it was rad and not radbecause I was able to use all the
functionality, but we also hadto keep healthy. A whole second instance
of LaunchDarkly. Yeah, a wholenother set of computers are gonna
have problems. So as being anon call person periodically, that
would be sometimes annoying.But it was, yeah, it was definitely
(31:50):
nice to have every... al the functionality.
That's not a trivial system tosupport either. Like LaunchDarkly
is like you're in the hot pathof a lot of very, very big companies. So
even a feature flagging systemoperating a feature flagging system
is still, I'd assume, handlinga ton of traffic.
Yeah, it's always like, kindof amazing to just like when I was
(32:11):
working there, see the numbersof like how many flags are being
evaluated and how many likeconcurrent clients are connected
to the service. But then itwould also be like, "Okay, you're
on call for this." And youknow, the impact of a LaunchDarkly
outage is pretty, prettyhigh. So it was always like, you
know, on call, I took itpretty seriously.
(32:32):
Yeah. So how are you seeingteams so like, you know, I know.
I think a lot of ops andplatform teams have different takes
on the extent in which AIshould be involved in our work. But
like, how, how are you seeingpeople start to use AI in and around
feature flags? And like, whatis, I guess the like, state of the
(32:54):
art, as it were? If ther canbe a state of the art, I feel like
the state of the art ischanging so quick.
But yeah, so I think there'slike a couple of different directions
here. Like, one is like, howare teams that are building kind
of like AI products usingflags? I think that there's some
variance there. And thenthere's also the question of like,
(33:16):
you know, teams are justbuilding normal software. How are
they using flags like asthey're using, like having AI help
them write code?
Yeah.
You know, for the, the secondone's maybe more generally applicable.
So we start there. But apattern that I have seen is where
people will just kind of likeput a flag around, you know, the
code you're writing with AIand you know, just develop the feature
kind of like, like normal, butthen you, you know, you have, you
(33:39):
know, you have that, that flagaround it. So if you. Something doesn't
work out, you're all good. Andsimilarly in that additional level
of control lets you kind ofdelegate more to an AI coding agent
or something like this to havethem produce a feature, like the
agent produce a feature andput it wrapped in a flag is definitely
(34:00):
something I've seen. And thenon the other side of people that
are building AI products,like, I think observability for AI
products has kind of been aninteresting place where it's hard
to always know, like there'snot like a, an error that's going
to be logged if you get like anot great response from your like
(34:23):
LLM call. So there's, youknow, a lot of products that are
made to help you figure outlike based, like offline, like, you
know, you had all theseevaluations and then there's a lot
of products that help youfigure out were these good or bad.
And one tool that it seemslike more mature teams in that space
use is experimentation to, asthey make a change to how they're
(34:46):
calling the model or whichmodel they're using, they'll use
just normal experimentation,A/B testin as kind of like the final
step of a release becausethat's kind of like the best signal
you'll get. Is did this changehave the effect on user behavior
(35:07):
that I anticipated? Andexperimentation is a really good
tool for that. So those are acouple of ways that people use it.
It's interesting too becausewhen the new models come out and
it's like everybody kind ofrecoils because it's like got so
much dumber, lost all mywhatever. But yeah, I feel like when
you've spent so much effortand I feel like it is a lot of effort
(35:29):
to get a good prompt, it'slike everybody's like, oh, why don't
you just AI it? It's like,well the prompt is so hard. Yeah,
but like once you, once you'vegot a good prompt and like a good
set of agents like configuredand like working together and then
a model changes like oof,that's disruptive. Right. And it's
hard. Kind of going back towhat I was saying earlier, like how
we use it, like migratesubsystems. I think that's a really
(35:50):
great place to be able to swapout like subsets of users. Say like,
hey, what is, what does itlook like for us with these prompts
and agents on the new model?Right? Just being able to see like
how much it has changed andlike slowly rolling your customers
onto that new model before youjust kind of like whole shebang,
like hey, let me just changethe model name in this config and
deploy to production and seehow different this is for everybody.
(36:13):
Right? Like we do have a bigbag of nondeterminon-determinismnismandthisisoneofthoseone
placeswhereit's likeyoucanexhibitsomecontrolover it. I think
that in the age of AI, thesefeature flags become even more important.
Another place I've seen itused a little bit is internally is
around the actual prompts,controlling the prompts themselves
(36:36):
through the feature flags. Andsomebody might be saying why. It's
like it's very easy to getyour rigging into prod, but this
one of those things that alsokind of sucks a bit. It's like your
dat that you have locally,like while you're testing a prompt
or whatever is very differentthan the data that you have in production.
Right? And like that wholelike test in Prod, it's like it's
very hard to get a good promptworking with like pseudo fake factory
(37:02):
eyes, like local data versuslike the stuff you have in prod.
And when you have supersensitive stuff, it's hard to take
super sensitive stuff fromPROD to local to test on it. And
like some of the things thatwe found is pretty Iinterestingnterestingislike,hey,
pushupourriggingfor,like,agentsandwhatnot,but we're
actually going to control theprompts until we're happy through
feature flags and essentiallyfeed them in.
Yeah, it's just like. It'sjust kind of like the. The idea of
(37:26):
going through your wholedeployment process to change, like,
some English sentences isjust, on the face of it, seems a
little crazy.
Yes.
So, yeah, that's. That'sdefinitely the. The way to. Way to
go is, you know, speed up yourfeedback loop for production by,
you know, using flags or asimilar tool to. To like, manage
(37:47):
your prompts. Because, yeah,like, there's no reason to buil
a new binary and all this justto change some strings. Right?
So I know we're coming up ontime here, and I know you got a busy,
busy new job to get back to,but, like, what are you most excited
about in the space right now?Like, what is some technology you've
(38:08):
seen or something that you'rereally excited to start working with?
Okay, so I've been, I don'tknow, maybe doing more, like, data
stuff lately, and I think thatthe data space is like, I don't know,
very rapidly moving towardslike, like these iceberg tables and
(38:28):
all this, like, a lot. So muchmore. So much more stuff is interoperating
at, like, that kind of cataloglevel. So I'm just excited to see
like dat processing systemsgo from these crazy, incoherent,
complicated things that areimpossible to interop with and have
(38:49):
big egress cost implicationsto, "Oh, maybe we actually can share
big amounts of data betweencompanies in a pretty easy way."
So that's something that I'mreally excited about.
What are these called? Did yousay iceberg?
Yeah, iceberg - Apache Iceberg.
Oh, okay.
Yeah, it's lik a data catalogover, like, kind of like object storage.
(39:10):
Interesting.
But it seems to become sortof. It's becoming more and more like
the lingua franca of datacatalog. So it's exciting to see.
Yeah.
Because that has been such amess. Like, the amount of time I've
spent building products thatjust, like, ferry data from one system
to another is more than I'dlike to consider.
Is this for... is it designedfor essentially moving data between
(39:34):
organizations? Like, is itmore internally... like internal
organizations or externalorganizations? Like, if I'm a big
company like Macy's and I'vegot a bunch of, like, catalog stuff
that I want to run through anAI system. Is it like a technology
for how do I manage giving mydata to another company to build
a model and have governanceover my data? Or is it like more
(39:54):
internal organization organizations?
So kind of both. Like acompany could very reasonably build
kin of like a data lakehousething using Apache Iceberg for just
their internal data. They'llhave data and other systems that
they want to query as well.And today that looks like, you know,
some combination of like useFivetran, build some of your own
(40:17):
ingest stuff and other likeETLs. And it seems like there's a
chance that the industry ismoving to something that is more
like just have your querylayer also talk to this like other
thing that's being exposedthrough Iceberg and a different object
storage system and use thatwhich, you know, makes these things,
(40:38):
these problems much easierbecause you don't have to like slurp
up all the data into your ownobject storage from somewhere else.
That is pretty cool. Is thissomething that you're just tinkering
with or is this like aninitiative at work?
It's something that's likevery closely related to my work.
One of the things, the thingI'm working on right now at work
is like this ETL system and itwould be really nice if in the medium
(41:01):
term the industry moves tosome kind of agreed upon standard
for how we can exchange datathat doesn't involve like ETL from
rest APIs.
Yeah, yeah. Is there a talk inyour future on this or...
Maybe. We'll see. Maybe in thenext year or two. It depends on if
(41:22):
it really becomes more of atrend or not.
Okay, I can wait. Awesome,dude. Well, thanks for coming on
the show. It was great tofinally catch back up with you after
meeting a few months back.Again, sorry about having to reschedule,
but really appreciate the time today.
Yeah, no, it's all good. Yeah,my pleasure. Thanks for having me
on. It's been fun. Yeah.
Where can people find you online?
(41:43):
Shoot. I don't know if I'vereally got a good online. I can talk
about LinkedIn, Mike Zorn. Youcan also send me an email, I'm Mike
at Zorn.Zone. Those areprobably the main two places. I'm
not really big on social mediaright now.
Dude, I envy you. I've peeledso much social media out of my life
(42:04):
and it's like the only socialmedia I do is for the Pod and for
my day job. And I'm so happyto be off, like, personally, but,
like, oh, man, I still log inand I just see, like, the nonsense
and I'm like, "I'm gettingpulled back into this."
Yeah. And it's like, I don'tknow, every now and then I'll go
onto, like, Instagram orsomething. Like, oh, my gosh, it
(42:26):
just like sucks you in. Yeah.It is like, oh, I wanted to do this
one task... like, look up ifthis beer garden has a food truck
today. And then it's like I'vebeen on Instagram for 30 minutes
doing completely unrelated things.
Yep.
Yep. Like, they've. They'vereally optimized it to. To get your
attention.
Yeah. Awesome, man. Well,thanks again for coming on the show.
(42:48):
I really appreciate it. AndI'm going to be up in San Francisco
soon, so I'm going to trackyou down.
All right, sounds good.Looking forward to it.
Awesome. Thanks so much. Havea good one.
Popular Podcasts
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
Are You A Charlotte?
In 1997, actress Kristin Davis’ life was forever changed when she took on the role of Charlotte York in Sex and the City. As we watched Carrie, Samantha, Miranda and Charlotte navigate relationships in NYC, the show helped push once unacceptable conversation topics out of the shadows and altered the narrative around women and sex. We all saw ourselves in them as they searched for fulfillment in life, sex and friendships. Now, Kristin Davis wants to connect with you, the fans, and share untold stories and all the behind the scenes. Together, with Kristin and special guests, what will begin with Sex and the City will evolve into talks about themes that are still so relevant today. "Are you a Charlotte?" is much more than just rewatching this beloved show, it brings the past and the present together as we talk with heart, humor and of course some optimism.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.