June 10, 2026 • 50 mins
When code gets cheaper to produce, feedback becomes the limiting factor - CI, reviews, and the handoffs between tools can quietly slow everything down.
Rob Zuber breaks down what platform engineers are seeing as teams adopt AI-assisted development: more branch builds, new failure modes, and growing pressure to shorten the loop between “change made” and “change validated.” He focuses on how CI can evolve from a human-first dashboard into a system that agents can interact with directly through APIs, CLIs, and MCP-style interfaces - so fixes can happen faster and with less waiting on manual triage.
Along the way, Rob and Cory dig into practical questions engineering leaders are wrestling with: how PR review becomes the next major bottleneck, what “agent experience” means in a delivery pipeline, why speed isn’t only about faster compute (it’s also about doing less unnecessary work), and how teams can share learnings so “agentic velocity” doesn’t only benefit a few power users.
If you’re building or running the systems that ship software, this is a clear look at where CI fits in an AI-accelerated workflow, and what needs to change to keep delivery safe, fast, and sustainable.
Guest: Rob Zuber, Chief Technology Officer at CircleCI
Rob Zuber is a 20-year veteran of software startups, a four-time founder, and three-time CTO. Since joining CircleCI, Rob has seen the company through its Series F funding and delivered on product innovation at scale while leading a team of 300+ engineers who are distributed around the globe.
Links to interesting things from this episode:
- “The Confident Commit” podcast
- Wardley Mapping
- “How one programmer broke the internet by deleting a tiny piece of code.”
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:04):
Welcome back to the PlatformEngineering podcast. I'm your host,
Cory O'Daniel. Today on theshow, I'm joined by Rob Zuber, CTO
at CircleCI. Rob's been in thesoftware space for three decades,
been a CTO three times over,and you've been at CircleCI for the
past eleven years?
Yep.
Welcome to the show. CI, thatis... that's where all of our compute
(00:25):
used to go.
Yeah. Yeah. Well, first ofall, thanks for having me. Excited
to be here. Yeah, it used tobe, and it's kind of... it's back
in a new and different way.You know, I think people are trying
to go as fast as they can. Itturns out when you go as fast as
you can, you actually want toknow if your stuff's any good. So
we're reimagining what it isthat we do, but we're right in the
(00:46):
thick of it now, for sure.
Yeah. I feel like there's alot of discussion... I mean, there's
a ton of discussion onlineabout AI in general, but, like, I
feel like a lot of what I seeon LinkedIn and Reddit is people
talking about, like, how dothe fundamental things that we've
kind of built on for the pasttwenty, thirty years, like change
in this new world? And I feellike CI is going to be at the heart
(01:08):
of a lot of it. So, superexcited to talk to you today and
see what you all areexperiencing and what you're seeing
on the other side of the world.
Yeah.
From us, who's just kind ofpushing Docker images up. But yeah,
so, I mean, like, you know,software is changing a fair amount.
We're getting a lot... a fair bit.
Yeah, just a bit.
Open a couple more PRs. Whathas... I guess internally at CircleCI,
(01:32):
what is the load like? I knowGitHub's talking about how much more
changes and pull requests andPRs they're seeing. Are you all seeing
the same type of just sheervelocity change from the amount of
code changes getting pushed through?
Yeah, absolutely. So we'reseeing significant increase in volume...
handling it well from areliability perspective. And then
(01:55):
what we are seeing a mixedchange, if that makes sense. Like,
you know, we think aboutbranch builds where people are checking
to see if the thing they'recurrently working on is working as
expected, and then mainline,trunk, whatever you want to call
it, meaning they've actuallymerged and they're trying to get
something out of production.And we've seen shifts in the mix,
like kind of more branchbuilds. I mean, of course, it's distributed
(02:18):
across customers differently.But in a lot of cases, more branch
builds, meaning there'sactivity, but maybe not as much of
an increase in main builds,meaning people are seeing, you know,
lower success rate. Right. Sothe work of their agent is maybe
okay, but they're using CI andthat branch build as a means t check
(02:38):
that it's good and they'refinding out that it's not. And so
we're investing a huge amountof energy right now in trying to
shorten that feedback cycle,give people things that they can
connect directly to theiragents so that they can get a shorter
cycle basically, and know thattheir work is good and basically
not just good or bad, but givethe feedback directly to the agent
(02:59):
to say, "Actually change this,change this, change this before we
even sort of go through theprocess or the ceremony of sort of
PR and beyond." And so, yeah,absolutely, we're seeing that volume
and what we're seeing as aresult is folks trying to find ways
to tune the whole process sothat it all doesn't just back up
at kind of like PR and then release.
(03:22):
Yeah, I feel like it's a spotof the stack where it's at least,
I feel like twofold going tobe one of the places that slows us
down. A) just the build times,right? CI takes time to run. We're
spending time pushing code upand now we're pushing way more PRs
(03:42):
up than we used to. I mean, Ithink last week I shipped like seventy
PRs or something like that tothe terror of my team, who is on
the other side.The other thingthat's going to slow us down a bit,
I feel like in this adoptionjourney, is the review. The humans
in the loop looking ateverything after it's been pushed
up. But it does seem like oneof those places where it's going
(04:06):
to be prettycriticalasmorepeoplestarttoadopttheseworkflows,thatCIworkswell,fastand itkindofkeeps
youintheloopwith what'sbeingchangedatanextremelyhighvelocity.Whichwehaven'tseenbefore.Whatischanging?Imean,besidesjust What
(04:29):
is changing? I mean, besidesjust the sheer volume. But what are,
I differentlyabout and whatare engineers requestin ayearortwoago?
MediumI think so. So there's awhole category around speed, which
we'll talk about. I'll getinto in a second. And then there's
(04:50):
what I would call agentexperience, which is like I'm actually
operating with the other toolsin my tool chain by letting the agent
do that, right? So theinterfaces, like agents, not going
to a web UI to figure out whatwent wrong. So we're exposing a lot
more direct capabilitiesthrough tools that we could, you
know, we would also use ashumans like CLIS and APIs, but also
(05:12):
MCP server, like those sortsof interfaces that allow the agent
to kind of run its own loopand say, "Okay, I pushed a thing,
I'm watching CI, CI failed,I've got the logs, I know what happened.
I'm pushing again." Likebasically allowing people to get
further away from thatprocess. I mean, we'll come back
to your whole thing abouthumans and where they get involved.
(05:33):
But then on the speed front,there's like absolutely raw compute...
you know, we spent a lot oftime over the years optimizing parallelism
and getting sort of wall clocktime down, not just compute time...
but then there are also some,you know, some shortcuts, right?
Like you could think aboutdoing the same thing as fast as possible,
(05:56):
right? How do I take all thesesteps and get really fast, compute,
optimize, caching, manageparallelism, like all the things
that we think about. But thenyou could think about doing less
work, right? So when we talkabout giving feedback directly to
the agent, first we isolatethe parts that are going to be really
valuable to the agent in thatstep. Right. Certainly these days
(06:16):
when I push something and itfails linting in CI, I'm like, "How
did this happen? How did I getthis far down the process? Oh, now
I gotta like, you know, forceor like, you know, amend my commit,
force push, all these kinds ofthings." It's just like a lot of
overhead that is... yes, itlike the individual step might take
time, but it's more that likeit comes back to me as a human, maybe
(06:38):
I went to get coffee, now Icome back and like, you know, a whole
bunch of time has passed. Soreally pushing all these things down
to the agent. Again, isolatingthe things that really matter, doing
those super, super quick,right? With kind of live standby
compute that, you know, we canrun that check in thirty seconds,
not your ten minute sort ofbuild time. And then things like
(06:58):
isolating specific tests thatwe know that are the ones that will
be impacted by a change andrunning those, maybe prioritizing
them, running them first, onlyrunning those on a branch. Like things
that are not just what isfaster the same... I guess the same
thing, like at a faster linearspeed... but how can we make nonlinear
improvements? Because people,everything else is getting sped up
(07:21):
before that in a way thatmakes... you know, a ten minute build
time used to feel great for alot of people. And now people are
like, "What am I supposed todo for ten minutes?" Right? I mean,
they spin up another agent,another agent, another agent. Like
whether they multi, you know,they multitask and get lost. And
then on the human side... andthis is like we're kind of... we
have touch points all throughthe process, Can we give you a enough
(07:44):
validation and enoughconfidence in the change to reduce
the human involvement? Becauseit is like... you know, I was joking
actually just earlier todaywith some folks on my team that what
I really want just to start islike estimated reading time. You
know, like I look at a pile ofPRs and I'm like, "What can I carve
(08:06):
out time for?" You know, likeon medium or something, it's like
this one's sixty seconds andthis one might take you twenty minutes.
Like just that would be great.But then, you know what we're layering
on top of that is this changeis a critical part of the system.
This change was made bysomeone who maybe doesn't know this
part of the system very well.This might take a little bit more.
This feels like something.Honestly, you could auto merge this,
(08:28):
right? I think we're going tomove away from kind of this big grandiose
ceremony around the PR andinto systems that are designed to
move at the speed. The PR hasalways kind of been like a laborious
gate. Talk to anyone aboutdeveloper efficiency and you end
up having a conversation aboutPR review time, lag, and then everyone
is this heavy exhale, like"ugh." Right. So it doesn't feel
(08:50):
like it's the thing that waseven tuned to be fast when we were
just typing code. And now, youknow, you're trying to go ten, a
hundred times faster.Something has to be different about
that. And so we're investingagain in, within that automation,
how can we give you thestrongest signal to either shorten
or even like reduce the numberof checks tha you have to make?
(09:12):
Yeah, I think that's actuallypretty interesting. There's some
couple of things you saidthere like just the intelligence
and the command or pullrequest itself. I think that with
the number of PRs I stillreview a day, I do think that there
would be a significant lifeimprovement just to seeing like this
one's going to take a minuteto review. It's like, "Okay, let
(09:34):
me get Joe happy and approvethese quick ones for him really quick."
Versus that one's going totake 25 minutes... because sometimes
you'll see a bunch of filechanges, but the file changes do
not correlate to the amount oftime it takes to process and understand
a pull request. That wasinteresting. The idea of being able
to, and I know you can do thistoday with labels and whatnot, but
(09:57):
again, it's one of thoselaborious tasks to do as a bag of
meat. But to be able to tagparts of your system or even repos
and say that this is acritical versus a non critical component
and treating your reviews andPRs differently, that is something
that's hard to do for manyorgs today. While it would be very
(10:19):
beneficial even without AI, Icould see that that would be... like
to be able to look at a biggersystem and be like, "I don't have
to worry about this as muchbecause it's not hitting a core part
of the stack." And like beingable to identify that, that would
be pretty awesome.
Yeah. As a platform that sitsin the middle of software delivery,
(10:41):
we have a pretty goodhistorical understanding of what
changes in your system lead todifficult outcomes. Let's put it
that way. Right. Like we knowwhen things break downstream and
so have a sense that like, oh,this is a place that you should probably
feel pretty good about makinga change. This is a place I would
put a little bit of extraeffort into. Right. There's known
(11:01):
side effects. Historicallythis has led to, you know, those
like tightly coupled placeswhere everybody changes one thing
and forgets another and thenwe're like, "Oh yeah, that incident
happened here, et cetera, etcetera." So like, I think there's...
I guess intelligence begetsintelligence in a way. Like we've
applied intelligence to a partof the process and now we have a
lot of opportunity to apply itelsewhere, which is really the only
(11:22):
way we're going to be able tospeed. Now I'm saying let's call
it artificial intelligence.Before we like we've applied humans
to the process before, butagain it's sort of this like intuition.
I find this really interestingin general, right. Like we have humans
who have a very unevenlydistributed mental model of our system.
(11:44):
Whether it's who in theorganization is really trustworthy
and who we should double checktheir work or is just growing or
whatever and where in thesystem things are a little sketch.
And I don't feel supercomfortable that the test coverage
here, even though the numberis high, really truly gives me confidence
that I can Put this thing out,this thing's, you know, super robust.
(12:06):
And it's like some people knowit, some people don't, some people,
whatever. And I think whatwe're really seeing now is the ability
to extract that and have itapplied more universally, right?
Like as people build outskills and context and whatever with
their organization to be ableto say, cool, this change. Like regardless
(12:30):
of who created it, this andwho's looking at it, this system
has all of that knowledge orcan gain all of that knowledge and
retain it in a useful way ifwe manage it effectively. I think
it was like the late nineties,we spent all this time talking about
expert systems, which waslike, you remember that? And now
(12:51):
we finally built them and allthey are is a pile of markdown and
a repo. But it's like we tookeveryone's knowledge and we put it
somewhere we actually are ableto apply it. Like that was a fool's
errand in the late nineties,right? And now it's just like, it's
just kind of happening becausepeople are type writing stuff down
or they work on something andthey're like, "Hey, Claude (or whoever/whatever
tool), can you write this downfor me and make a record of it?"
And as that stuff gets morecentralized, right. We're able to
(13:15):
extract and consolidate theselittle pockets of, of understanding
that we're just, you know, ifthis person happens to be the person
that reads your pr, it'll gookay. Which is not great. Organizational
design was never designed.It's just what happened.
Yeah, the code owner, it'sjust like, yeah, Cory owns it. If
he. It's going to be fine.It's like, yeah, maybe, maybe not.
(13:39):
Yeah, no, that is. I feel likethere's going to be a lot of really
interesting changes to the waythat we write and deliver software.
I mean, I know even in my ownpractice, like, I'm a very TDD oriented
engineer and I'm a person whoopened to PR very early. I love seeing
a failing build. I love seeingthem go green. I just open and let
(14:01):
him rip. But I've done a lotof engineering myself just around
my workflow that I've had torethink my own development workflow
itself as I started to adoptAI tools. So a thing that I used
to do a lot was I don't lovethe idea of a linter failing my build
on a CI run. I don't mindseeing a test fail that I'm working
(14:24):
on. Great. That's what'sexpected. I want my teammates to
see that test failing, so theycan see where I'm at in the process.
But I've always leaned intothe pre commits and whatnot. It's
like, hey, get most of thisstuff. Just make sure it's good before
you even waste anybody's time.And now I'm just like, there's way
too many agents running forthem all to be running all my pre
commit tooling locally. Thatis CI's problem now, right? And it's
(14:48):
just like. So it's likesomething that I had spent a lot
of time like, ah, trying to belike a good steward of CI. I'm just
like, commodity compute, letthat all run over there. Like, I
don't have time for this thingto run locally anymore. But it is
funny because it's like that,that ten minutes that you're waiting,
then you're like, oh, I guessI start up another agent. Like, that's
exactly what happened to me. Iwas like, when I first started, I
(15:08):
was like, I'm only going to doone at a time so I don't drive myself
insane with a thousand PRs.And then the first time you're staring,
you're like, shit, I don'thave anything to do for ten minutes.
You're like, I could writethat spec in ten minutes.
Yeah.
And then all of a sudden yougot two. And then all of a sudden
you got, I think eight rightnow running in the background. So
as far as like your own team'sconcerned, like, what are some of
(15:31):
the things that, you know,you've got a, you got a big team,
you got a few hundredengineers right around.
About a hundred, yeah.
A hundred. So like, what areyou seeing not in the CI process
itself falling apart, butlike, like, what are you seeing,
like, starting to break justfor, like, larger engineering teams
that are starting to adoptthese tools.
There's a lot of things tothink about there. And certainly
(15:52):
I'm working also with a veryspecific team as well as the broader
organization. And in thatspecific team, one of our goals or
mandates was like, let's runas fast as we can and see what breaks.
And yeah, it took an hourbefore we were like, well, this is
never going to work. And itwas PR reviews, like two of us sitting
across the table from eachother hammering out PRs. And we're
(16:13):
like, okay, when are we goingto stop and review each other's PRs?
And then I looked at them andI was like, how am I going to get
through this? We're going tobe here for the rest of the week.
Right. And so then we startedbuilding tooling to review PRs, right?
Like how do we. And weactually built something, I mean,
speaking of the people youtrust, we built a thing that extracts
kind of the most common andmost trusted reviewers from the history
(16:34):
of a repo, turns that intocontext and applies it to the next
round of reviews. So it says,okay, well we know someone's going
to comment on this and on thisand on this and like how many senior
or staff or you know, whateverthe top level of your org is, engineers,
open up a PR and go,seriously, again with this mistake.
Like, we've made this mistakea hundred times. We've talked about
(16:56):
it. Like speaking of thelinter, right? And why would I put
this in front of someone elseand use their time. We know what
all the first round of reviewcomments are going to be, so let's
just have them happen locally,right? Like basically we, within
our agents, right, we haveskills that are like execute this
type of review, you know,before it ever gets committed and
pushed. So that by the timewe're asking a human to look at it,
(17:20):
I mean, honestly, they're alsopulling some little local CLI and
running some review over it,saying pull out the things that I
care about kind of thing. Solike trying to optimize a lot of
that stuff. So review isprobably not surprising, but it's
pretty easy to pull some stufftogether that will, that will optimize
that a lot. And then, youknow, I would say a hot topic in
every circle that I travel inat the moment, engineering leaders,
(17:41):
and that circleci is nodifferent is token consumption. And
what's interesting there isefficiency, right? Like the way that
some people use LLMs versusthe way that other people use LLMs,
just very, very different. Andas a result, right? Like there's
a lot of discussion ofleaderboards and token maxing and
(18:03):
all this kind of stuff. Like Idon't really care about that. I would
prefer if people aren't justtrying to burn tokens to make it
look like they're doing work.But for folks that are using them
in good faith, like we seereally big differences in sort of
outcomes relative to kind ofinputs and outputs, right? And so
trying to help peopleunderstand how can I use this effectively,
(18:26):
where was all this energygetting used, right? And then can
we tool around that? Can webuild little CLIs or skills or whatever's,
you know, whatever's the rightthing to get you to an outcome faster?
Because yes, that's like tokencost is real, but also just as A
human, you're trying to getsomething done and someone else has
figured out how to get the LMto get that thing done much faster
than you seem to be able toget it done. And that's, that's no
(18:48):
shade on people. Like, we'reall learning this toolkit together.
So how do we bring thosethings together? How do we create
sort of shared repos of skillsand stuff like that internally to
again give people the boost ofother people's learning? Like, I
think that's. If I were tosummarize all that. Like it started
with token costs, but reallyit's fractured learning, right? Because
there's no. I can't just, youknow, go on to Amazon and buy the
(19:11):
book on how to be awesome atusing whatever today's coding agent
is, because by tomorrow it'sgoing to be a different one or like
a new version and optimize fordifferent things. New model versions
drop. And like your promptsthat used to be great are now a little,
you know, not working verywell. So there's, there's all kinds
of those different pocketsagain of like fractured learning.
Ironically, I was talkingearlier about how we're bringing
(19:31):
all our knowledge together andso this is a new area that's just
like we're learning soquickly. So that had. Bringing everyone
along is hard, right? Like ineven. Even I'll say just 100 people,
right? There's a decent sizedengineering org, but I know people
in engineering orgs with35,000 engineers. I can't even imagine
what that's like, let alonehow you try to educate them all on
(19:53):
how to be really effectivewith this kind of tooling. So I don't
like we have the standardproblems, right? Cycle times of reviews,
you know, we optimize a lot ofit. But it's really, how do we do
that then consistently acrossthe org. Not that everything is like
cookie cutter stamped it, butjust without leaving people behind
who are sort of likestruggling or.
(20:14):
Yeah, yeah, no, I feel likeeven at 100 engineers, that's gotta
be tough. I don't know how.I'll be very curious how these larger,
larger, larger organizationshandle it. But like we're fairly
small team and you know, as westarted leaning into it, like it
was just, it just felt likeall of a sudden there was forty employees
one day. Like it was just likethe amount of PRs. It was like, oh,
(20:37):
two people on the team havefigured it out. And then it was like,
we got to get. Now the rest ofthe team has to figure it out. Or
like, the value extraction'soff, right? Like, if two people,
you know what I'm saying, arelike, are outputting 40x, that sucks
for everybody else that has toreview that code. Right now, their
job is reviewing code, notwriting code, which, that's the worst
place to be. Like, I haven'tfigured out how to use this tool
(20:59):
yet, and my reward isreviewing everybody else's output
because I haven't caught up.
Ops teams, you're probablyused to doing all the heavy lifting
when it comes toinfrastructure as code wrangling
root modules, CI/CD scriptsand Terraform, just to keep things
moving along. What if yourdevelopers could just diagram what
they want and you still gotall the control and visibility you
need? That's exactly whatMassdriver does. Ops teams upload
your trusted infrastructure ascode modules to our registry.Your
(21:22):
developers, they don't have totouch Terraform, build root modules,
or even copy a single line ofCI/CD scripts. They just diagram
their cloud infrastructure.Massdriver pulls the modules and
(21:47):
deploys exactly what's ontheir canvas. The result? It's still
managed as code, but withcomplete audit trails, rollbacks,
preview environments and costcontrols. You'll see exactly who's
using what, where and whatresources they're producing, all
without the chaos. Stop doingtwice the work. Start making Infrastructure
as Code simpler withMassdriver. Learn more at Massdriver.cloud.
I was talking to my co founderabout this. It's like, he's like,
(22:08):
it. It just feel like beforehe'd leaned into it, before he kind
of like, got the hang of it.He's like, it just. He's like. It
feels like I, you know, don'treally like, what value am I adding
to the team? Like, like, youguys move so fast without me, right?
And then it's just like. Thenyou see him, like, start to catch
up, and then all of a suddenit's just the volume of code is there,
right? And it's just likethey're all kind of working on the
(22:28):
guardrails and catching stuffand making sure that good code quality
is going in. And it's likeonce you get to that part and you
start to feel it humming, itdoes feel really good. But then it's
that next step of when webring on our next teammate. What
does interviewing look like,right? What does bringing them into
the fold look like? Where allof a sudden it's like, you know,
(22:49):
you think about five yearsago, six years ago, I feel like our
number one goal in, like,bringing on a new hire was like,
can we get them to putsomething into production on day
one? Like, could you imaginesitting down at a job and you're
like, oh, let's go take a lookat the ol at how many get up PRs
there are. And it's like, oh,there's six thousand that were open
today. Like the, just thesheer intimidation of joining a team
(23:11):
that's kind of figuring itout. Could be, yeah.
It feels like in a way it's aninteresting description. Like you're
trying, you know, you'retrying to jump onto a moving freight
train kind of thing. But youcould also, like, your speed is so
enhanced, right? Like myability to drop into a code base
I've never looked at and belike, how does this work? Tell me
what the key pieces are. Howwould I implement this thing? And
just, you know, grepping,grepping, grepping, whatever. Like
(23:34):
what I used to do to figureout a code base. Oh, let me pop open
this file, look up this file.Oh, maybe here's a. Here, this looks
interesting. I wonder if I canfind a reference to this function
somewhere. Right? It all getsdone for me and then I get a little
written description and it'sprobably 80% right, 90% right. It's
close enough for me to startworking. So like that idea that I've
(23:55):
been in that exact same place,right? Like, how do we get someone
to ship on the first day? Andit's like, well, let's find this
really simple isolated bugthat we know they could fix in about
ten minutes because the restwill be setting up their laptop and
figuring out how to push andmaking sure all the access is right.
And then everyone's standingaround and waiting to get paged.
That whole thing has shifted.
(24:16):
Right.
It doesn't have to be this oneliner change anymore. It's like,
I don't know, I could probablyrefactor this whole part of the code
base in about twenty minutes.We can grab lunch and then we'll
push it to production, like.And so I think it's accelerated on
both sides of that in a waythat I think is particularly interesting.
And then kind of on yourinterviewing front, I think even
(24:39):
more broadly, right to yourpoint of you and your co founder
kind of working on this stufftogether. I think as an engineering
leader, the only way for me,the only way for me, everyone's mileage
may vary, whatever, but theonly way for me to like, reason about
that problem is to do some ofit right. Like, I don't, I can't
say, oh, well, I need you toshow up and do this in an interview
(25:02):
or we should look for thesetraits, right? As I'm sort of, like,
thinking through somethingwith, like, directors or whatever,
because I don't know. I don'tknow what it's like to do the job
anymore unless I do it right?Like, I spent 30 years or whatever
the number is in thisindustry, like, living off of my
expertise, the expertise thatI built over all that time. And then
(25:23):
one morning, like, the waythat I did the job was irrelevant,
you know, and to be like,okay, everybody, I need you to do
this today, or whatever. Notthat that that's how we operate,
but you like, you know, togeneralize and make it. Make it simple.
I was like, I don't actuallyknow what I need anyone to do anymore.
So how am I supposed to helpthem even ask good questions, Help
them understand how they couldthink about the problem differently?
(25:44):
Because we're all goingthrough this together, right? And
so I think it's, for me,again, the way that my brain operates,
like, I need to do some of it.It's not all I do is sit around and
write code at this point, butI have to do some of it just to understand
the problems, to understand.And then also, I mean, for me, our
customers are all goingthrough it. And so to think about
(26:05):
the product, to go talk to mycustomers and have a real conversation
about what they're grapplingwith, right? To not have done a bunch
of this work myself would.Would kind of be ludicrous.
What's interesting, too, is,like, I feel like we're quickly approaching.
I feel like a handful ofproblems around, like, novices in
(26:26):
the space. I feel like we'regonna have a. I feel like we're gonna
have almost like a. Just anextremely wide gap between people
like yourself with 30 years ofengineering experience that have
built systems that underst.You know, like, you've written the
code, you've built thesystems. And now we've hit juniors
that are coming in. They'relike, I haven't done that. I've read
a book. I haven't put anythingin prod yet. And I know how to use
(26:51):
an LLM to generate some code,but I don't know what a healthy,
good production system lookslike. I feel like that gap is going
to be a real interesting placefor us to kind of train and make
sure that we don't have a. Aticking time bomb, ten years out
when some of us startretiring, hopefully, please, God,
let me retire in ten years. Ihope they don't have a tick. So I
(27:15):
feel like there's that withthe novices, but then there's this
whole other. That's slightlyterrifying. It seems like a ticking
time bomb of people that wedefinitely need to be able to pass
off experience to, such thatthe machines aren't just doing all
the things blindly. But thenthe other side, which we're also
experiencing, is the nonsoftware developers that are writing
(27:36):
software now, right? And yousee a lot of this on the Twitters
and the blue skies where it'slike a solo founder, I got an MBA
and I built some software.Now, magically, that's fun. But I
think the scarier version ofthat is I have a thirty-five thousand
employee company and the SDRhas magicked software into existence
(27:59):
on his machine, right? Andit's like, oh, we went from we all
run our software in the cloudto SDRs are just making software
on their laptops now, Right?And now our entire organization's
laptops are potential softwaredeployment targets.
Yeah, right.
And I think, I think these twothings are happening at the same
time, which I think is goingto be very interesting for us as
(28:21):
an industry to solve. Like,how do we make sure that these juniors
are getting this knowledgefrom seniors? Like how are we apprenticing
people?
And then at the same time these.
Complete novices are justlike, ah, my laptop is a production
machine now. And it's like,oh, shit.
Yeah, yeah. I mean, I thinkthe latter. I definitely want to
(28:43):
come back to the juniors. Thelatter is an interesting one because
I mean I think of likeMicrosoft Access and Excel, right?
Like Excel is still like thedominant no code programming platform.
I mean, it's got bits of codeor whatever in it, right? But oh
yeah, probably on the planet,or maybe it's Google Sheets at some
point, but like, you know thatit's. I'm a big fan of Wardly Mapping,
(29:09):
which is probably super nerdyhere, but like, like if you go all
the way to left in likeGenesis and just looking at who's
doing what, totally bespoke,like totally for themselves. And
then is there enough of thatto indicate there's a product opportunity,
right? Like if I ran an ITdepartment inside a company, I'd
be going and looking at whateveryone was building in Excel and
say, oh my goodness, can Isolve at least this problem for you?
(29:32):
Right? And so people havingthese tools in their hands and expressing
because they're terrible attelling you what they want, but if
they build it, you can belike, oh, please don't Run that.
But I'll replace it for you inabout three hours. Right. At this
point, because your IT team orwhoever can be like, oh, we could
knock one of those out andit'll be secure and it'll be integrated
(29:52):
into SSO and all these otherthings that we need. I think it's
a huge opportunity if you kindof treat it right. And this is like
shadow it as a. I don't knowif that's a domain, but as a concept
has been around for as long asI've been in this industry and always
stems from trying to preventpeople from solving their own problems.
(30:12):
Yeah, right. We make thingstoo hard to do, so they go find a
way to go around it, right?
Oh, yeah, baby.
And you know, you getcorporate documents sent on Yahoo.
Email or whatever. Like, I'mdefinitely dating myself here. But
like, because, oh, the limit.The limit is too low, right? Everyone's
trying to clamp down theexchange server, so we're just using
Gmail to send each other like,really important critical documents
(30:34):
or whatever. Of course, noweverybody uses Gmail anyway, like,
though those things, right?And why. Why does everyone use Gmail
for their companies now?Because it solved the problem instead
of trying to like, clamp itdown. Right. So I think looking at
how people are expressingtheir needs is really helpful. If
you ignore it, then you'regoing to end up with, yeah, like
(30:55):
connectors out to productionsystems and whatever, Whatever. So
also don't let your SDRs have.Have connectivity to production.
That seems like a good thingto prevent. Right? Like, like find
a way. So whatever. We'llmanage through it. On the junior
front, I'm so torn. Right.Even the cloud. So I mean, I've been
(31:15):
around long enough that I hadto fly to Virginia like everybody
else. We all saw each other inDulles Airport and we drove down
to our respective data centersand racked boxes and crimped cables
and all this kind of stuff.And I happily don't do that anymore.
But I know what a data centerlooks like on the inside. Right?
(31:35):
But I work with engineers who,like, the cloud is this magical thing
where compute magicallyappears. And there are sets of problems,
right? There are classes ofproblems that occur because of the
physics of data centers thatyou can reason about as a software
engineer if you either havestudied enough or have been there,
(32:00):
right? And you're like, oh,well, obviously if these things are
this far apart and this thingneeds to get this from disk instead
of whatever, like the magicstops and the pain starts. Right?
And so I think that we gothrough that same transition. Right.
But the number of cases wherethat actually occurs seems to get
fewer and fewer over time.
(32:21):
Right.
Because people build betterand better abstractions in the cloud.
Right. Higher level servicesthat we can then build on that, that
abstract away some of thechallenging concepts. Right. And
whether that's like using RDSinstead of deploying my own database
in the cloud, because theyhave or even Aurora, like a level
above, primarily AWS will be alittle AWS centric on the tooling,
(32:44):
but like a layer above that.Or spanner, you know, shout out to
gcp like I'm building layersof abstraction so that the people
using them don't have toreason about anymore. The underlying
pieces.
Ideally, yeah.
Which means I reduce thenumber of failure modes where I'm
like, wow, I actually need tounderstand, you know, I, I definitely.
(33:04):
We had a major incident typewhere the way that EBS snapshots
got restored impacted thebehavior of our software and impacted
our service. But like, I couldtell you about that one time in 11
years, you know what I mean?Like, it's not that important that
I understand the details. Sotoday we're in this super hybrid
(33:26):
mode, right? Like AI isgenerating code, but we're looking
at it, we're understanding it,we're not totally sure that the AI
wrote the right thing, etcetera, et cetera. But then the question
is, what tools can we build?It might not just be better LLMs
or better models or bettertraining, but tooling around that,
right? You mentioned tdd.People are super into harnesses now.
And like, how do I, to me islike TDD in a bash script kind of
(33:50):
thing. Like, how do I put thisinto a process where the outcome
is guaranteed or much higherprobability of being good so that
as a junior, what I'm learningis the skill of building harnesses,
not the skill of likeunderstanding all this underlying
software. And I don't knowwhat time horizon we're on to get
there, but I'm hopeful, right?Because I, I think all we do is just
(34:13):
like the people that did itbefore are the only people that can
do it now. Like we'll havelong, illustrious careers. It's been
to like our retirement years.But that's not what we want for the
world, it's not what I wantfor the world. Whereas I think we
really need to rethink what itis we're trying to teach people and
like projecting forward into aworld we don't understand. If I'm
(34:36):
totally honest to say, theskills you're going to need a year
from now are these. So focuson these. Like we're guessing, right?
I mean the, the rate of changeis beyond anything most of us I think
can comprehend to say I reallyneed you to invest in like memory
management. Right. Orwhatever. Like whatever sort of like
lower level concept of, ofcomputing that just people are not
(34:56):
going to have to think about anymore.
Yeah. And it's, it's rough toobecause like as you even give like
a friend who's just switchedcareers like two years ago into computer
science and he's just like,what should I be focusing on? I'm
like, I have no idea. Becauselike it's like, like the problem,
(35:16):
like the thing you might belearning might be solved in like
six months. Right. Which isjust hard. It's like, I mean the
fundamentals, the, thefundamentals and how to give a good
code review. I don't know, it,it does, it does seem like there'll
be quite a big, quite a bigknowledge gap. But yeah, I think
that, you know, educating andtraining people is going to be one
of our bigger challenges. Butyes, it is, it is a wild one. Just
(35:38):
to think like it is, it is ata clip that we have not been at before.
Yeah. The best I can come upwith is like welcome them in and
bring them along for the ride,you know, like this again. I'm working
with a small team as part ofalso like our larger organization
and we have a couple prettyjunior folks on that team. They're
super open to learning. No oneis saying this is not how we've done
(36:00):
it before. Like that's notgoing to work. You know what I mean?
They just have no reason to beskeptical because they're just like
so excited about learningwhatever, which is amazing. Great
enthusiasm and learning thingsthat like, you know, they're just
at a point in their careerwhere they're super absorptive. I
don't know what the right wordis, but like, like just willing to
(36:21):
take on new, new information.New information. And it's not challenging
a core set of beliefs. And sothey're super productive and super
excited to have theopportunity, all that. So, you know,
I think it's like, you know,one option is to worry and try to
plan how we're going to trainpeople. I guarantee we'll be wrong.
So the other option is justlike, let's go on this ride together
(36:43):
and we'll figure it out.
Yeah. What do you thinkhappens to open Source over the next
few years? I know there's beena few like open source is dead kind
of takes. But thinking about,I wouldn't say when I was a junior,
but I was originally a PHPengineer and then I was introduced
to this wonderful thing calledRuby on rails 1.0 a long, long time
(37:07):
ago. And I think it was a jokearound like Ruby 3, where like the
next version of Ruby wouldjust have tab completion for sasses.
I don't know if you rememberthat joke. It was just like it just
got to the point where it'slike it could generate so much. Yeah,
right. There's just like theRails generators were beautiful.
You could just, you could whipcode out in no time, right? But like,
you know, going back to beinga junior engineer, like doing development,
(37:29):
right? Like even seniorengineers, like we're obviously reaching
for dependencies constantly,right? In our software, whether it's,
we're grabbing some serviceand throwing it into a Kubernetes
cluster, whether it's, I'mgrabbing a new library, right? There's
been a lot of takes that likeopen source is data, just doesn't
know it yet. And I feel likebetween that and the amount of agentic
(37:50):
attacks, the supply chainattacks, et cetera, like there is
something interestinghappening in the open source space.
Like, you know, as a CTO of acompany that runs CI and is dealing
with builds, but also, youknow, somebody who's writing software
and has to take dependenciesor has to make decisions about taking
dependencies. Like how are youviewing the world of open source
(38:12):
and the value you get out ofan open source library or piece of
software in this era whereit's so much more likely, I guess,
that there will be agenticattacks or that there will be supply
chain attacks. And how are youguys kind of thinking of open source
in this world?
Yeah, I think there's, I mean,there's clearly market factors at
(38:34):
play, right? And you've nameda couple of them. The security of
open source, themaintainability of open source. We
recently had an opportunity tomeet some folks who run a registry
because their consumption,right, we're talking about consumption
increasing, their consumptionhas gone up to a point. And most
of them are like not forprofit, right? The consumption of
(38:57):
just downloading constantlyout of builds. But even like some
of these data companies, theirentire pipeline just pulls everything
back out of the registries onevery run, right? And they're like,
we can't. You're using it,we're paying for it. Like we can't
actually pay for this thinganymore. How are we going to rebuild
the entire model of Packageregistries, right? That's a real
(39:19):
thing that's happening in our,in our market right now. And I totally
respect that. I have no ideawhat else these people should do.
Like running infrastructure isexpensive, right? And when everyone's,
you know, build volume goes up100x or whatever. And again, it's
not just builds but likeeverywhere this stuff is getting
pulled. So there are a hugenumber of forces at play just in
volume, in exposure. Right.Like again, if you think of who's
(39:43):
building a lot of these opensource packages, it's like one person
did it as like a little petproject for a minute and now they're
holding up the Internet, youknow, with their spare time sort
of thing. And so I think, Idon't believe as a software engineer
that I don't need your opensource package because I can just
write it myself. Like Claudewill spit me out a replacement, right?
(40:04):
Because now I've got a thingthat's never been tested in production,
barely. You know, may not evenbe a good implementation based on
how I got it out of Claude orwhatever. Like I put a lot of value
on sort of production hardenedsoftware, whether it's a third party
service, whether it's an opensource package, whatever it might
be. Also put a lot of value onfocusing on your core domain, right?
(40:28):
Like we feel like we haveinfinite time because we're going
so much faster, but ourbacklogs are still full. Right. Like
we always have more thingsthat we can do for our customers.
Our customers are trying tomove faster. The world is changing.
I need to adapt to that. Likeam I going to sit down and write
a payment platform 100%? No.Like no way. Right? That is. And
(40:48):
there's so much complexity inthat domain that I know nothing about.
So now am I going to downloadleft pad, right? Like the butt of
every open source package?Dependency joke. Like no, it's two
lines of code, right? Like Ithink the pendulum swung a little
bit in some ecosystems too farto the, like I can just get another
(41:10):
package, right? And whenthere's thousands of dependencies
in my project, how am I evergoing to know that one of them got
taken over? Like someone, youknow, table flipped and handed over
the keys to someone they'venever met and that person is only
there so they can drop callsto a C2 system into your life? Like
that's terrifying, right? So Ithink, I think those market dynamics
(41:34):
will combine to have morestable, reliable components. Sorry,
Fewer, larger, stable,reliable components is how I would
think about it. Again, I Don'tbuild most of these things, so who
knows? But I think that superlong tail of like, oh, look, someone
on their afternoon implementedthis thing that I'm thinking about
(41:56):
implementing. Let me justdownload that. I have no idea who
that person is or whether theystill work on it. I think that goes
away, but I think this SaaS isdead. I can build everything myself.
IT systems are dead. I canbuild them all myself. Sure, you
can take the things that yourSDRs are building and running on
their laptop, but is that howyou want to run your organization?
(42:17):
Like, I think there's going tobe, you know, we'll. We'll swing
too far to one side for asecond and then we'll be like, we're
maintaining a ton of stuffthat is not our core business and
it's taking all of our timewhen we could have just bought it
from someone else. And we'retrying to manage internal systems
again that we could have justbought from someone else. Like, I
think we'll. People's pricingmodels will change. The long tail
(42:40):
will maybe disappear. Like,it'll shift and we'll find a new
balance, but it's not going tozero in my mind. I just don't see
how that happens.
Yeah, we've been doing a lotof, like, trying to assess, like,
what is. And I feel like thisis one of those things. It's like
engineering teams should havebeen doing this all along. But, like,
when do we have time? Youknow, like, let's look, really look
(43:01):
at the worthwhileness of adependency. But we started to definitely
do it more and it's come downmore to like, how closely does it
tie to our core businessdomain? And it's like, would it make
sense to own this versus Isthis just a thing that we're going
to have to maintain for nopurpose? But here's a really interesting
one. I think it was the chainguard CEO said OSS is dead, but just
(43:24):
doesn't know it yet. And I waslike, that's a weird one. That's
a weird one to hear say OSS isdead. I was like, oh, oh my gosh,
I hope it's not dead because Istill want free labor for other people.
Yeah, I mean, I think theeconomic economics change, right?
Yeah, the economic and likebuild versus buy or however you think
(43:45):
about acquiring, you know,leveraging dependencies or whatever
is all economic decisions.But. And the economics have changed
significantly, right? Like theeconomics of building have changed
significantly. I don't knowthat we figured out how to apply
LLMs or Gen AI or whatever tothe economics of operating and maintaining,
(44:07):
which is the part that wealready blew that off in build versus
buy decisions. Like, asengineers, we were always terrible
and we're like, oh, noproblem. I could just bang that out
on a weekend. Like, everycompany's CI platform is a bash script
that someone wrote on aweekend. Right. And then suddenly
there's three hundredengineers saying, why is your bash
script broken? And they'relike, I'm supposed to be on vacation.
I hate my job. We're like, whydid you build that? Right? Like,
(44:29):
and just repeat. Like, youknow, copy and repeat for every sort
of internal system. But yeah,I think the economics will change
and to your point, you'll makea better, like a clear decision.
Is this something we shouldreally own? Is it worth the risk
and exposure of taking it fromsomeone else? Like, the math will
change, but it'll still bethere. I don't think it goes to zero.
(44:49):
Well, I know we're coming upon time. I want to be respectful
of your day. I would just loveto know, you know, before we go,
like, what is the mostconcerning shift that you have with
the way that we're writingsoftware? Like, what do you think
is going to be one of thebiggest problems we have to face
coming up?
That's a big one.
Sorry.
Let me know. It's all good. Ithink, you know, I think we've covered
(45:12):
a few of them.
You guys are like. You guysare like, right at the. I feel like
you're at the heart of, like,so much of. We're going to see the
change. I feel like you have aperspective that most folks don't.
Yeah, absolutely. Although Ithink the things that we think about
a lot, I'm not so worriedabout. I just think they'll happen.
Right. Like, we will. We'lloptimize agent flows, we'll remove
humans. We'll figure out howto measure the quality of software
(45:35):
without having to sit and readevery line. Like, I just. I think
that's almost inevitable andit's a question of who does it and
how well they do it and whatshape it takes. Yeah, and we're,
you know, we're very excitedabout that, but I don't. It doesn't
worry me. It's justinteresting what worries me, you
know, how we all manage cost.I think there's efficiency, things
(45:56):
that we haven't quite figuredout, although, like in does in industrialization.
Generally we're good atsorting out efficiency once we decide
there's value in something. SoI think honestly, like the junior
thing and educating andgrowing people, like, again, it's
a little inevitable. We'llfigure it out. But the thing, like,
(46:17):
if I'm going to worry aboutsomething, it's going to be impact
on people, right? And thesetransitions, like, it would be easy
to not create opportunitiesfor junior folks because they don't
have the tools to navigate acomplex transition like this. Right.
It would be easy tounderinvest in training them because
we don't know what you'regoing to need to know, so just figure
it out for yourself kind ofthing. And when I said, like, I don't
(46:39):
know how to train you, butI'll bring you along, like, that
takes. It's likeapprenticeship, right? Like, which
is kind of how I became asoftware engineer. I didn't study
it. I just went into a companywhere someone took me under their
wing and was like, this is howwe build software. I was like, like,
what a great experience.Right? So if we can offer that to
people, then I think that'sgreat. But I don't know that as a,
as an industry, we're great atthat. You know what I mean? And so
(47:00):
I think that's going to takesome real motivation from some specific
people who are driven toreally grow folks. And I think the
open questions, like, when weface open questions as humans, sometimes
we get stuck in indecisionrather than saying, I don't know
what the answer is, but I'mgoing to lead this charge and I'm
going to take these folksunder my wing and we're all going
to be great at this. Right? Soif, if I could worry about something,
(47:22):
it would be, you know, thatkind of human impact.
Yeah. I think that, you know,one of the things I've been thinking
a lot about is just like, whatdoes the value extraction of AI look
like to a business versus thelabor in the business? And I think,
like, that right there is oneof those places where you can kind
of like, you know, it's allnumbers. It's not. It's numbers on,
on two sides of a ledger.Like, you can move them around a
(47:43):
little bit, but I think likethe value that you can shift back
to labor and not just the orggetting value from AI is in that
apprenticeship. I think that'sone of the places that we really
probably could focus. A lot ofthe value that we're getting back
as engineers is like, how dowe spend more of this time? Right.
We're producing a lot morecode, we're producing more revenue,
(48:03):
right? With all the new code,we're producing more revenue. Definitely
questionable for some folks.Right. But, like, now that we're
making more, we're gettingmore done. The backlog is not getting
shorter for some reason, butwe're getting more stuff done. Like,
how do we spend more time onthe people in our orgs? I think that,
yeah, I think thatapprenticeship could definitely be
one of the places we couldshift value extraction that would
(48:24):
be beneficial to teammates andnot just the org.
Yeah, well, the answer, by theway, is we have Gen AI to create
tickets in backlogs also.Right? So, like, it wasn't just engineering
that got to take advantage ofthis amazing new capability. Right.
And so we're, like, sneakilydelivering tickets and, like, PMs
(48:47):
are sitting there typing out,like. And I mean, speaking of humans,
just how all those roles evenshift is super interesting. Not again,
not worried. But I think it'llbe. It'll be interesting. To your
point of, you know, how do webring everyone along and it'll be
different. Like, I'm notgonna. I'm not gonna pretend everyone's
just doing the same job andthey're doing it faster. Right. And
(49:07):
it'll be bumpy. And I justhope that we invest in it and making
sure that. That we do this ina sensible way.
Yeah. Awesome. Well, Rob,thanks so much for coming on the
show today. It was awesome toget to talk to you. Where can people
find you online?
LinkedIn's probably theeasiest place. I'm easy to find there,
so I'll just leave it at thatinstead of trying to rhyme off a
bunch. I also do have my ownpodcast called "The Confident Commit",
(49:28):
if anyone wants to check it out.
Ooh, yeah, put that in theshow notes.
Nice.
Awesome. Well, thanks so muchfor the time. Really appreciate it.
Yeah, thanks for having me.Cory, this has been awesome.
Welcome back to the PlatformEngineering podcast. I'm your host,
Cory O'Daniel. Today on theshow, I'm joined by Rob Zuber, CTO
at CircleCI. Rob's been in thesoftware space for three decades,
been a CTO three times over,and you've been at CircleCI for the
past eleven years?
Yep.
Welcome to the show. CI, thatis... that's where all of our compute
(00:25):
used to go.
Yeah. Yeah. Well, first ofall, thanks for having me. Excited
to be here. Yeah, it used tobe, and it's kind of... it's back
in a new and different way.You know, I think people are trying
to go as fast as they can. Itturns out when you go as fast as
you can, you actually want toknow if your stuff's any good. So
we're reimagining what it isthat we do, but we're right in the
(00:46):
thick of it now, for sure.
Yeah. I feel like there's alot of discussion... I mean, there's
a ton of discussion onlineabout AI in general, but, like, I
feel like a lot of what I seeon LinkedIn and Reddit is people
talking about, like, how dothe fundamental things that we've
kind of built on for the pasttwenty, thirty years, like change
in this new world? And I feellike CI is going to be at the heart
(01:08):
of a lot of it. So, superexcited to talk to you today and
see what you all areexperiencing and what you're seeing
on the other side of the world.
Yeah.
From us, who's just kind ofpushing Docker images up. But yeah,
so, I mean, like, you know,software is changing a fair amount.
We're getting a lot... a fair bit.
Yeah, just a bit.
Open a couple more PRs. Whathas... I guess internally at CircleCI,
(01:32):
what is the load like? I knowGitHub's talking about how much more
changes and pull requests andPRs they're seeing. Are you all seeing
the same type of just sheervelocity change from the amount of
code changes getting pushed through?
Yeah, absolutely. So we'reseeing significant increase in volume...
handling it well from areliability perspective. And then
(01:55):
what we are seeing a mixedchange, if that makes sense. Like,
you know, we think aboutbranch builds where people are checking
to see if the thing they'recurrently working on is working as
expected, and then mainline,trunk, whatever you want to call
it, meaning they've actuallymerged and they're trying to get
something out of production.And we've seen shifts in the mix,
like kind of more branchbuilds. I mean, of course, it's distributed
(02:18):
across customers differently.But in a lot of cases, more branch
builds, meaning there'sactivity, but maybe not as much of
an increase in main builds,meaning people are seeing, you know,
lower success rate. Right. Sothe work of their agent is maybe
okay, but they're using CI andthat branch build as a means t check
(02:38):
that it's good and they'refinding out that it's not. And so
we're investing a huge amountof energy right now in trying to
shorten that feedback cycle,give people things that they can
connect directly to theiragents so that they can get a shorter
cycle basically, and know thattheir work is good and basically
not just good or bad, but givethe feedback directly to the agent
(02:59):
to say, "Actually change this,change this, change this before we
even sort of go through theprocess or the ceremony of sort of
PR and beyond." And so, yeah,absolutely, we're seeing that volume
and what we're seeing as aresult is folks trying to find ways
to tune the whole process sothat it all doesn't just back up
at kind of like PR and then release.
(03:22):
Yeah, I feel like it's a spotof the stack where it's at least,
I feel like twofold going tobe one of the places that slows us
down. A) just the build times,right? CI takes time to run. We're
spending time pushing code upand now we're pushing way more PRs
(03:42):
up than we used to. I mean, Ithink last week I shipped like seventy
PRs or something like that tothe terror of my team, who is on
the other side.The other thingthat's going to slow us down a bit,
I feel like in this adoptionjourney, is the review. The humans
in the loop looking ateverything after it's been pushed
up. But it does seem like oneof those places where it's going
(04:06):
to be prettycriticalasmorepeoplestarttoadopttheseworkflows,thatCIworkswell,fastand itkindofkeeps
youintheloopwith what'sbeingchangedatanextremelyhighvelocity.Whichwehaven'tseenbefore.Whatischanging?Imean,besidesjust What
(04:29):
is changing? I mean, besidesjust the sheer volume. But what are,
I differentlyabout and whatare engineers requestin ayearortwoago?
MediumI think so. So there's awhole category around speed, which
we'll talk about. I'll getinto in a second. And then there's
(04:50):
what I would call agentexperience, which is like I'm actually
operating with the other toolsin my tool chain by letting the agent
do that, right? So theinterfaces, like agents, not going
to a web UI to figure out whatwent wrong. So we're exposing a lot
more direct capabilitiesthrough tools that we could, you
know, we would also use ashumans like CLIS and APIs, but also
(05:12):
MCP server, like those sortsof interfaces that allow the agent
to kind of run its own loopand say, "Okay, I pushed a thing,
I'm watching CI, CI failed,I've got the logs, I know what happened.
I'm pushing again." Likebasically allowing people to get
further away from thatprocess. I mean, we'll come back
to your whole thing abouthumans and where they get involved.
(05:33):
But then on the speed front,there's like absolutely raw compute...
you know, we spent a lot oftime over the years optimizing parallelism
and getting sort of wall clocktime down, not just compute time...
but then there are also some,you know, some shortcuts, right?
Like you could think aboutdoing the same thing as fast as possible,
(05:56):
right? How do I take all thesesteps and get really fast, compute,
optimize, caching, manageparallelism, like all the things
that we think about. But thenyou could think about doing less
work, right? So when we talkabout giving feedback directly to
the agent, first we isolatethe parts that are going to be really
valuable to the agent in thatstep. Right. Certainly these days
(06:16):
when I push something and itfails linting in CI, I'm like, "How
did this happen? How did I getthis far down the process? Oh, now
I gotta like, you know, forceor like, you know, amend my commit,
force push, all these kinds ofthings." It's just like a lot of
overhead that is... yes, itlike the individual step might take
time, but it's more that likeit comes back to me as a human, maybe
(06:38):
I went to get coffee, now Icome back and like, you know, a whole
bunch of time has passed. Soreally pushing all these things down
to the agent. Again, isolatingthe things that really matter, doing
those super, super quick,right? With kind of live standby
compute that, you know, we canrun that check in thirty seconds,
not your ten minute sort ofbuild time. And then things like
(06:58):
isolating specific tests thatwe know that are the ones that will
be impacted by a change andrunning those, maybe prioritizing
them, running them first, onlyrunning those on a branch. Like things
that are not just what isfaster the same... I guess the same
thing, like at a faster linearspeed... but how can we make nonlinear
improvements? Because people,everything else is getting sped up
(07:21):
before that in a way thatmakes... you know, a ten minute build
time used to feel great for alot of people. And now people are
like, "What am I supposed todo for ten minutes?" Right? I mean,
they spin up another agent,another agent, another agent. Like
whether they multi, you know,they multitask and get lost. And
then on the human side... andthis is like we're kind of... we
have touch points all throughthe process, Can we give you a enough
(07:44):
validation and enoughconfidence in the change to reduce
the human involvement? Becauseit is like... you know, I was joking
actually just earlier todaywith some folks on my team that what
I really want just to start islike estimated reading time. You
know, like I look at a pile ofPRs and I'm like, "What can I carve
(08:06):
out time for?" You know, likeon medium or something, it's like
this one's sixty seconds andthis one might take you twenty minutes.
Like just that would be great.But then, you know what we're layering
on top of that is this changeis a critical part of the system.
This change was made bysomeone who maybe doesn't know this
part of the system very well.This might take a little bit more.
This feels like something.Honestly, you could auto merge this,
(08:28):
right? I think we're going tomove away from kind of this big grandiose
ceremony around the PR andinto systems that are designed to
move at the speed. The PR hasalways kind of been like a laborious
gate. Talk to anyone aboutdeveloper efficiency and you end
up having a conversation aboutPR review time, lag, and then everyone
is this heavy exhale, like"ugh." Right. So it doesn't feel
(08:50):
like it's the thing that waseven tuned to be fast when we were
just typing code. And now, youknow, you're trying to go ten, a
hundred times faster.Something has to be different about
that. And so we're investingagain in, within that automation,
how can we give you thestrongest signal to either shorten
or even like reduce the numberof checks tha you have to make?
(09:12):
Yeah, I think that's actuallypretty interesting. There's some
couple of things you saidthere like just the intelligence
and the command or pullrequest itself. I think that with
the number of PRs I stillreview a day, I do think that there
would be a significant lifeimprovement just to seeing like this
one's going to take a minuteto review. It's like, "Okay, let
(09:34):
me get Joe happy and approvethese quick ones for him really quick."
Versus that one's going totake 25 minutes... because sometimes
you'll see a bunch of filechanges, but the file changes do
not correlate to the amount oftime it takes to process and understand
a pull request. That wasinteresting. The idea of being able
to, and I know you can do thistoday with labels and whatnot, but
(09:57):
again, it's one of thoselaborious tasks to do as a bag of
meat. But to be able to tagparts of your system or even repos
and say that this is acritical versus a non critical component
and treating your reviews andPRs differently, that is something
that's hard to do for manyorgs today. While it would be very
(10:19):
beneficial even without AI, Icould see that that would be... like
to be able to look at a biggersystem and be like, "I don't have
to worry about this as muchbecause it's not hitting a core part
of the stack." And like beingable to identify that, that would
be pretty awesome.
Yeah. As a platform that sitsin the middle of software delivery,
(10:41):
we have a pretty goodhistorical understanding of what
changes in your system lead todifficult outcomes. Let's put it
that way. Right. Like we knowwhen things break downstream and
so have a sense that like, oh,this is a place that you should probably
feel pretty good about makinga change. This is a place I would
put a little bit of extraeffort into. Right. There's known
(11:01):
side effects. Historicallythis has led to, you know, those
like tightly coupled placeswhere everybody changes one thing
and forgets another and thenwe're like, "Oh yeah, that incident
happened here, et cetera, etcetera." So like, I think there's...
I guess intelligence begetsintelligence in a way. Like we've
applied intelligence to a partof the process and now we have a
lot of opportunity to apply itelsewhere, which is really the only
(11:22):
way we're going to be able tospeed. Now I'm saying let's call
it artificial intelligence.Before we like we've applied humans
to the process before, butagain it's sort of this like intuition.
I find this really interestingin general, right. Like we have humans
who have a very unevenlydistributed mental model of our system.
(11:44):
Whether it's who in theorganization is really trustworthy
and who we should double checktheir work or is just growing or
whatever and where in thesystem things are a little sketch.
And I don't feel supercomfortable that the test coverage
here, even though the numberis high, really truly gives me confidence
that I can Put this thing out,this thing's, you know, super robust.
(12:06):
And it's like some people knowit, some people don't, some people,
whatever. And I think whatwe're really seeing now is the ability
to extract that and have itapplied more universally, right?
Like as people build outskills and context and whatever with
their organization to be ableto say, cool, this change. Like regardless
(12:30):
of who created it, this andwho's looking at it, this system
has all of that knowledge orcan gain all of that knowledge and
retain it in a useful way ifwe manage it effectively. I think
it was like the late nineties,we spent all this time talking about
expert systems, which waslike, you remember that? And now
(12:51):
we finally built them and allthey are is a pile of markdown and
a repo. But it's like we tookeveryone's knowledge and we put it
somewhere we actually are ableto apply it. Like that was a fool's
errand in the late nineties,right? And now it's just like, it's
just kind of happening becausepeople are type writing stuff down
or they work on something andthey're like, "Hey, Claude (or whoever/whatever
tool), can you write this downfor me and make a record of it?"
And as that stuff gets morecentralized, right. We're able to
(13:15):
extract and consolidate theselittle pockets of, of understanding
that we're just, you know, ifthis person happens to be the person
that reads your pr, it'll gookay. Which is not great. Organizational
design was never designed.It's just what happened.
Yeah, the code owner, it'sjust like, yeah, Cory owns it. If
he. It's going to be fine.It's like, yeah, maybe, maybe not.
(13:39):
Yeah, no, that is. I feel likethere's going to be a lot of really
interesting changes to the waythat we write and deliver software.
I mean, I know even in my ownpractice, like, I'm a very TDD oriented
engineer and I'm a person whoopened to PR very early. I love seeing
a failing build. I love seeingthem go green. I just open and let
(14:01):
him rip. But I've done a lotof engineering myself just around
my workflow that I've had torethink my own development workflow
itself as I started to adoptAI tools. So a thing that I used
to do a lot was I don't lovethe idea of a linter failing my build
on a CI run. I don't mindseeing a test fail that I'm working
(14:24):
on. Great. That's what'sexpected. I want my teammates to
see that test failing, so theycan see where I'm at in the process.
But I've always leaned intothe pre commits and whatnot. It's
like, hey, get most of thisstuff. Just make sure it's good before
you even waste anybody's time.And now I'm just like, there's way
too many agents running forthem all to be running all my pre
commit tooling locally. Thatis CI's problem now, right? And it's
(14:48):
just like. So it's likesomething that I had spent a lot
of time like, ah, trying to belike a good steward of CI. I'm just
like, commodity compute, letthat all run over there. Like, I
don't have time for this thingto run locally anymore. But it is
funny because it's like that,that ten minutes that you're waiting,
then you're like, oh, I guessI start up another agent. Like, that's
exactly what happened to me. Iwas like, when I first started, I
(15:08):
was like, I'm only going to doone at a time so I don't drive myself
insane with a thousand PRs.And then the first time you're staring,
you're like, shit, I don'thave anything to do for ten minutes.
You're like, I could writethat spec in ten minutes.
Yeah.
And then all of a sudden yougot two. And then all of a sudden
you got, I think eight rightnow running in the background. So
as far as like your own team'sconcerned, like, what are some of
(15:31):
the things that, you know,you've got a, you got a big team,
you got a few hundredengineers right around.
About a hundred, yeah.
A hundred. So like, what areyou seeing not in the CI process
itself falling apart, butlike, like, what are you seeing,
like, starting to break justfor, like, larger engineering teams
that are starting to adoptthese tools.
There's a lot of things tothink about there. And certainly
(15:52):
I'm working also with a veryspecific team as well as the broader
organization. And in thatspecific team, one of our goals or
mandates was like, let's runas fast as we can and see what breaks.
And yeah, it took an hourbefore we were like, well, this is
never going to work. And itwas PR reviews, like two of us sitting
across the table from eachother hammering out PRs. And we're
(16:13):
like, okay, when are we goingto stop and review each other's PRs?
And then I looked at them andI was like, how am I going to get
through this? We're going tobe here for the rest of the week.
Right. And so then we startedbuilding tooling to review PRs, right?
Like how do we. And weactually built something, I mean,
speaking of the people youtrust, we built a thing that extracts
kind of the most common andmost trusted reviewers from the history
(16:34):
of a repo, turns that intocontext and applies it to the next
round of reviews. So it says,okay, well we know someone's going
to comment on this and on thisand on this and like how many senior
or staff or you know, whateverthe top level of your org is, engineers,
open up a PR and go,seriously, again with this mistake.
Like, we've made this mistakea hundred times. We've talked about
(16:56):
it. Like speaking of thelinter, right? And why would I put
this in front of someone elseand use their time. We know what
all the first round of reviewcomments are going to be, so let's
just have them happen locally,right? Like basically we, within
our agents, right, we haveskills that are like execute this
type of review, you know,before it ever gets committed and
pushed. So that by the timewe're asking a human to look at it,
(17:20):
I mean, honestly, they're alsopulling some little local CLI and
running some review over it,saying pull out the things that I
care about kind of thing. Solike trying to optimize a lot of
that stuff. So review isprobably not surprising, but it's
pretty easy to pull some stufftogether that will, that will optimize
that a lot. And then, youknow, I would say a hot topic in
every circle that I travel inat the moment, engineering leaders,
(17:41):
and that circleci is nodifferent is token consumption. And
what's interesting there isefficiency, right? Like the way that
some people use LLMs versusthe way that other people use LLMs,
just very, very different. Andas a result, right? Like there's
a lot of discussion ofleaderboards and token maxing and
(18:03):
all this kind of stuff. Like Idon't really care about that. I would
prefer if people aren't justtrying to burn tokens to make it
look like they're doing work.But for folks that are using them
in good faith, like we seereally big differences in sort of
outcomes relative to kind ofinputs and outputs, right? And so
trying to help peopleunderstand how can I use this effectively,
(18:26):
where was all this energygetting used, right? And then can
we tool around that? Can webuild little CLIs or skills or whatever's,
you know, whatever's the rightthing to get you to an outcome faster?
Because yes, that's like tokencost is real, but also just as A
human, you're trying to getsomething done and someone else has
figured out how to get the LMto get that thing done much faster
than you seem to be able toget it done. And that's, that's no
(18:48):
shade on people. Like, we'reall learning this toolkit together.
So how do we bring thosethings together? How do we create
sort of shared repos of skillsand stuff like that internally to
again give people the boost ofother people's learning? Like, I
think that's. If I were tosummarize all that. Like it started
with token costs, but reallyit's fractured learning, right? Because
there's no. I can't just, youknow, go on to Amazon and buy the
(19:11):
book on how to be awesome atusing whatever today's coding agent
is, because by tomorrow it'sgoing to be a different one or like
a new version and optimize fordifferent things. New model versions
drop. And like your promptsthat used to be great are now a little,
you know, not working verywell. So there's, there's all kinds
of those different pocketsagain of like fractured learning.
Ironically, I was talkingearlier about how we're bringing
(19:31):
all our knowledge together andso this is a new area that's just
like we're learning soquickly. So that had. Bringing everyone
along is hard, right? Like ineven. Even I'll say just 100 people,
right? There's a decent sizedengineering org, but I know people
in engineering orgs with35,000 engineers. I can't even imagine
what that's like, let alonehow you try to educate them all on
(19:53):
how to be really effectivewith this kind of tooling. So I don't
like we have the standardproblems, right? Cycle times of reviews,
you know, we optimize a lot ofit. But it's really, how do we do
that then consistently acrossthe org. Not that everything is like
cookie cutter stamped it, butjust without leaving people behind
who are sort of likestruggling or.
(20:14):
Yeah, yeah, no, I feel likeeven at 100 engineers, that's gotta
be tough. I don't know how.I'll be very curious how these larger,
larger, larger organizationshandle it. But like we're fairly
small team and you know, as westarted leaning into it, like it
was just, it just felt likeall of a sudden there was forty employees
one day. Like it was just likethe amount of PRs. It was like, oh,
(20:37):
two people on the team havefigured it out. And then it was like,
we got to get. Now the rest ofthe team has to figure it out. Or
like, the value extraction'soff, right? Like, if two people,
you know what I'm saying, arelike, are outputting 40x, that sucks
for everybody else that has toreview that code. Right now, their
job is reviewing code, notwriting code, which, that's the worst
place to be. Like, I haven'tfigured out how to use this tool
(20:59):
yet, and my reward isreviewing everybody else's output
because I haven't caught up.
Ops teams, you're probablyused to doing all the heavy lifting
when it comes toinfrastructure as code wrangling
root modules, CI/CD scriptsand Terraform, just to keep things
moving along. What if yourdevelopers could just diagram what
they want and you still gotall the control and visibility you
need? That's exactly whatMassdriver does. Ops teams upload
your trusted infrastructure ascode modules to our registry.Your
(21:22):
developers, they don't have totouch Terraform, build root modules,
or even copy a single line ofCI/CD scripts. They just diagram
their cloud infrastructure.Massdriver pulls the modules and
(21:47):
deploys exactly what's ontheir canvas. The result? It's still
managed as code, but withcomplete audit trails, rollbacks,
preview environments and costcontrols. You'll see exactly who's
using what, where and whatresources they're producing, all
without the chaos. Stop doingtwice the work. Start making Infrastructure
as Code simpler withMassdriver. Learn more at Massdriver.cloud.
I was talking to my co founderabout this. It's like, he's like,
(22:08):
it. It just feel like beforehe'd leaned into it, before he kind
of like, got the hang of it.He's like, it just. He's like. It
feels like I, you know, don'treally like, what value am I adding
to the team? Like, like, youguys move so fast without me, right?
And then it's just like. Thenyou see him, like, start to catch
up, and then all of a suddenit's just the volume of code is there,
right? And it's just likethey're all kind of working on the
(22:28):
guardrails and catching stuffand making sure that good code quality
is going in. And it's likeonce you get to that part and you
start to feel it humming, itdoes feel really good. But then it's
that next step of when webring on our next teammate. What
does interviewing look like,right? What does bringing them into
the fold look like? Where allof a sudden it's like, you know,
(22:49):
you think about five yearsago, six years ago, I feel like our
number one goal in, like,bringing on a new hire was like,
can we get them to putsomething into production on day
one? Like, could you imaginesitting down at a job and you're
like, oh, let's go take a lookat the ol at how many get up PRs
there are. And it's like, oh,there's six thousand that were open
today. Like the, just thesheer intimidation of joining a team
(23:11):
that's kind of figuring itout. Could be, yeah.
It feels like in a way it's aninteresting description. Like you're
trying, you know, you'retrying to jump onto a moving freight
train kind of thing. But youcould also, like, your speed is so
enhanced, right? Like myability to drop into a code base
I've never looked at and belike, how does this work? Tell me
what the key pieces are. Howwould I implement this thing? And
just, you know, grepping,grepping, grepping, whatever. Like
(23:34):
what I used to do to figureout a code base. Oh, let me pop open
this file, look up this file.Oh, maybe here's a. Here, this looks
interesting. I wonder if I canfind a reference to this function
somewhere. Right? It all getsdone for me and then I get a little
written description and it'sprobably 80% right, 90% right. It's
close enough for me to startworking. So like that idea that I've
(23:55):
been in that exact same place,right? Like, how do we get someone
to ship on the first day? Andit's like, well, let's find this
really simple isolated bugthat we know they could fix in about
ten minutes because the restwill be setting up their laptop and
figuring out how to push andmaking sure all the access is right.
And then everyone's standingaround and waiting to get paged.
That whole thing has shifted.
(24:16):
Right.
It doesn't have to be this oneliner change anymore. It's like,
I don't know, I could probablyrefactor this whole part of the code
base in about twenty minutes.We can grab lunch and then we'll
push it to production, like.And so I think it's accelerated on
both sides of that in a waythat I think is particularly interesting.
And then kind of on yourinterviewing front, I think even
(24:39):
more broadly, right to yourpoint of you and your co founder
kind of working on this stufftogether. I think as an engineering
leader, the only way for me,the only way for me, everyone's mileage
may vary, whatever, but theonly way for me to like, reason about
that problem is to do some ofit right. Like, I don't, I can't
say, oh, well, I need you toshow up and do this in an interview
(25:02):
or we should look for thesetraits, right? As I'm sort of, like,
thinking through somethingwith, like, directors or whatever,
because I don't know. I don'tknow what it's like to do the job
anymore unless I do it right?Like, I spent 30 years or whatever
the number is in thisindustry, like, living off of my
expertise, the expertise thatI built over all that time. And then
(25:23):
one morning, like, the waythat I did the job was irrelevant,
you know, and to be like,okay, everybody, I need you to do
this today, or whatever. Notthat that that's how we operate,
but you like, you know, togeneralize and make it. Make it simple.
I was like, I don't actuallyknow what I need anyone to do anymore.
So how am I supposed to helpthem even ask good questions, Help
them understand how they couldthink about the problem differently?
(25:44):
Because we're all goingthrough this together, right? And
so I think it's, for me,again, the way that my brain operates,
like, I need to do some of it.It's not all I do is sit around and
write code at this point, butI have to do some of it just to understand
the problems, to understand.And then also, I mean, for me, our
customers are all goingthrough it. And so to think about
(26:05):
the product, to go talk to mycustomers and have a real conversation
about what they're grapplingwith, right? To not have done a bunch
of this work myself would.Would kind of be ludicrous.
What's interesting, too, is,like, I feel like we're quickly approaching.
I feel like a handful ofproblems around, like, novices in
(26:26):
the space. I feel like we'regonna have a. I feel like we're gonna
have almost like a. Just anextremely wide gap between people
like yourself with 30 years ofengineering experience that have
built systems that underst.You know, like, you've written the
code, you've built thesystems. And now we've hit juniors
that are coming in. They'relike, I haven't done that. I've read
a book. I haven't put anythingin prod yet. And I know how to use
(26:51):
an LLM to generate some code,but I don't know what a healthy,
good production system lookslike. I feel like that gap is going
to be a real interesting placefor us to kind of train and make
sure that we don't have a. Aticking time bomb, ten years out
when some of us startretiring, hopefully, please, God,
let me retire in ten years. Ihope they don't have a tick. So I
(27:15):
feel like there's that withthe novices, but then there's this
whole other. That's slightlyterrifying. It seems like a ticking
time bomb of people that wedefinitely need to be able to pass
off experience to, such thatthe machines aren't just doing all
the things blindly. But thenthe other side, which we're also
experiencing, is the nonsoftware developers that are writing
(27:36):
software now, right? And yousee a lot of this on the Twitters
and the blue skies where it'slike a solo founder, I got an MBA
and I built some software.Now, magically, that's fun. But I
think the scarier version ofthat is I have a thirty-five thousand
employee company and the SDRhas magicked software into existence
(27:59):
on his machine, right? Andit's like, oh, we went from we all
run our software in the cloudto SDRs are just making software
on their laptops now, Right?And now our entire organization's
laptops are potential softwaredeployment targets.
Yeah, right.
And I think, I think these twothings are happening at the same
time, which I think is goingto be very interesting for us as
(28:21):
an industry to solve. Like,how do we make sure that these juniors
are getting this knowledgefrom seniors? Like how are we apprenticing
people?
And then at the same time these.
Complete novices are justlike, ah, my laptop is a production
machine now. And it's like,oh, shit.
Yeah, yeah. I mean, I thinkthe latter. I definitely want to
(28:43):
come back to the juniors. Thelatter is an interesting one because
I mean I think of likeMicrosoft Access and Excel, right?
Like Excel is still like thedominant no code programming platform.
I mean, it's got bits of codeor whatever in it, right? But oh
yeah, probably on the planet,or maybe it's Google Sheets at some
point, but like, you know thatit's. I'm a big fan of Wardly Mapping,
(29:09):
which is probably super nerdyhere, but like, like if you go all
the way to left in likeGenesis and just looking at who's
doing what, totally bespoke,like totally for themselves. And
then is there enough of thatto indicate there's a product opportunity,
right? Like if I ran an ITdepartment inside a company, I'd
be going and looking at whateveryone was building in Excel and
say, oh my goodness, can Isolve at least this problem for you?
(29:32):
Right? And so people havingthese tools in their hands and expressing
because they're terrible attelling you what they want, but if
they build it, you can belike, oh, please don't Run that.
But I'll replace it for you inabout three hours. Right. At this
point, because your IT team orwhoever can be like, oh, we could
knock one of those out andit'll be secure and it'll be integrated
(29:52):
into SSO and all these otherthings that we need. I think it's
a huge opportunity if you kindof treat it right. And this is like
shadow it as a. I don't knowif that's a domain, but as a concept
has been around for as long asI've been in this industry and always
stems from trying to preventpeople from solving their own problems.
(30:12):
Yeah, right. We make thingstoo hard to do, so they go find a
way to go around it, right?
Oh, yeah, baby.
And you know, you getcorporate documents sent on Yahoo.
Email or whatever. Like, I'mdefinitely dating myself here. But
like, because, oh, the limit.The limit is too low, right? Everyone's
trying to clamp down theexchange server, so we're just using
Gmail to send each other like,really important critical documents
(30:34):
or whatever. Of course, noweverybody uses Gmail anyway, like,
though those things, right?And why. Why does everyone use Gmail
for their companies now?Because it solved the problem instead
of trying to like, clamp itdown. Right. So I think looking at
how people are expressingtheir needs is really helpful. If
you ignore it, then you'regoing to end up with, yeah, like
(30:55):
connectors out to productionsystems and whatever, Whatever. So
also don't let your SDRs have.Have connectivity to production.
That seems like a good thingto prevent. Right? Like, like find
a way. So whatever. We'llmanage through it. On the junior
front, I'm so torn. Right.Even the cloud. So I mean, I've been
(31:15):
around long enough that I hadto fly to Virginia like everybody
else. We all saw each other inDulles Airport and we drove down
to our respective data centersand racked boxes and crimped cables
and all this kind of stuff.And I happily don't do that anymore.
But I know what a data centerlooks like on the inside. Right?
(31:35):
But I work with engineers who,like, the cloud is this magical thing
where compute magicallyappears. And there are sets of problems,
right? There are classes ofproblems that occur because of the
physics of data centers thatyou can reason about as a software
engineer if you either havestudied enough or have been there,
(32:00):
right? And you're like, oh,well, obviously if these things are
this far apart and this thingneeds to get this from disk instead
of whatever, like the magicstops and the pain starts. Right?
And so I think that we gothrough that same transition. Right.
But the number of cases wherethat actually occurs seems to get
fewer and fewer over time.
(32:21):
Right.
Because people build betterand better abstractions in the cloud.
Right. Higher level servicesthat we can then build on that, that
abstract away some of thechallenging concepts. Right. And
whether that's like using RDSinstead of deploying my own database
in the cloud, because theyhave or even Aurora, like a level
above, primarily AWS will be alittle AWS centric on the tooling,
(32:44):
but like a layer above that.Or spanner, you know, shout out to
gcp like I'm building layersof abstraction so that the people
using them don't have toreason about anymore. The underlying
pieces.
Ideally, yeah.
Which means I reduce thenumber of failure modes where I'm
like, wow, I actually need tounderstand, you know, I, I definitely.
(33:04):
We had a major incident typewhere the way that EBS snapshots
got restored impacted thebehavior of our software and impacted
our service. But like, I couldtell you about that one time in 11
years, you know what I mean?Like, it's not that important that
I understand the details. Sotoday we're in this super hybrid
(33:26):
mode, right? Like AI isgenerating code, but we're looking
at it, we're understanding it,we're not totally sure that the AI
wrote the right thing, etcetera, et cetera. But then the question
is, what tools can we build?It might not just be better LLMs
or better models or bettertraining, but tooling around that,
right? You mentioned tdd.People are super into harnesses now.
And like, how do I, to me islike TDD in a bash script kind of
(33:50):
thing. Like, how do I put thisinto a process where the outcome
is guaranteed or much higherprobability of being good so that
as a junior, what I'm learningis the skill of building harnesses,
not the skill of likeunderstanding all this underlying
software. And I don't knowwhat time horizon we're on to get
there, but I'm hopeful, right?Because I, I think all we do is just
(34:13):
like the people that did itbefore are the only people that can
do it now. Like we'll havelong, illustrious careers. It's been
to like our retirement years.But that's not what we want for the
world, it's not what I wantfor the world. Whereas I think we
really need to rethink what itis we're trying to teach people and
like projecting forward into aworld we don't understand. If I'm
(34:36):
totally honest to say, theskills you're going to need a year
from now are these. So focuson these. Like we're guessing, right?
I mean the, the rate of changeis beyond anything most of us I think
can comprehend to say I reallyneed you to invest in like memory
management. Right. Orwhatever. Like whatever sort of like
lower level concept of, ofcomputing that just people are not
(34:56):
going to have to think about anymore.
Yeah. And it's, it's rough toobecause like as you even give like
a friend who's just switchedcareers like two years ago into computer
science and he's just like,what should I be focusing on? I'm
like, I have no idea. Becauselike it's like, like the problem,
(35:16):
like the thing you might belearning might be solved in like
six months. Right. Which isjust hard. It's like, I mean the
fundamentals, the, thefundamentals and how to give a good
code review. I don't know, it,it does, it does seem like there'll
be quite a big, quite a bigknowledge gap. But yeah, I think
that, you know, educating andtraining people is going to be one
of our bigger challenges. Butyes, it is, it is a wild one. Just
(35:38):
to think like it is, it is ata clip that we have not been at before.
Yeah. The best I can come upwith is like welcome them in and
bring them along for the ride,you know, like this again. I'm working
with a small team as part ofalso like our larger organization
and we have a couple prettyjunior folks on that team. They're
super open to learning. No oneis saying this is not how we've done
(36:00):
it before. Like that's notgoing to work. You know what I mean?
They just have no reason to beskeptical because they're just like
so excited about learningwhatever, which is amazing. Great
enthusiasm and learning thingsthat like, you know, they're just
at a point in their careerwhere they're super absorptive. I
don't know what the right wordis, but like, like just willing to
(36:21):
take on new, new information.New information. And it's not challenging
a core set of beliefs. And sothey're super productive and super
excited to have theopportunity, all that. So, you know,
I think it's like, you know,one option is to worry and try to
plan how we're going to trainpeople. I guarantee we'll be wrong.
So the other option is justlike, let's go on this ride together
(36:43):
and we'll figure it out.
Yeah. What do you thinkhappens to open Source over the next
few years? I know there's beena few like open source is dead kind
of takes. But thinking about,I wouldn't say when I was a junior,
but I was originally a PHPengineer and then I was introduced
to this wonderful thing calledRuby on rails 1.0 a long, long time
(37:07):
ago. And I think it was a jokearound like Ruby 3, where like the
next version of Ruby wouldjust have tab completion for sasses.
I don't know if you rememberthat joke. It was just like it just
got to the point where it'slike it could generate so much. Yeah,
right. There's just like theRails generators were beautiful.
You could just, you could whipcode out in no time, right? But like,
you know, going back to beinga junior engineer, like doing development,
(37:29):
right? Like even seniorengineers, like we're obviously reaching
for dependencies constantly,right? In our software, whether it's,
we're grabbing some serviceand throwing it into a Kubernetes
cluster, whether it's, I'mgrabbing a new library, right? There's
been a lot of takes that likeopen source is data, just doesn't
know it yet. And I feel likebetween that and the amount of agentic
(37:50):
attacks, the supply chainattacks, et cetera, like there is
something interestinghappening in the open source space.
Like, you know, as a CTO of acompany that runs CI and is dealing
with builds, but also, youknow, somebody who's writing software
and has to take dependenciesor has to make decisions about taking
dependencies. Like how are youviewing the world of open source
(38:12):
and the value you get out ofan open source library or piece of
software in this era whereit's so much more likely, I guess,
that there will be agenticattacks or that there will be supply
chain attacks. And how are youguys kind of thinking of open source
in this world?
Yeah, I think there's, I mean,there's clearly market factors at
(38:34):
play, right? And you've nameda couple of them. The security of
open source, themaintainability of open source. We
recently had an opportunity tomeet some folks who run a registry
because their consumption,right, we're talking about consumption
increasing, their consumptionhas gone up to a point. And most
of them are like not forprofit, right? The consumption of
(38:57):
just downloading constantlyout of builds. But even like some
of these data companies, theirentire pipeline just pulls everything
back out of the registries onevery run, right? And they're like,
we can't. You're using it,we're paying for it. Like we can't
actually pay for this thinganymore. How are we going to rebuild
the entire model of Packageregistries, right? That's a real
(39:19):
thing that's happening in our,in our market right now. And I totally
respect that. I have no ideawhat else these people should do.
Like running infrastructure isexpensive, right? And when everyone's,
you know, build volume goes up100x or whatever. And again, it's
not just builds but likeeverywhere this stuff is getting
pulled. So there are a hugenumber of forces at play just in
volume, in exposure. Right.Like again, if you think of who's
(39:43):
building a lot of these opensource packages, it's like one person
did it as like a little petproject for a minute and now they're
holding up the Internet, youknow, with their spare time sort
of thing. And so I think, Idon't believe as a software engineer
that I don't need your opensource package because I can just
write it myself. Like Claudewill spit me out a replacement, right?
(40:04):
Because now I've got a thingthat's never been tested in production,
barely. You know, may not evenbe a good implementation based on
how I got it out of Claude orwhatever. Like I put a lot of value
on sort of production hardenedsoftware, whether it's a third party
service, whether it's an opensource package, whatever it might
be. Also put a lot of value onfocusing on your core domain, right?
(40:28):
Like we feel like we haveinfinite time because we're going
so much faster, but ourbacklogs are still full. Right. Like
we always have more thingsthat we can do for our customers.
Our customers are trying tomove faster. The world is changing.
I need to adapt to that. Likeam I going to sit down and write
a payment platform 100%? No.Like no way. Right? That is. And
(40:48):
there's so much complexity inthat domain that I know nothing about.
So now am I going to downloadleft pad, right? Like the butt of
every open source package?Dependency joke. Like no, it's two
lines of code, right? Like Ithink the pendulum swung a little
bit in some ecosystems too farto the, like I can just get another
(41:10):
package, right? And whenthere's thousands of dependencies
in my project, how am I evergoing to know that one of them got
taken over? Like someone, youknow, table flipped and handed over
the keys to someone they'venever met and that person is only
there so they can drop callsto a C2 system into your life? Like
that's terrifying, right? So Ithink, I think those market dynamics
(41:34):
will combine to have morestable, reliable components. Sorry,
Fewer, larger, stable,reliable components is how I would
think about it. Again, I Don'tbuild most of these things, so who
knows? But I think that superlong tail of like, oh, look, someone
on their afternoon implementedthis thing that I'm thinking about
(41:56):
implementing. Let me justdownload that. I have no idea who
that person is or whether theystill work on it. I think that goes
away, but I think this SaaS isdead. I can build everything myself.
IT systems are dead. I canbuild them all myself. Sure, you
can take the things that yourSDRs are building and running on
their laptop, but is that howyou want to run your organization?
(42:17):
Like, I think there's going tobe, you know, we'll. We'll swing
too far to one side for asecond and then we'll be like, we're
maintaining a ton of stuffthat is not our core business and
it's taking all of our timewhen we could have just bought it
from someone else. And we'retrying to manage internal systems
again that we could have justbought from someone else. Like, I
think we'll. People's pricingmodels will change. The long tail
(42:40):
will maybe disappear. Like,it'll shift and we'll find a new
balance, but it's not going tozero in my mind. I just don't see
how that happens.
Yeah, we've been doing a lotof, like, trying to assess, like,
what is. And I feel like thisis one of those things. It's like
engineering teams should havebeen doing this all along. But, like,
when do we have time? Youknow, like, let's look, really look
(43:01):
at the worthwhileness of adependency. But we started to definitely
do it more and it's come downmore to like, how closely does it
tie to our core businessdomain? And it's like, would it make
sense to own this versus Isthis just a thing that we're going
to have to maintain for nopurpose? But here's a really interesting
one. I think it was the chainguard CEO said OSS is dead, but just
(43:24):
doesn't know it yet. And I waslike, that's a weird one. That's
a weird one to hear say OSS isdead. I was like, oh, oh my gosh,
I hope it's not dead because Istill want free labor for other people.
Yeah, I mean, I think theeconomic economics change, right?
Yeah, the economic and likebuild versus buy or however you think
(43:45):
about acquiring, you know,leveraging dependencies or whatever
is all economic decisions.But. And the economics have changed
significantly, right? Like theeconomics of building have changed
significantly. I don't knowthat we figured out how to apply
LLMs or Gen AI or whatever tothe economics of operating and maintaining,
(44:07):
which is the part that wealready blew that off in build versus
buy decisions. Like, asengineers, we were always terrible
and we're like, oh, noproblem. I could just bang that out
on a weekend. Like, everycompany's CI platform is a bash script
that someone wrote on aweekend. Right. And then suddenly
there's three hundredengineers saying, why is your bash
script broken? And they'relike, I'm supposed to be on vacation.
I hate my job. We're like, whydid you build that? Right? Like,
(44:29):
and just repeat. Like, youknow, copy and repeat for every sort
of internal system. But yeah,I think the economics will change
and to your point, you'll makea better, like a clear decision.
Is this something we shouldreally own? Is it worth the risk
and exposure of taking it fromsomeone else? Like, the math will
change, but it'll still bethere. I don't think it goes to zero.
(44:49):
Well, I know we're coming upon time. I want to be respectful
of your day. I would just loveto know, you know, before we go,
like, what is the mostconcerning shift that you have with
the way that we're writingsoftware? Like, what do you think
is going to be one of thebiggest problems we have to face
coming up?
That's a big one.
Sorry.
Let me know. It's all good. Ithink, you know, I think we've covered
(45:12):
a few of them.
You guys are like. You guysare like, right at the. I feel like
you're at the heart of, like,so much of. We're going to see the
change. I feel like you have aperspective that most folks don't.
Yeah, absolutely. Although Ithink the things that we think about
a lot, I'm not so worriedabout. I just think they'll happen.
Right. Like, we will. We'lloptimize agent flows, we'll remove
humans. We'll figure out howto measure the quality of software
(45:35):
without having to sit and readevery line. Like, I just. I think
that's almost inevitable andit's a question of who does it and
how well they do it and whatshape it takes. Yeah, and we're,
you know, we're very excitedabout that, but I don't. It doesn't
worry me. It's justinteresting what worries me, you
know, how we all manage cost.I think there's efficiency, things
(45:56):
that we haven't quite figuredout, although, like in does in industrialization.
Generally we're good atsorting out efficiency once we decide
there's value in something. SoI think honestly, like the junior
thing and educating andgrowing people, like, again, it's
a little inevitable. We'llfigure it out. But the thing, like,
(46:17):
if I'm going to worry aboutsomething, it's going to be impact
on people, right? And thesetransitions, like, it would be easy
to not create opportunitiesfor junior folks because they don't
have the tools to navigate acomplex transition like this. Right.
It would be easy tounderinvest in training them because
we don't know what you'regoing to need to know, so just figure
it out for yourself kind ofthing. And when I said, like, I don't
(46:39):
know how to train you, butI'll bring you along, like, that
takes. It's likeapprenticeship, right? Like, which
is kind of how I became asoftware engineer. I didn't study
it. I just went into a companywhere someone took me under their
wing and was like, this is howwe build software. I was like, like,
what a great experience.Right? So if we can offer that to
people, then I think that'sgreat. But I don't know that as a,
as an industry, we're great atthat. You know what I mean? And so
(47:00):
I think that's going to takesome real motivation from some specific
people who are driven toreally grow folks. And I think the
open questions, like, when weface open questions as humans, sometimes
we get stuck in indecisionrather than saying, I don't know
what the answer is, but I'mgoing to lead this charge and I'm
going to take these folksunder my wing and we're all going
to be great at this. Right? Soif, if I could worry about something,
(47:22):
it would be, you know, thatkind of human impact.
Yeah. I think that, you know,one of the things I've been thinking
a lot about is just like, whatdoes the value extraction of AI look
like to a business versus thelabor in the business? And I think,
like, that right there is oneof those places where you can kind
of like, you know, it's allnumbers. It's not. It's numbers on,
on two sides of a ledger.Like, you can move them around a
(47:43):
little bit, but I think likethe value that you can shift back
to labor and not just the orggetting value from AI is in that
apprenticeship. I think that'sone of the places that we really
probably could focus. A lot ofthe value that we're getting back
as engineers is like, how dowe spend more of this time? Right.
We're producing a lot morecode, we're producing more revenue,
(48:03):
right? With all the new code,we're producing more revenue. Definitely
questionable for some folks.Right. But, like, now that we're
making more, we're gettingmore done. The backlog is not getting
shorter for some reason, butwe're getting more stuff done. Like,
how do we spend more time onthe people in our orgs? I think that,
yeah, I think thatapprenticeship could definitely be
one of the places we couldshift value extraction that would
(48:24):
be beneficial to teammates andnot just the org.
Yeah, well, the answer, by theway, is we have Gen AI to create
tickets in backlogs also.Right? So, like, it wasn't just engineering
that got to take advantage ofthis amazing new capability. Right.
And so we're, like, sneakilydelivering tickets and, like, PMs
(48:47):
are sitting there typing out,like. And I mean, speaking of humans,
just how all those roles evenshift is super interesting. Not again,
not worried. But I think it'llbe. It'll be interesting. To your
point of, you know, how do webring everyone along and it'll be
different. Like, I'm notgonna. I'm not gonna pretend everyone's
just doing the same job andthey're doing it faster. Right. And
(49:07):
it'll be bumpy. And I justhope that we invest in it and making
sure that. That we do this ina sensible way.
Yeah. Awesome. Well, Rob,thanks so much for coming on the
show today. It was awesome toget to talk to you. Where can people
find you online?
LinkedIn's probably theeasiest place. I'm easy to find there,
so I'll just leave it at thatinstead of trying to rhyme off a
bunch. I also do have my ownpodcast called "The Confident Commit",
(49:28):
if anyone wants to check it out.
Ooh, yeah, put that in theshow notes.
Nice.
Awesome. Well, thanks so muchfor the time. Really appreciate it.
Yeah, thanks for having me.Cory, this has been awesome.
Popular Podcasts
Hey Jonas!
Hey Jonas! The official Jonas Brothers podcast. Hosted by Kevin, Joe, and Nick Jonas. It’s the Jonas Brothers you know... musicians, actors, and well, yes, brothers. Now, they’re sharing another side of themselves in the playful, intimate, and irreverent way only they can. Spend time with the Jonas Brothers here and stay a little bit longer for deep conversations like never before.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com