May 13, 2026 • 55 mins
What happens when a non-deterministic AI system is asked to touch production telemetry or generate changes for an SRE pipeline? The cost of being “close enough” can be lost data, downtime, or a security incident.
Cribl’s Nikhil Mungel joins Cory to break down what it takes to build AI that sysadmins can actually trust. The conversation digs into harness engineering and the practical guardrails that turn probabilistic models into repeatable, verifiable outcomes. They cover why breaking work into small chunks matters, how validation and testing become the real leverage point for AI-native development, and what “code factories” mean for review, CI, and platform reliability when teams can generate a thousand PRs an hour.
Platform engineers will also hear a pragmatic take on the future of the job. The focus shifts away from typing code and toward building systems for verification, simulation, and safe deployment at scale, plus clearer ways to decide what needs human scrutiny and what can ship automatically.
Guest: Nikhil Mungel, Head of AI R&D at Cribl
Nikhil Mungel is the Head of AI R&D at Cribl, where he's building LLM-powered systems for IT and Security data transformation and analysis. Before Cribl, he spent over a decade developing distributed systems across the observability and consumer social tech landscape. He lives in San Francisco with his wife and two kids. His current focus is applying AI to make complex infrastructure more intuitive and explainable.
Links to interesting things from this episode:
- Cribl Guard
- “Open source died in March. It just doesn't know it yet.” by Dan Lorenc, CEO of Chainguard
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:04):
Welcome back to the platformengineering podcast. I'm Cory O'Daniel.
This week I sat down withNikhil Mungel, head of AI R&D at
Cribl. We got into what itactually takes to build AI into a
product where your users aresysadmins who can't tolerate hallucinations,
harness engineering, codefactories that spit out a thousand
PRs an hour, and how platformengineering teams figure out which
code gets human reviews andwhat just ships. guest today has
(00:29):
spent over a decade buildingdistributed systems across observability
and consumer social, and nowheads up AI R&D at Cribl. For anyone
not familiar, Cribl is thedata pipeline platform for logs,
metrics and securitytelemetry. His team isn't just integrating
LLMs into the product, they'reactually training their own models
and building their own AIplatform underneath. Nikhil Mungel,
welcome to the show.
(00:50):
Thank you, Cory, appreciate you.
Yeah, I would love to know,like, you've been building distributed
systems for a decade, what hassteered you towards AI and R&D? Like,
what's going on there?
Yeah, so I've been, you know,my first brush with technology was
all the way back when I usedto do Windows 95 administration and
(01:10):
printer setup for, you know,small businesses and stuff like that.
It was a pretty fun gig. Thatwas my first. That's how I got into
technology. I remember, youknow, fixing point of sale stations.
These were like Windows XP, bythat time it was not Windows 95 anymore.
Trying to like, make surecoffee's not like dropping on me
or something. And usually theproblem used to be like a sticky
(01:32):
cable or something of thatsort, you know, in the back of the
computer.
Oh yeah.
Those were pretty interestingtimes, you know, with Windows. And
then when I first gotintroduced to Linux, that's when
like a light bulb went off inmy mind. And that's when I really,
you know, got into technology.And then one thing led to another
and I was, you know, soonfound myself working as a professional
(01:56):
software engineer. Happy to gointo a little bit more about how
I came up here. And then Ispent a lot of time at Splunk, which
was a big data observability company.
Oh yeah.
And spent a bunch of timeoptimizing. So most of these tools
(02:16):
have a pretty complex languagethat you have to type in order to
get back your results and getback your search results. It's a
structured query language, youknow, kind of like SQL or something
else. And it's kind of a painbecause Customers and users have
to learn this new language inorder to retrieve data, which seemed
like the perfect opportunityto, you know, use AI and you know,
(02:38):
use some form of machinelearning to recommend auto completions.
And that was... gosh, that waslike 10 years ago now. So that was
my first brush with AI. Andthen I've just been fascinated by
neural networks and you know,how you can use pattern matching
instead of just rules basedapproaches to make technology easier.
(03:00):
And then, I mean the biggestunlock was ChatGPT, obviously, you
know, when it came up. OrGPT... I was using one of the completion
models back in twentytwenty-two, twenty-twenty-one. And
that was just... tha feltlike magic. I mean it still does,
you know, every time you useany tool like ChatGPT or Claude,
it's just absolutelyfantastic. I can't believe computers
(03:22):
are really doing this.
ChatGPTdidyoufirst have thatlike that realization that this stuff
could potentially work? Like,were you very early? Like you saw
chatGPT like in like the earlytwenty twenties, you were like, "Holy
moly." Or like, were you alate... like I'm a late comer. Like
it was maybe like November oflast year before I was like, "Oh,
this actually may have legs to it."
instruction-queuedfirstahamomentwas,I'mgoingtosaybackin twenty
(03:47):
twenty-one where it was notchatGPT, it was just GPT, you know,
two or two point... maybe itwas like... it may have been three.
It was not even. It was,gosh... I don't remember the name
of the model, but it was acompletion model. It was not an instructionqueuedmodel,whichmeansyoucouldn'tchatwithitassuch.Yougaveit,youknow,somecontentandthenitcompleted
(04:10):
thatcontent foryou.So it's justacompletionmodel.
Oh, old school.
Yeah, old school. And then youcould give it a few examples. They
could be like, you know, rosesare red and you know, violets are
blue and then it wouldcontinue that. Right? And if you
gave it something in the styleof Shakespeare, then it would continue,
you know, just rambling off ofthat. And that was like the biggest
(04:31):
unlock, right? Because then ifyou gave it a couple of examples
of structured queries, then itwould help you formulate the next
one. And its ability to justgeneralize information from the provided
context was fascinating. Andobviously OpenAI and Anthropic have
done tremendous amounts ofwork to make that instruction tuned
(04:55):
and make it so that you canactually have a conversation with
it now, which has just been awesome.
I agree, I've actually... meand my coworkers, we've kind of leaned
into AI native developmentover the past few months. And I feel
like the hardest part, youknow, is doing the stuff that's always
been good engineering. Likwriting the code. Like we spend so
(05:17):
much time writing the code,but like so much of our job is the
stuff around it, right? Likethe guardrails around like linting,
tests that run quickly and aretrue, formatting, you know, security
scanners, like all the CI/CDstuff that like people typically
aren't super excited about.And I feel like a lot of folks when
they have bad experiences withAI code generation, it's because
(05:41):
they haven't done that workaround their current code base. They
may have code that is hard tofollow and when it's reproducing
these patterns, it'sreproducing patterns that are hard
to follow. Right? When youhave code that is very well designed
and organized, it isreproducing very good patterns. With
your core audience at Cribl...I assume it's mostly sysadmin folk,
(06:04):
right?
Yeah, a lot of practitionerssysadmins, IT admins, SRE, DevOps,
SecOps.
Yeah. So for folks wheredeterminism and high accuracy are
the name of the game, LLMs areinherently non deterministic. How
do you build an AI nativeproduct for them when they can't
(06:26):
tolerate the kind of errorsthat we might get in something that
comes back from a ChatGPT?Where it makes up a Python library
that doesn't exist.
Yeah, that's such a good,that's such a good point. The trade
offs are pretty asymmetricwhen it comes to systems administration,
SRE Ops, and any system that'sreally in charge of controlling like
(06:48):
the operationalization ofsoftware. So you know, software goes
through multiple stages. Youknow, you probably know this really
well where you're... at somepoint it's just a little nascent
thing on your computer whereit's pretty contained and you have
a pretty high tolerance for,you know, hallucinations or anything
that's not quite working well.And then as it goes through the stages
(07:11):
and you know, as it goesthrough CI/CD and then gets merged
and then it gets deployed whenit's actually the same software that's
now running on productionhardware where it's like servicing,
you know, customers, in ourcase it's maybe processing hundreds
of terabytes of data everysingle day. Right. So the stakes
are really high and you don'treally want anything messing that
(07:34):
up, right? So AI can make youa little bit more productive, but
then if it gets somethingwrong, there's going to be a pretty
steep downside. Right. So mostof our use cases and most of the
users that use or in thesesort of roles tend to be optimizing
(07:54):
for, "Hey, let's not dropdata, let's not lose production data,
let's not lose uptime,"because there's nothing more important
in that role. They're notnecessarily optimizing for, "How
productive can I be?" It waslike, well, if I drop transaction
data, that's going to bringthe business down. And it's a very
different modality of applyingAI to those sort of problems. And
(08:19):
we have, over the last threeor four years, we have perfected
a set of techniques that haveenabled us to do that thing exactly
right. I mean, there were somany things. One thing is you don't,
you try to minimize the stuffthat you're using AI for and surround
it with a lot of harness. Sothings that, you know, deterministically
check the output. We also justbreak it up into really small chunks
(08:43):
as opposed to one shotting anentire file or you know, like an
entire pipeline. Right. Youjust break it up into small functions
and you know, let's do it oneby one. You take that to the LLM,
you see what you get back fromthe LLM. You validate that you, you
run it and then you insert itinto the stack. Just giving an example,
but really just breakingthings up into small pieces. Lots
(09:03):
of guardrails, lots of harnessthat's written to validate the output,
to make sure that it'sperformant and it doesn't have any
security vulnerabilities, allthose sort of things, and then stitching
it all together in adeterministic way. And that has actually
been a pretty cool unlock inthat we have been now able to use
(09:24):
AI and agentic user interfaceparadigms to produce code or to produce
configuration objects that arevery close to being deterministic.
So if you wanted to convert Ato B, you could write it by hand
or you could use one of theseagentic experiences that we have
within the product. And itwill do a very good job of it, not
(09:47):
just once, but repeatedly.Like if you do it, if you, if you,
if you try to do it multipletimes, you're going to get the same
output.
Yeah, it's funny because like,what you're saying there is like,
it's a lot... it's guard railsagain. Even in a tool like this,
it's a lot of engineeringaround the thing, right? And I feel
like lot of folks, and Iknow... I'll call it the Anthropic
(10:09):
guy right here... but like alot of folks are calling out like,
"Oh, white collar jobs aretoast." Right? And it's like, I don't
know man, like, I mean I feellike the job has changed quite a
bit, but like, I still needmore hands. Like, I need more hands
in this like world of likegenerating AI code. Like, because
we still have all this otherstuff to do around it, right? We're
(10:30):
not necessarily laboring overevery single line of code, but we're
thinking about how to dothings like this. How do we generate
something and stack a bunch oftools together that it can use? I'm
still writing that software,I'm still building the guardrails
around it. Is that how yourteam is experiencing it? Are you
still seeing we do need asmany engineers, they're just doing
different things and they'remore efficient at it, or are you
(10:53):
seeing I don't need as manyhands to get this done because we've
got AI here?
Yeah, I think it's a mix ofboth. We have certainly felt... I
mean, I'm actively hiringright now, I'll even make a little
plug for it right now. Ifyou're interested, please reach out.
We are always hiring topquality engineers and candidates.
But the job or the role or theset of tasks, if I'm talking about
(11:18):
that a software engineer wasresponsible for, is evolving. A typical
software engineer would spenda certain fraction of their time
writing code and a lot moretime thinking about the problem,
thinking about how thatproblem fits in or how their solution
fits in to other systems thatalready exist within an enterprise.
(11:40):
What are the trade offs andall those sort of things. I feel
like that's still around. It'sthe code authoring part that has
been accelerated with theexisting set of agentic software
that we have like Claude Codeand Cursor and everything else. And
a new task family has expandedon an engineer's set of things to
(12:01):
do, which is verification andvalidation. Right now you have AI
generating so much code andthere's so many things that are,
you know, engineered. Everyoneis like suddenly 10x more productive.
So you have a lot more codebeing generated. Well, who's going
to validate that? Who's goingto verify that, that it's going to
work, you know, just the wayyou intend for it to work in a hardcore
(12:25):
production setting, you know,where there are millions or billions
at stake. The way the approachthat we have taken is we have started
to develop tooling that helpshuman software engineers to do that
a little bit better, do that alittle bit faster, right? To make
that verification, validationa little bit easier.
(12:49):
That's like most of the jobnow, right? I was talking to one
of my friends the other dayand it's like... yeah, I feel like
our job for the longest timewas, you know, we get a task assigned
to us by a product manager,from a Scrum Master, whoever... and
we sit and we think long andhard about how all these little pieces
plug into a library or astandard library or an API. And now
(13:12):
the job is like that stuff...which I don't know, personally, I
like coding, but I didn't...I've never really found the rewarding
part of coding, like figuringout all the things that I need. It's
like, it's the elegance oflike the solution and the experience
afterwards, right? And like,that's the whole job now, right?
Like that's... your stack ofcode is not the thing that's going
(13:34):
to be your moat anymore. It'sthis experience and joy of the user
afterwards of what you'vecreated. And I feel like a lot of
the job now is just likeverifying and making sure that this
is good quality functionality.It's something that I feel like we're
gonna have a problem with thisin the space in the near future,
because I feel like thatintuition comes from years of production
(13:55):
experience... like being ableto look at things and understand
how the code all goestogether. I feel like we're gonna
have a gap in juniors in thenear future of like, people that
don't know how to do thatanymore because they haven't written
production code. But like, Ifeel like a lot of the job now is
gonna be guardrailengineering, observability engineering,
quality engineering, less solines of code engineering. So I would
(14:18):
absolutely love to hear whatyou guys are doing there.
Yeah, I mean, I would... Ilike how you broke it up into observability
engineering and guardrailengineering. I've just been calling
it harness engineering for thelast few months now, where it's just
really building the systemsthat enable teams to use agentic
decoding to ship software. Imean, when I think about it from
(14:40):
a platform engineeringstandpoint, since you know, a lot
of our listeners are rightnow, you know, platform engineers
in many ways. The job ismoving into developing these systems
that then allow your productteams to quickly ship software. So
how do you abstract awaycreating new guardrails or creating
(15:01):
new validation checks? Whatare the new primitives that you build
in this new world thatdevelopment teams inside your organization
or product teams inside ourorganization can take advantage of?
Because libraries... For along time, there was just libraries
(15:22):
and there was APIs, andarguably that stuff is being generated,
at least in some settings,these days. But there's still a lot
of things that are not beinggenerated, which is, "How do you
plug this in?" Like, I'll giveyou an example. I'm sure you've tried
your hand at vibe coding.
Oh, yeah.
Which is, you know, all therage back in twenty twenty-five.
And now it... I feel like it'sstill going pretty strong.
(15:44):
Yeah, I feel like it is. I'm aperson... I don't like typing, I
get really bad tendinitis. Ihave leaned fully into, like, Whisper
tied into Claude Code, and Iwill either, like, talk my way through
a spec and then just copypaste it in or I will just kind of
talk through and just rambleabout, like, what I want to build
(16:05):
and then watch it materialize,which is absolutely magical.
100% right.
Yeah.
Next.jsa pretty good workflow,you know, just using speech. I should
give that a shot. But if youthink about the early days of vibe
coding, when it was... notthat long ago, it feels like long
ago or like some time haspassed, but maybe it was like a year
(16:29):
ago or something... but whenit was first crystallizing, it would
generate everything. Right? Imean, it wouldn't use something like
React, it wouldn't usesomething like even Express. It would
just write raw JavaScript orit would just write raw languag
without using any libraries,and it would still work. You know,
like, it would still look likea little bespoke application. And
(16:52):
then, like, if you try anyvibe coding platform... you know,
there's so many of theseplatforms these days... or even if
you just use Claude, what itdoes now is it just uses React, it
uses Tailwind, it usesNext.js, right? it'salmostinjectingallof
all of these building blocksinto the vibe coding thing and creating
an app that kind of lookspretty robust and itdoesn'thaveasmany
(17:16):
failuremodes asanapp thatwasperhapswipecodedayear an app that
was perhaps vibe coded a yearago. And those building blocks platformengineering,I'vebeenencouragingmyteamstostartproducingthosebuildingblocks.Likewhatareuniquebuildingblocks
thatavibecodingharnesspullfromandinsertinto applicationsasyourproductteamsarenow
(17:43):
buildingtheseapplications.Right.So maybeit'sabespokecomponentlibrary.Maybeit'sasetofskills,youknow,that,
thathavejust.It'slikeit'samarkdownfile.Right.Alotof.A lotofengineeringnow
hasjustcomedowntobeingableto.Clearly able just... It's like
it's inEnglish, which Aamarkdownfile.Andwhatarethoseskillsthatyoucanproduceasa to
(18:04):
clearly articulatethatcanbeusedbywhite gradingharnesses
and pullinthosedependenciesasbuildingblocks?I'dbecurioustoget yourtakeonthat.v
ibecodin g
Yeah, honestly, one of thethings that's changed, I'd say, I
hate to admit this, since I'vebeen a developer for such a long
time, but one thing that'sreally changed about the way that
(18:25):
I develop, I've always been afan of test driven development. I've
been a TDD engineer for thelongest time. And for a good portion
of like my vibe coding era, Iwould still write all of my tests.
And then I still do this tosome extent today. And I would use
my tests as my prompts. It'slike, "Hey, this is what I want to
exist. Make this so." Right?And it's very good when you do that.
(18:48):
Now the thing that I'vestarted running into is it takes
a lot of time to write tests.It's always taken a lot of time to
write tests. And like, nowthat I have this speed in the development
section side, I'm like, "Iwant that speed on my prompting side
too." Like, I'm spending a lotof time prompting code gets generated.
I'm like, "How do I optimizethis side now that side's maxed out?"
(19:09):
And so now I've started tojust talk through my tests, which
is weird, but I still do TDDwith my LLMs. Like, I will talk through
the ideas of the tests, I willwatch it, generate it. They all fail.
That's great. That's what Iwant, right? And then we iterate
through it, right? And I'llusually open my first PR with just
like failing tests. So it'slike I can see the code. It happened
(19:30):
in a work tree, right? I cansee what the PR is going to change
to the test suite.
Right.
I can acknowledge that is whatI intended. I'll put a comment or
something if I wanted toswitch. But the thing that's fundamentally
changed about my developmentis I'm very good at documenting things
now, which I have just neverbeen good at. And I think a lot of
engineers, like, they don'tlike... engineers like to write code.
(19:53):
You still see so manyengineers that don't like writing
tests, like, people are like,allergic to it. It's like, "Oh, I
hate writing tests." That tome has always been like the joyous
part. It's like, "Oh, I'veexpressed my intent." Yeah, it's
like I've expressed my intentand it works. This is exactly what
I'm looking for. But I'venever been great at docs. It's like,
"Hey, it works. Get thisfucker out the door." An now I
(20:15):
spend so much time onarchitecture decision records. Why
is this the way it is? Whatimpact from customers or our production
infrastructure has caused usto think about solving this problem
this way? And then it's likelocked down in code. Right? And that's
one of those things. It'slike, I feel like many teams that
(20:35):
really leaned into things likeADRs and deep detailed specs on how
your V2 API works, theconstraints and things you put into
it. Those are things thatreally established teams with plenty
of time to do so do. And thenother teams don't. Right? Now, that
is again, part of my job, andwhat's cool about this is it helps
(20:58):
the LLMs quite a bit. And thehumans, turns out they love documentation
that's accurate too. Right?And it's just like, tha has always
been like such a hard part todo, especially in startups where
you're just constantly runningand running and running, like moving
on to the next MVP that acustomer wants. It's so hard to go
back and document the lastthing. But, like, again, like, that's
(21:19):
the job now. It's this, like,quality engineering is kind of what
I see? It's just like the codeitself is just... it's... I don't
care, like, if it rewrote itin Python tomorrow, I do not care
so long as my qualityengineering on the outside can tell
you this is how the API worksand I can guarantee it works. Doesn't
matter to me what's inside thebox. I love to get your take on
(21:43):
this because I've heard a fewpeople start to say this... I think
it was the CEO of Chainguardrecently wrote a blog post about
like open source is dead, itjust doesn't know it yet... and I
feel like I don't necessarilybelieve that, but I've got this gut
feeling that it might be true.And so it's like for an example,
(22:03):
I would never say, like, "Hey,let's spin up a new React." It's
good, there's a really goodecosystem, but at the same time there's
a lot of vulnerabilities thatare getting snuck in through like
a dependency of a dependencyof a dependency. While at the same
time, like in our ownproducts, like we get to things where
we're like, "Hey, we're addingskim integration" and it's just like,
that's important to us, wewill actually take the standard and
(22:25):
we will generate our ownlibrary that we can just own. It's
so critical to... it'sexternal system managing permissions.
I want to own all of thatcode. It's not something I ever would
have wanted to own in a worldtwo or three years ago because...
I've got to focus on myproduct. But now it's like, this
thing is really important tous and microsecurity, so I want to
(22:45):
understand how it works and Iwant to make sure that we have full
control over its stack ofdependencies. So yeah, so I don't
know, man, it just feels likea lot of it is good engineering principles
that we've always had thatwe've never had the time for are
the work now. And that'ssomething that will benefit the LLMs
and it's definitely going tobenefit the person that replaces
(23:07):
me, right? Or the personthat's joining my team that I don't
have time to go througheverything with. It's like, "Hey,
here's an entire markdowndirectory of decision records of
how we came to thearchitecture that we have and the
tools that we have, and whichlibraries we do and don't use, and
how we decide on when or whennot to use a library versus just
generating and owningsomething ourselves."
(23:30):
Ops teams, you're probablyused to doing all the heavy lifting
when it comes toinfrastructure as code wrangling
root modules, CI/CD scriptsand Terraform, just to keep things
moving along. What if yourdevelopers could just diagram what
they want and you still gotall the control and visibility you
need? That'sexactly whatMassdriver does. Ops teams upload
your trusted infrastructure ascode modules to our registry.Your
developers, they don't have totouch Terraform, build root modules,
(23:57):
or even copy a single line ofCI/CD scripts. They just diagram
their cloud infrastructure.Massdriver pulls the modules and
deploys exactly what's ontheir canvas. The result? It'sstill
managed as code, but withcomplete audit trails, rollbacks,
preview environments and costcontrols. You'll see exactly who's
using what, where and whatresources they're producing, all
without the chaos. Stop doingtwice the work. Startmaking Infrastructure
as Code simpler withMassdriver. Learn more at Massdriver.cloud.
(24:28):
Like, how are you guys seeingthat? Like you guys have a huge product,
you got a big team. Like youguys are... you have customers that
are very concerned aboutsecurity. You guys are very concerned
about security. Like how areyou seeing open source in your own
organization? Like is it stillsomething you just, "Oh, that library,
Python library, it's popular,I'm just going to grab it." Or do
you think through like thesupply chain of those libraries?
(24:50):
thingthat's such a goodpoint. Right? I mean, let me first
talk about TDD and you knowhow much I love TDD. My first job
was ThoughtWorks and you knowwe were like... ThoughtWorks was
a big proponent of test drivendevelopment. And you know, it was
just like the greats likeMartin Fowler and you know, like
a whole... some industrystalwarts, I mean who were part of
that movement. So that'sreally how I cut my teeth. And it's
(25:15):
th same thing that we seetoday, right? It's the spec first,
right? It's being reallythoughtful about what is it that
you want to build and beingable to express th constraints of
the product in a tightdeterministic way. I mean that's
test, right? And you write thetest, first it fails and then you
write the code that fulfillsthat and just that and nothing more.
(25:36):
And now that test is, youknow, it's either a spoken record
in your case or you know, it'sa detailed spec markdown file that
you know, describes exactlyhow the software is supposed to be...
exactly how the system issupposed to behave. Which can be
used for two things, right?One is for using it to generate the
(25:58):
code that implements thatbehavior and then more importantly
the code that validates thatbehavior. Right? And you can also
think about splitting yourtests in the same way that you split
training data sets when you'retraining models, which is you know,
you have 80% and then youwithhold 20% and use that for validation
(26:20):
to make sure that your spec,you know, your model is not overfitting
on the 80%. Right? So youcould also do... you could also split
tests up in that way where youcan have, you know, certain part
of the spec which is hiddenfrom the LLM when it's generating
the code. And then it's onlyused for validation to make sure
that it's validating the spec.Right? So there's a pretty interesting
way to structure a smallerspec which is not handed to the implementation
(26:44):
aspect of it, which is apretty cool pack that we've used.
When it comes to supply chainattacks that we have seen, you know,
in the last few months. Imean, you know, they've been rising
at a pretty exponential rate.And so, yeah, I do get that point.
You know, open source isunfortunately in a tricky, tricky
(27:05):
spot. And my take is that it'sgoing to evolve from open source
implementations into opensource standards. So there's going
to be a lot morestandardization where you pull the
standard from, you know, somesort of a GitHub or an open source
repository. And the standardagain has validations and tests and
(27:28):
all those sort of things. Andthen the implementation, the actual
production code that runs inthe hot path in your system is generated
by an AI that you have or yourteam of engineers. And then the open
source is literally just aspec and the set of tests that tell
(27:49):
you that your implementationis watertight and that way you can
defend your supply chain fromany, you know... because unfortunately
the reality is untrusted codeexecution is the biggest threat everywhere
right now, especially withvibe coding. Be that a supply chain
infiltration or you know, bethat a vibe coding surface in your
(28:13):
product where you're allowingcustomers to extend the product by,
you know, giving this like,"Oh, enter a prompt here, now generate
something." It's all untrustedcode. And I feel like it's... my
take is that the way to reignin that untrusted code is by tests
and it's by specs. Oneinteresting,youknow,Iwastalkingtosomebodywhoworksatalargedefense tech
(28:38):
companythatwe allknowofandoneofthe,oneofthethings thathe the
it,you know,heworksatFrontierAI it... you know,
he works at Frontier AI, intheir world... and they have a really
sophisticated real worldsimulation. So it's like a real world
simulation that's built onlike a game engine that, you know,
(29:01):
allows them to, you know,really test how things work in a
versatile setting withoutreally having to engineer anything.
I mean, like, that's a reallycool way to validate software, you
know, as well, like, which isnot... and again, in defense tech,
(29:21):
you're producing real worldthings. So your simulation is really
a physics engine. And now thelaws of physics are, you know, pretty
well known and, you know, wellwritten down for a long time now.
So it's almost easier to builda physics engine. But what does it
mean toIhadtolikebreakdownsomelawsofphysicsforintheobservabilityspace,youknow, whenyouhavenetworkflowlogsthataregoingfromoneplacetoanother,there'sa
(29:52):
if you're an observabilitycompany. Like, if I had Thesize ofthepacket,right?st
Yep.
You don't break that law. Noone ever breaks that law. Right.
Like no networking equipment,firewall, router is ever going to
break that law. They're alwaysgoing to log that. Right? It's never
going to be incorrect. Andthen figuring out those laws that
govern the environment foryour own domain and creating this
(30:18):
simulation realm where you canrun your software and validate it
to the point of a mathematicalproof almost. So like, "Oh, if it
works here, we know it's goingto work in production." That might
be a pretty interesting pathfor platform engineering teams to,
you know, think about and sortof pull the thread on a little bit.
(30:40):
Yeah. One of the things, likekind of going back to you were saying
earlier about like, just likethe different, like smaller tools
that you kind of put together.When I talk to my LLM to do my tests...
and for anybody who doesn'tlike TDD, you're going to recoil
when you hear what I'm aboutto say that I do, but I swear it's
the thing that helps us moveso quick... I speak to it in plain
(31:03):
English because I speak to itin like BDD, like Gherkin, like,
"Given this, win this." Andour tests... our test suite has skewed
that direction. And the thingthat's interesting about it is like
we now have these very like,factual statements, right? It's like,
"When there is a cube set ofdeployments," it's a very English
phrase that we use around theorg all the time, like it's a sentence
(31:25):
that even the sales guyunderstands what I'm saying. And
so the interesting thing isit's so easy to reason about now
because it's like it's a plainEnglish sentence and that ties to
a tiny function that does thatexact thing, right? And so now when
I'm talking to my LLM, it'slike a lot of the tests, it doesn't
even have to write because Ialready have stuff tied to each of
the given wins thens, right?So I get to speak in plain English.
(31:48):
And even our sales guy canwrite like, "Hey, you know, I was
talking to the customer andthey said they need this." It's like,
"Oh, yeah, well, you can talkto this thing in plain English and
it will start buildingfeatures for you." And the crazy
part is like seventy, eighty,ninety percent of the test suite
at any point in time isalready written because we have all
these common phrases. So it'scoming to the point where it's like,
"Hey, let's talk about doingsome flow log stuff." It's like,
(32:11):
oh, we just have to add a rulein plain English and a little function
and it'll write that functionfor us, and it's so easy to reason
about. And I see these peoplewere it's like, "Oh, I've got a 10,000
line PR." It's like, I've got10,000 three line PRs. And it's very
easy to reason about wherealmost anybody in the org can look
at it and go, "Oh, we've addeda new rule that does this." I can
see th code that it producesfor the test. And like, now there's
(32:33):
some logic that's going tofulfill that, right? I don't know,
like, that feels like a reallygood flow. Again, I know that people
that don't like TDD are like,"Ugh, cucumber. Like, I don't want
Cucumber and Gherkin back inhere." But it's so easy to develop
this way, I cannot imaginegoing back. And like, folks might
(32:55):
be hearing like, "Oh, you'renot developing anymore though." It's
like, "No, I'm doing morearchitecture and thought about software
than I ever have."
Right?
vibe the last part of the TDD,right? It's red, green, refactor.
Most people don't get to therefactor part. Like, how do you get
your PM to give you time torefactor your code? It's like, it's
part of the job. Now it's thejob, right? It's like, I see stuff
come in and I'm like, "Oh,this all looks like shit." Like,
(33:16):
now I get to think about like,what does a grander, better architecture
look like for customers on theAPI or even internal developers using
stuff. Like, how do I buildsomething that works well for them?
And it's like I get to focus alot more time getting rid of debt
than I ever have in anyproduct I've ever worked on. It's
really, really interesting.So, coming back to platform engineers,
(33:38):
I'd love to get your thoughtson this because I think the most
underappreciated part ofplatform engineering - it's often
invisible, right? Even a goodDevOps team, good Sysadmins, like
when it's working, nobodynotices you, right? Nobody's like
running around like highfiving this admin and they're like,
(34:00):
"That auto scaler really doesits job well." It's like when it
doesn't, that's when peoplerecognize you. Right? And so like
in this new era of, you know,a lot of code being generated by
agents and vibe coded by endusers, like, how does that change
what we're responsible for asplatform engineers? And stability?
(34:21):
Like, we can have a very...and I know it's kind of two phases
- is the platform engineeringteam vibe coding stuff and then is
the engineers on top of theplatform vive coding stuff - but
I would love to know how yousee the world through the lens of
a platform engineer, wherewe're just getting things that may
be a lot less stable at first,but it's unstable and coming through
(34:42):
at a faster rate... which wecan now apply quality to and get
it stable.
Cucumberoff, Imean,with BDDand Cucumber, I mean, you're bringing
back some fun memories. Rubyon Rails. Man, that stuff is so well
rooted in that... it still isso well rooted in that ecosystem.
TypeScript has taken over theworld these days, but I do miss using
(35:05):
that. I remember the syntaxand those sort of things used to
be a little bit unwieldy withcucumber, but that stuff got figured
out pretty quickly. And now, Imean, it's just, you know, using
natural language to expressyourself and not even bothering with
those, you know, those, thosesyntactical nuances and quirks. And
really, I mean, that becomesyour spec... the guardrail in which
(35:28):
your software has to operate.And I feel when it comes to platform
engineers, you're 100% right.It's pretty asymmetrical. Lik when
the system is working well,when you're doing a fantastic job,
that means nobody's noticingthat the system is working well.
And with vibe coding and withAI, that's exactly the thing that
(35:50):
platform engineering has touphold, right? Like can it sustain
a ten x or even hundred x oreven a thousand x more software running
on the platform? So justscaling it up in a reliable way and
really supporting these newcoding paradigms that are emerging
(36:11):
now. Which is, you know...coding factories is like my favorite
one, right? I mean, where younow have... or development teams
are moving toward a worldwhere instead of writing code...
So, okay, I mean, so if youhad to like back up, right, like
before twenty twenty-one,developers were writing code character
by character, right? Onekeystroke at a time. And then came
(36:36):
AI based ID autocomplete,right? Where you start typing something
and the AI was like, "Oh cool,maybe these are the parameters in
the function that you wanted."And then came prompt driven agentic
decoding, right? Where toolslike Cursor and Claude Code became
popular, where instead ofstarting in the IDE, you now start
on the right hand panel andyou, you know, it's like, "Hey, this
(36:58):
is what I wanted to do." Andthen, you know, it proposes a diff
that you then accept orreject. And now I see more and more
teams going toward a worldwhere they're codifying their practices
and, you know, the thing theyproduce in a set of markdown files
and then creating a systemwhere you just insert the spec from
(37:21):
one end and it produces afully functional pull request from
the other side, right?
Yeah.
Taking into account like allthe quirks of your system and your
enterprise and your companyand all the coding standards and
everything else. And I feellike that entire harness again, that
(37:41):
factory harness, which iscalled the factory pattern, right?
I mean, it's different thanthe factory pattern that I used to
use when writing code way backwhen, but now it's just like, okay,
cool, like you have this, youhave this agentic harness that can
one shot an entire spec into afully functional PR. And that's going
to be the next thing thatplatform engineering has to tackle,
(38:02):
right? Like maintaining thesefactories and maintaining the assembly
lines. And not onlymaintaining, but innovating on that
and you know, making themfaster and you know, creating systems
where these pull requests whenthey get produced... because guess
what happens when you have afactory producing 1,000 pull requests
an hour, right? I mean, that'sa lot of code. Where is that going
(38:25):
to be deployed? How's thatgoing to be validated? So scaling
up not only the productiononline infrastructure, but also everything
around it is like, well, let'screate systems that can deploy PRs
and validate PRs at agentspeed, not at human speed. Right?
So like a thousand PRs an hourshould be deployed, should be validated.
(38:48):
You know, like some sort of asimulation environment. I would expect
that's the next frontier thatplatform engineering has to, you
know, conquer. I mean, there'salso. Sorry, go on.
No, no, I'm just like, thatis. I mean, you're just. Yeah, my
brain's going... It's funnybecause like so much of platform
(39:08):
engineering has always beenabout like the golden paths to production,
right? I can get you somecloud resources easily, or I can
get your Docker Container ontoa K8s cluster, or your functions
into a lambda. Like differentteams got different paths. But like
now it's like if every team...you got a front end team, you got
a backend team, you got acheckout team, you got a payments
team... if all of these teamsare kind of inventing their own little,
(39:31):
you know, guardrails andharnesses everywhere. Like, it's
a lot of rework, right? Ithink you're right. I think you'll
probably see a lot of thatcome into the platform. And that's
interesting because thinkabout... I mean, we're already to
the point on my team whereit's like, the number of PRs that
we produce today is stressful.I was talking to my dad recently.
(39:56):
He was an air trafficcontroller in Guantanamo. So like
a pretty intense job. Like wayback in the day, not like recently.
Sure. Yeah.
I was just like, "I feel liketoday developing is so much more
exhausting than it was a yearago." And I was like, "And it feels
weird because I don't writesoftware anymore." I was like, "I'm
just reading and processing."And he's like, "That's the scope,
(40:18):
dude." And I was like, "Whatdo you mean, the scope?" And the
scope is what they call likethe little radar thing that they
watch. He's like, "That's whatthe entire job of ATC is like, you
do not get to use the flowstate part of your brain whatsoever.
You're constantly analyzingand like using the executive function
side of your brain." And it'sliterally more exhausting because
you don't tend to use thatside of your brain a lot. You're
(40:41):
using the creative side ofyour brain, you're thinking through
things, right? You're havingthese deep flow states versus like,
"A plane is landing, a planeis landing. Two planes are near each
other." And that's what we'redoing with these PRs. It's like,
"These two PRs conflict withmain, like, what does that mean?"
I'm shifting contextconstantly and he's like, "The way
they solve this in ATC isevery two hours you have to take
(41:03):
a break." Because your brainlike after about two hours, like
it just... the executive sidejust plummets, right? And with all
of these changes... it's likethe chang rate is insane. A to
of PR is going through and youknow, to have people overlooking
all of those is hard. It'shard on people, right? And so like,
(41:24):
you know, as we think about,"How do we get value out of AI for
a business?" Like the AIcan't... like the business can't
take all that value becauseit's going to exhaust humans. Right?
Humans got to get some valueout of it too. Like do our hours
change? Do we work less? Do westart to see companies as benefits?
Saying, "Hey, engineers workfor two hours a day and then they
have off until after lunch andthey work for two more hours." Right.
(41:46):
Like, I think the job's reallygoing to change a lot. But you know,
at the same time with allthose PRs coming through, it's like
starting to... again buildingblocks and guardrails. Like maybe
every single PR now that youhave more time, maybe every single
PR does go through k6 and doload testing on it. Maybe every single
one does land in a previewenvironment that has a full suite
(42:07):
of browser tests go across itthat you've never had the time to
do before because you werekind of slaving away over every single
save to a database. Right? Andthat quality work is the thing that
every PM wish you had time todo but you never did because they're
focusing on revenue, right?Like that's the job now. And so it
will be interesting to seemore of that start to land on platform
(42:30):
engineers' plates.
That's such a good comparisonwith ATC. I mean, like that's exactly
how I feel when you... it'sthe shift away from being able to
do deep generative work intolike more reactive work. When we
(42:51):
talk about, well, AI is goingto, you know, ten x or hundred x
a team's productivity,ultimately the team is accountable
for that ten x output, whichmeans it's going to... from first
principles... is going to comeat ten x more cognitive load on that
team. Because I don't see usas an industry getting to a point
where we are okay withshifting that accountability to AI,
(43:12):
right? The team is still...the human team is still going to
be accountable for whateverthe team produces, right? Whether
they use AI or not. So thereis a pretty deep cognitive burden
I feel when it comes toplatform engineering. I mean just
inventing tools and techniquesthat simplify that burden... that
(43:33):
reactive burden... andhelping, you know, that review process
easier. One of the ideas thatwe have talked about at Cribl i
not all code in your code baseis equal. You know, there is certain
code that's important and thatthere's certain code that's maybe
not as important. S likereally splitting it up or you know,
drawing rings around your codelike, "Oh, this is the ring 0, the
(43:57):
kernel." Like in Cribl's casethat would be the score stream processing
engine that can processhundreds of terabytes of data or
transform, you know, hundredsof terabytes of data in an hour.
Lik that's the kernel andthen the ring around that is maybe
like the control plane for thekernel, right? So what sort of stuff
(44:18):
are you putting in?Optimizations, those sort of things,
then kind of like expandingthat. And then at the very edge you
have smaller things like somesettings panel hidden in the UI,
UX somewhere. So likesplitting that between kernel land
and user land, kind of how theLinux system does it and then treating
(44:40):
different grades of PRs. So(a) like a PR should not span across
multiple rings, it should bein its own ring. And then (b) treating
those differently. Right? Imean like if it's PR that introduces
a bug, but then you can... ifthe bug is caught, like you know,
if you... it's easy to remedythat, you know, like you can push
(45:02):
out a fix in ten minute andthere would be no impact versus messing
something up in like the core.You'll probably have like more human
scrutiny.
Yeah, I'm going to give youanother Thoughtworks throwback that
I feel like you're sayinghere, but you're not using the words
- it's domain driven design.Like you're talking about like the
core domain, right? Like thatwas always the thing. It's like it's
so hard. And so many codebases where it's just like not a
(45:24):
monolithic repo, but like amonolithic code base where it's just
like... you know, in Rails youopen the app folder and there's just
seven thousand model inthere, right? But like starting to
organize. And again it's... thi is quality engineering. This
is the stuff we all wished wehad time to do in two thousand fifteen
and never did. And now it'slike thinking about the different
parts of your business thatare critical. This stuff is going
to get... you know, that coredomain, that string transformer...
(45:48):
like that's always going toget humanized. I don't know, maybe
the user settings page islike, "Eh, we don't give a shit.
Like let's just let it merge,like the testing, like let it go.
Like let's focus on makingsure that the secure parts of the
app are secure, the core hotpaths are good. And the domains that
(46:08):
are not the core domain, maybethey don't need as much attention
in guardrails, and we focus onthe part that does." And it's easier
to reason about stuff in thatworld too, right? Because it's like
everything's scoped down tothe little circle that you're working
in. Whoah, baby.
Yeah, yeah, yeah. It'schanging fast. The world is changing
(46:28):
fast. Software engineering,platform engineering, our entire
industry is at the tip of thespear when it comes to AI transformation.
Since code is such a... I meancoding is like the first breakout
case of generative AI, right?I mean like it's the first breakout
success. I don't thinkanything is quite that breakout even.
(46:49):
I mean the next best thing isprobably self driving cars. Those
are in like a handful ofcities, right? Whereas agentic coding
is all around. Like everysoftware engineer, every... like
no matter where you're in theworld is using some form of agentic
coding. At this point we getto decide, you know, or we get to
be the guinea pigs of, youknow, how this is going to play out.
Hey, and we don't... you don'thave to be the guinea pig. This is
(47:12):
the thing that's exciting tome about this that I was like, I
feel like all these like AICEOs, like I feel like they are fucking
terrible. They're great atraising money, but they're terrible
at marketing. Like coming outand being like, "Fifty percent of
you are going to lose yourjobs in the next three years." It's
like, "Hey, you know whypeople throw Molotov cocktails at
you? Like, it's because youaren't making people happy with your
(47:33):
words." B it's... at the same time, it's like we've had a
few of these... like, I don'twant to say revolutions... but these,
like, big changes in, like,software engineering over the past,
you know, twenty, thirtyyears. And we're in the next one
right now, right? And I feellike this is the time where it's
like, you can get a lot offree labor from these subsidized
tokens to make your code basesreally good. And then you can think
(47:55):
a lot more about what does thefuture of engineering look like for
our industry and what does itlook like for our orgs, right? And
that to me is exciting. It'like, it's still engineering work.
It's headier stuff. I thinkit's more interesting problems to
solve than save user todatabase. Like, you've done that
at your last seven jobs indifferent languages. We're at an
(48:17):
exciting time. And I thinkmost people that are afraid of it
are afraid of it because ofthe words that these CEOs are saying,
right? And it's like, I thinkthey're saying that because they
want to raise money again. Ineed more people because I'm making
more code, right? It's like, Ineed more people, right? So I don't
(48:39):
know. It's wild times. We canbe guinea pigs, but I think a lot
of us can also be computerscientists now. Don't be the guinea
pig. Be that, right? Be theguinea pig scientist. I don't know.
Like, let's figure out howthis new world looks because there's
a lot of stuff to figure out.I think CI is going to look a lot
different in the next fewyears. People are starting to talk
about... like, I know a lot ofpeople are pointing to GitHub and
(49:00):
be like, "GitHub's fallingover." It's like, "Dude, GitHub is
getting way more commits andrepos than they've ever had in their
lives. Like, they have a lotof just like, stability to deal with
from just like the onslaughtof data being thrown their way."
Like, you can, you can blameit on core AI all you want, but,
like, they're handling morewrites than most of us, right? So
it's like the world, ourengineering stuff that we have kind
(49:22):
of clung onto since the firstDevOps talk from Flickr, like CI/C
and Git and all this stuff...It's like, this stuff is likely going
to change and it's us who getsto make the decision as to what that
looks like. It's us that hasto get familiar with how these systems
work to be able to make thosedecisions. And I feel like if you're
(49:44):
sitting around like afraid ofLLMs and AI right now, like those
decisions are going to be madefor you, which sucks.
A hundred percent. Right. Imean, everyone has got to be thinking
about how do they become anowner of whatever their domain is.
Right? I mean just startacting like an owner of the problem
that you have, of the problemthat you're trying to solve, and
then, you know, use every AIagentic software tool out there to
(50:08):
help you conquer that and justreally deliver that ROI. Really.
So like if you're anengineer... if you're a front end
engineer, you know that looksdifferent for you, then if you're
a platform engineer, thatlooks different for you. But ultimately
it's just showing relentlessownership of the problem that you're
trying to solve - What areusers looking for? Where your customers
(50:28):
are paying you money for andthen going and really delivering
on that front. Which is a farbetter place to be in than being
a guinea pig.
Yeah. Awesome. Well, I reallyappreciate you coming on the show
today. It was super fun. Wherecan people follow your work and learn
more about what you and yourteam at Cribl are building?
Yeah. So you know, if youfollow me on LinkedIn. Follow me
(50:51):
on Twitter. We're prettyactive on LinkedIn where we post,
you know, all the cool stuffthat we are doing at Cribl. Just
this week we released a bunchof benchmarks on LinkedIn where we
compared privacy sensitivedata scanning models that we build
in-house, that we trainedin-house, and how they stack up with
OpenAI's new privacy filter.So OpenAI released a 1.5-billion
(51:13):
parameter open source model onHugging Face. It's awesome, definitely
go check it out. It can scanand redact sensitive information
from any data that you throwat it. And at Cribl, we have our
own in-house built models thatare way faster than OpenAI's models.
Much more accurate as well. Soour F1 scores are... which is a score
(51:37):
that really tells you aboutprecision and recall when it comes
to AI systems, really howaccurate a model is... we're able
to benchmark our models atmore than ninety percent wherea
OpenAI's model was aboutfifty-seven something percent. And
then our model is also ahundred and thirty-three x faste
than OpenAI's model becausewhat we have been able to do is we
have been able to compressmuch higher accuracy in a really
(52:01):
small model. So you're talkingabout a hundred millio parameter
model that can run on a lot oflogs, a lot of telemetry data, observability
data. Because guess what? Withagentic AI, LLMs are everywhere.
And people are sticking allkinds of things into LLMs these days,
right? Personally addedinformation, health records, Social
(52:21):
Security numbers, phonenumbers. And one of the biggest challenge
that IT teams and securityteams have to conquer is how do you
scrub all of that informationfrom access logs, from audit logs...
a really pressing problemacross the industry. So I'm really
happy that we are leading thecharge on that and developing these
models that enable you toscrub sensitive information so it
(52:42):
doesn't land in places that itshouldn't. You don't get one of those
weird letters in your mailsaying that, "Oh, there's been a
data breach." Or, "You'resigning up for one year free of some
monitoring..."
You know what happens when youhave six Equifax monitorings that
all overlap from all yourbreaches? Nothing. Your data's still
out there.
It's all out there. Right, exactly.
(53:03):
That is funny. We actually...we just added a hook to our stack
that essentially like... Idon't want to say... kin of like
a loud list, like what wecould put into OTel. We caught that...
you know, we lean into OTELpretty heavily. Folks that have listened
to the show before know that Iused to do a lot of what I call the
OTel driven development. Idon't do as much of it anymore. But
(53:25):
like we lean into OTEL prettyheavily. Like, we don't... we actually
don't emit a single log fromour server, we just... everything
goes into OTel. And I'll tellyou what man, LLMs love to put the
entire record into OTelattributes if it can. And so like
we have rules on like thetypes of data that can go in. Like
even stuff that's not... wedon't necessarily deal with PII,
(53:45):
but we deal with other thingsthat... you know, password secrets,
credentials for databases,like all sorts of stuff like that.
And yeah, so it's just like,"Dude, that is important." Like,
yo have a big team... ahundred, two hundred, three hundred
engineers... you are going tostart leaking stuff into logs that
probably should not be therefor sure. That is awesome. Is that
(54:07):
mode a part of Cribl or isthat open sourced as well?
It's right now part of Cribl.It's Cribl Guard, which is a product
we released a few months agoand that's how you access that model.
And we're going to release apaper on arXiv in the next week or
so that will enable you toreproduce that sort of a model on
your site as well. So thatpart will be published.
(54:30):
Awesome. We'll put a link inthe show notes to Cribl Guard and
then if... yeah, when thatpaper's published we'd love to put
that in the show notes as wellfor folks.
Awesome. Yeah, we'd love that.
Yeah. Awesome. Well, thanks somuch for coming on the show today.
Everybody listening, I'll seeyou next time.
Welcome back to the platformengineering podcast. I'm Cory O'Daniel.
This week I sat down withNikhil Mungel, head of AI R&D at
Cribl. We got into what itactually takes to build AI into a
product where your users aresysadmins who can't tolerate hallucinations,
harness engineering, codefactories that spit out a thousand
PRs an hour, and how platformengineering teams figure out which
code gets human reviews andwhat just ships. guest today has
(00:29):
spent over a decade buildingdistributed systems across observability
and consumer social, and nowheads up AI R&D at Cribl. For anyone
not familiar, Cribl is thedata pipeline platform for logs,
metrics and securitytelemetry. His team isn't just integrating
LLMs into the product, they'reactually training their own models
and building their own AIplatform underneath. Nikhil Mungel,
welcome to the show.
(00:50):
Thank you, Cory, appreciate you.
Yeah, I would love to know,like, you've been building distributed
systems for a decade, what hassteered you towards AI and R&D? Like,
what's going on there?
Yeah, so I've been, you know,my first brush with technology was
all the way back when I usedto do Windows 95 administration and
(01:10):
printer setup for, you know,small businesses and stuff like that.
It was a pretty fun gig. Thatwas my first. That's how I got into
technology. I remember, youknow, fixing point of sale stations.
These were like Windows XP, bythat time it was not Windows 95 anymore.
Trying to like, make surecoffee's not like dropping on me
or something. And usually theproblem used to be like a sticky
(01:32):
cable or something of thatsort, you know, in the back of the
computer.
Oh yeah.
Those were pretty interestingtimes, you know, with Windows. And
then when I first gotintroduced to Linux, that's when
like a light bulb went off inmy mind. And that's when I really,
you know, got into technology.And then one thing led to another
and I was, you know, soonfound myself working as a professional
(01:56):
software engineer. Happy to gointo a little bit more about how
I came up here. And then Ispent a lot of time at Splunk, which
was a big data observability company.
Oh yeah.
And spent a bunch of timeoptimizing. So most of these tools
(02:16):
have a pretty complex languagethat you have to type in order to
get back your results and getback your search results. It's a
structured query language, youknow, kind of like SQL or something
else. And it's kind of a painbecause Customers and users have
to learn this new language inorder to retrieve data, which seemed
like the perfect opportunityto, you know, use AI and you know,
(02:38):
use some form of machinelearning to recommend auto completions.
And that was... gosh, that waslike 10 years ago now. So that was
my first brush with AI. Andthen I've just been fascinated by
neural networks and you know,how you can use pattern matching
instead of just rules basedapproaches to make technology easier.
(03:00):
And then, I mean the biggestunlock was ChatGPT, obviously, you
know, when it came up. OrGPT... I was using one of the completion
models back in twentytwenty-two, twenty-twenty-one. And
that was just... tha feltlike magic. I mean it still does,
you know, every time you useany tool like ChatGPT or Claude,
it's just absolutelyfantastic. I can't believe computers
(03:22):
are really doing this.
ChatGPTdidyoufirst have thatlike that realization that this stuff
could potentially work? Like,were you very early? Like you saw
chatGPT like in like the earlytwenty twenties, you were like, "Holy
moly." Or like, were you alate... like I'm a late comer. Like
it was maybe like November oflast year before I was like, "Oh,
this actually may have legs to it."
instruction-queuedfirstahamomentwas,I'mgoingtosaybackin twenty
(03:47):
twenty-one where it was notchatGPT, it was just GPT, you know,
two or two point... maybe itwas like... it may have been three.
It was not even. It was,gosh... I don't remember the name
of the model, but it was acompletion model. It was not an instructionqueuedmodel,whichmeansyoucouldn'tchatwithitassuch.Yougaveit,youknow,somecontentandthenitcompleted
(04:10):
thatcontent foryou.So it's justacompletionmodel.
Oh, old school.
Yeah, old school. And then youcould give it a few examples. They
could be like, you know, rosesare red and you know, violets are
blue and then it wouldcontinue that. Right? And if you
gave it something in the styleof Shakespeare, then it would continue,
you know, just rambling off ofthat. And that was like the biggest
(04:31):
unlock, right? Because then ifyou gave it a couple of examples
of structured queries, then itwould help you formulate the next
one. And its ability to justgeneralize information from the provided
context was fascinating. Andobviously OpenAI and Anthropic have
done tremendous amounts ofwork to make that instruction tuned
(04:55):
and make it so that you canactually have a conversation with
it now, which has just been awesome.
I agree, I've actually... meand my coworkers, we've kind of leaned
into AI native developmentover the past few months. And I feel
like the hardest part, youknow, is doing the stuff that's always
been good engineering. Likwriting the code. Like we spend so
(05:17):
much time writing the code,but like so much of our job is the
stuff around it, right? Likethe guardrails around like linting,
tests that run quickly and aretrue, formatting, you know, security
scanners, like all the CI/CDstuff that like people typically
aren't super excited about.And I feel like a lot of folks when
they have bad experiences withAI code generation, it's because
(05:41):
they haven't done that workaround their current code base. They
may have code that is hard tofollow and when it's reproducing
these patterns, it'sreproducing patterns that are hard
to follow. Right? When youhave code that is very well designed
and organized, it isreproducing very good patterns. With
your core audience at Cribl...I assume it's mostly sysadmin folk,
(06:04):
right?
Yeah, a lot of practitionerssysadmins, IT admins, SRE, DevOps,
SecOps.
Yeah. So for folks wheredeterminism and high accuracy are
the name of the game, LLMs areinherently non deterministic. How
do you build an AI nativeproduct for them when they can't
(06:26):
tolerate the kind of errorsthat we might get in something that
comes back from a ChatGPT?Where it makes up a Python library
that doesn't exist.
Yeah, that's such a good,that's such a good point. The trade
offs are pretty asymmetricwhen it comes to systems administration,
SRE Ops, and any system that'sreally in charge of controlling like
(06:48):
the operationalization ofsoftware. So you know, software goes
through multiple stages. Youknow, you probably know this really
well where you're... at somepoint it's just a little nascent
thing on your computer whereit's pretty contained and you have
a pretty high tolerance for,you know, hallucinations or anything
that's not quite working well.And then as it goes through the stages
(07:11):
and you know, as it goesthrough CI/CD and then gets merged
and then it gets deployed whenit's actually the same software that's
now running on productionhardware where it's like servicing,
you know, customers, in ourcase it's maybe processing hundreds
of terabytes of data everysingle day. Right. So the stakes
are really high and you don'treally want anything messing that
(07:34):
up, right? So AI can make youa little bit more productive, but
then if it gets somethingwrong, there's going to be a pretty
steep downside. Right. So mostof our use cases and most of the
users that use or in thesesort of roles tend to be optimizing
(07:54):
for, "Hey, let's not dropdata, let's not lose production data,
let's not lose uptime,"because there's nothing more important
in that role. They're notnecessarily optimizing for, "How
productive can I be?" It waslike, well, if I drop transaction
data, that's going to bringthe business down. And it's a very
different modality of applyingAI to those sort of problems. And
(08:19):
we have, over the last threeor four years, we have perfected
a set of techniques that haveenabled us to do that thing exactly
right. I mean, there were somany things. One thing is you don't,
you try to minimize the stuffthat you're using AI for and surround
it with a lot of harness. Sothings that, you know, deterministically
check the output. We also justbreak it up into really small chunks
(08:43):
as opposed to one shotting anentire file or you know, like an
entire pipeline. Right. Youjust break it up into small functions
and you know, let's do it oneby one. You take that to the LLM,
you see what you get back fromthe LLM. You validate that you, you
run it and then you insert itinto the stack. Just giving an example,
but really just breakingthings up into small pieces. Lots
(09:03):
of guardrails, lots of harnessthat's written to validate the output,
to make sure that it'sperformant and it doesn't have any
security vulnerabilities, allthose sort of things, and then stitching
it all together in adeterministic way. And that has actually
been a pretty cool unlock inthat we have been now able to use
(09:24):
AI and agentic user interfaceparadigms to produce code or to produce
configuration objects that arevery close to being deterministic.
So if you wanted to convert Ato B, you could write it by hand
or you could use one of theseagentic experiences that we have
within the product. And itwill do a very good job of it, not
(09:47):
just once, but repeatedly.Like if you do it, if you, if you,
if you try to do it multipletimes, you're going to get the same
output.
Yeah, it's funny because like,what you're saying there is like,
it's a lot... it's guard railsagain. Even in a tool like this,
it's a lot of engineeringaround the thing, right? And I feel
like lot of folks, and Iknow... I'll call it the Anthropic
(10:09):
guy right here... but like alot of folks are calling out like,
"Oh, white collar jobs aretoast." Right? And it's like, I don't
know man, like, I mean I feellike the job has changed quite a
bit, but like, I still needmore hands. Like, I need more hands
in this like world of likegenerating AI code. Like, because
we still have all this otherstuff to do around it, right? We're
(10:30):
not necessarily laboring overevery single line of code, but we're
thinking about how to dothings like this. How do we generate
something and stack a bunch oftools together that it can use? I'm
still writing that software,I'm still building the guardrails
around it. Is that how yourteam is experiencing it? Are you
still seeing we do need asmany engineers, they're just doing
different things and they'remore efficient at it, or are you
(10:53):
seeing I don't need as manyhands to get this done because we've
got AI here?
Yeah, I think it's a mix ofboth. We have certainly felt... I
mean, I'm actively hiringright now, I'll even make a little
plug for it right now. Ifyou're interested, please reach out.
We are always hiring topquality engineers and candidates.
But the job or the role or theset of tasks, if I'm talking about
(11:18):
that a software engineer wasresponsible for, is evolving. A typical
software engineer would spenda certain fraction of their time
writing code and a lot moretime thinking about the problem,
thinking about how thatproblem fits in or how their solution
fits in to other systems thatalready exist within an enterprise.
(11:40):
What are the trade offs andall those sort of things. I feel
like that's still around. It'sthe code authoring part that has
been accelerated with theexisting set of agentic software
that we have like Claude Codeand Cursor and everything else. And
a new task family has expandedon an engineer's set of things to
(12:01):
do, which is verification andvalidation. Right now you have AI
generating so much code andthere's so many things that are,
you know, engineered. Everyoneis like suddenly 10x more productive.
So you have a lot more codebeing generated. Well, who's going
to validate that? Who's goingto verify that, that it's going to
work, you know, just the wayyou intend for it to work in a hardcore
(12:25):
production setting, you know,where there are millions or billions
at stake. The way the approachthat we have taken is we have started
to develop tooling that helpshuman software engineers to do that
a little bit better, do that alittle bit faster, right? To make
that verification, validationa little bit easier.
(12:49):
That's like most of the jobnow, right? I was talking to one
of my friends the other dayand it's like... yeah, I feel like
our job for the longest timewas, you know, we get a task assigned
to us by a product manager,from a Scrum Master, whoever... and
we sit and we think long andhard about how all these little pieces
plug into a library or astandard library or an API. And now
(13:12):
the job is like that stuff...which I don't know, personally, I
like coding, but I didn't...I've never really found the rewarding
part of coding, like figuringout all the things that I need. It's
like, it's the elegance oflike the solution and the experience
afterwards, right? And like,that's the whole job now, right?
Like that's... your stack ofcode is not the thing that's going
(13:34):
to be your moat anymore. It'sthis experience and joy of the user
afterwards of what you'vecreated. And I feel like a lot of
the job now is just likeverifying and making sure that this
is good quality functionality.It's something that I feel like we're
gonna have a problem with thisin the space in the near future,
because I feel like thatintuition comes from years of production
(13:55):
experience... like being ableto look at things and understand
how the code all goestogether. I feel like we're gonna
have a gap in juniors in thenear future of like, people that
don't know how to do thatanymore because they haven't written
production code. But like, Ifeel like a lot of the job now is
gonna be guardrailengineering, observability engineering,
quality engineering, less solines of code engineering. So I would
(14:18):
absolutely love to hear whatyou guys are doing there.
Yeah, I mean, I would... Ilike how you broke it up into observability
engineering and guardrailengineering. I've just been calling
it harness engineering for thelast few months now, where it's just
really building the systemsthat enable teams to use agentic
decoding to ship software. Imean, when I think about it from
(14:40):
a platform engineeringstandpoint, since you know, a lot
of our listeners are rightnow, you know, platform engineers
in many ways. The job ismoving into developing these systems
that then allow your productteams to quickly ship software. So
how do you abstract awaycreating new guardrails or creating
(15:01):
new validation checks? Whatare the new primitives that you build
in this new world thatdevelopment teams inside your organization
or product teams inside ourorganization can take advantage of?
Because libraries... For along time, there was just libraries
(15:22):
and there was APIs, andarguably that stuff is being generated,
at least in some settings,these days. But there's still a lot
of things that are not beinggenerated, which is, "How do you
plug this in?" Like, I'll giveyou an example. I'm sure you've tried
your hand at vibe coding.
Oh, yeah.
Which is, you know, all therage back in twenty twenty-five.
And now it... I feel like it'sstill going pretty strong.
(15:44):
Yeah, I feel like it is. I'm aperson... I don't like typing, I
get really bad tendinitis. Ihave leaned fully into, like, Whisper
tied into Claude Code, and Iwill either, like, talk my way through
a spec and then just copypaste it in or I will just kind of
talk through and just rambleabout, like, what I want to build
(16:05):
and then watch it materialize,which is absolutely magical.
100% right.
Yeah.
Next.jsa pretty good workflow,you know, just using speech. I should
give that a shot. But if youthink about the early days of vibe
coding, when it was... notthat long ago, it feels like long
ago or like some time haspassed, but maybe it was like a year
(16:29):
ago or something... but whenit was first crystallizing, it would
generate everything. Right? Imean, it wouldn't use something like
React, it wouldn't usesomething like even Express. It would
just write raw JavaScript orit would just write raw languag
without using any libraries,and it would still work. You know,
like, it would still look likea little bespoke application. And
(16:52):
then, like, if you try anyvibe coding platform... you know,
there's so many of theseplatforms these days... or even if
you just use Claude, what itdoes now is it just uses React, it
uses Tailwind, it usesNext.js, right? it'salmostinjectingallof
all of these building blocksinto the vibe coding thing and creating
an app that kind of lookspretty robust and itdoesn'thaveasmany
(17:16):
failuremodes asanapp thatwasperhapswipecodedayear an app that
was perhaps vibe coded a yearago. And those building blocks platformengineering,I'vebeenencouragingmyteamstostartproducingthosebuildingblocks.Likewhatareuniquebuildingblocks
thatavibecodingharnesspullfromandinsertinto applicationsasyourproductteamsarenow
(17:43):
buildingtheseapplications.Right.So maybeit'sabespokecomponentlibrary.Maybeit'sasetofskills,youknow,that,
thathavejust.It'slikeit'samarkdownfile.Right.Alotof.A lotofengineeringnow
hasjustcomedowntobeingableto.Clearly able just... It's like
it's inEnglish, which Aamarkdownfile.Andwhatarethoseskillsthatyoucanproduceasa to
(18:04):
clearly articulatethatcanbeusedbywhite gradingharnesses
and pullinthosedependenciesasbuildingblocks?I'dbecurioustoget yourtakeonthat.v
ibecodin g
Yeah, honestly, one of thethings that's changed, I'd say, I
hate to admit this, since I'vebeen a developer for such a long
time, but one thing that'sreally changed about the way that
(18:25):
I develop, I've always been afan of test driven development. I've
been a TDD engineer for thelongest time. And for a good portion
of like my vibe coding era, Iwould still write all of my tests.
And then I still do this tosome extent today. And I would use
my tests as my prompts. It'slike, "Hey, this is what I want to
exist. Make this so." Right?And it's very good when you do that.
(18:48):
Now the thing that I'vestarted running into is it takes
a lot of time to write tests.It's always taken a lot of time to
write tests. And like, nowthat I have this speed in the development
section side, I'm like, "Iwant that speed on my prompting side
too." Like, I'm spending a lotof time prompting code gets generated.
I'm like, "How do I optimizethis side now that side's maxed out?"
(19:09):
And so now I've started tojust talk through my tests, which
is weird, but I still do TDDwith my LLMs. Like, I will talk through
the ideas of the tests, I willwatch it, generate it. They all fail.
That's great. That's what Iwant, right? And then we iterate
through it, right? And I'llusually open my first PR with just
like failing tests. So it'slike I can see the code. It happened
(19:30):
in a work tree, right? I cansee what the PR is going to change
to the test suite.
Right.
I can acknowledge that is whatI intended. I'll put a comment or
something if I wanted toswitch. But the thing that's fundamentally
changed about my developmentis I'm very good at documenting things
now, which I have just neverbeen good at. And I think a lot of
engineers, like, they don'tlike... engineers like to write code.
(19:53):
You still see so manyengineers that don't like writing
tests, like, people are like,allergic to it. It's like, "Oh, I
hate writing tests." That tome has always been like the joyous
part. It's like, "Oh, I'veexpressed my intent." Yeah, it's
like I've expressed my intentand it works. This is exactly what
I'm looking for. But I'venever been great at docs. It's like,
"Hey, it works. Get thisfucker out the door." An now I
(20:15):
spend so much time onarchitecture decision records. Why
is this the way it is? Whatimpact from customers or our production
infrastructure has caused usto think about solving this problem
this way? And then it's likelocked down in code. Right? And that's
one of those things. It'slike, I feel like many teams that
(20:35):
really leaned into things likeADRs and deep detailed specs on how
your V2 API works, theconstraints and things you put into
it. Those are things thatreally established teams with plenty
of time to do so do. And thenother teams don't. Right? Now, that
is again, part of my job, andwhat's cool about this is it helps
(20:58):
the LLMs quite a bit. And thehumans, turns out they love documentation
that's accurate too. Right?And it's just like, tha has always
been like such a hard part todo, especially in startups where
you're just constantly runningand running and running, like moving
on to the next MVP that acustomer wants. It's so hard to go
back and document the lastthing. But, like, again, like, that's
(21:19):
the job now. It's this, like,quality engineering is kind of what
I see? It's just like the codeitself is just... it's... I don't
care, like, if it rewrote itin Python tomorrow, I do not care
so long as my qualityengineering on the outside can tell
you this is how the API worksand I can guarantee it works. Doesn't
matter to me what's inside thebox. I love to get your take on
(21:43):
this because I've heard a fewpeople start to say this... I think
it was the CEO of Chainguardrecently wrote a blog post about
like open source is dead, itjust doesn't know it yet... and I
feel like I don't necessarilybelieve that, but I've got this gut
feeling that it might be true.And so it's like for an example,
(22:03):
I would never say, like, "Hey,let's spin up a new React." It's
good, there's a really goodecosystem, but at the same time there's
a lot of vulnerabilities thatare getting snuck in through like
a dependency of a dependencyof a dependency. While at the same
time, like in our ownproducts, like we get to things where
we're like, "Hey, we're addingskim integration" and it's just like,
that's important to us, wewill actually take the standard and
(22:25):
we will generate our ownlibrary that we can just own. It's
so critical to... it'sexternal system managing permissions.
I want to own all of thatcode. It's not something I ever would
have wanted to own in a worldtwo or three years ago because...
I've got to focus on myproduct. But now it's like, this
thing is really important tous and microsecurity, so I want to
(22:45):
understand how it works and Iwant to make sure that we have full
control over its stack ofdependencies. So yeah, so I don't
know, man, it just feels likea lot of it is good engineering principles
that we've always had thatwe've never had the time for are
the work now. And that'ssomething that will benefit the LLMs
and it's definitely going tobenefit the person that replaces
(23:07):
me, right? Or the personthat's joining my team that I don't
have time to go througheverything with. It's like, "Hey,
here's an entire markdowndirectory of decision records of
how we came to thearchitecture that we have and the
tools that we have, and whichlibraries we do and don't use, and
how we decide on when or whennot to use a library versus just
generating and owningsomething ourselves."
(23:30):
Ops teams, you're probablyused to doing all the heavy lifting
when it comes toinfrastructure as code wrangling
root modules, CI/CD scriptsand Terraform, just to keep things
moving along. What if yourdevelopers could just diagram what
they want and you still gotall the control and visibility you
need? That'sexactly whatMassdriver does. Ops teams upload
your trusted infrastructure ascode modules to our registry.Your
developers, they don't have totouch Terraform, build root modules,
(23:57):
or even copy a single line ofCI/CD scripts. They just diagram
their cloud infrastructure.Massdriver pulls the modules and
deploys exactly what's ontheir canvas. The result? It'sstill
managed as code, but withcomplete audit trails, rollbacks,
preview environments and costcontrols. You'll see exactly who's
using what, where and whatresources they're producing, all
without the chaos. Stop doingtwice the work. Startmaking Infrastructure
as Code simpler withMassdriver. Learn more at Massdriver.cloud.
(24:28):
Like, how are you guys seeingthat? Like you guys have a huge product,
you got a big team. Like youguys are... you have customers that
are very concerned aboutsecurity. You guys are very concerned
about security. Like how areyou seeing open source in your own
organization? Like is it stillsomething you just, "Oh, that library,
Python library, it's popular,I'm just going to grab it." Or do
you think through like thesupply chain of those libraries?
(24:50):
thingthat's such a goodpoint. Right? I mean, let me first
talk about TDD and you knowhow much I love TDD. My first job
was ThoughtWorks and you knowwe were like... ThoughtWorks was
a big proponent of test drivendevelopment. And you know, it was
just like the greats likeMartin Fowler and you know, like
a whole... some industrystalwarts, I mean who were part of
that movement. So that'sreally how I cut my teeth. And it's
(25:15):
th same thing that we seetoday, right? It's the spec first,
right? It's being reallythoughtful about what is it that
you want to build and beingable to express th constraints of
the product in a tightdeterministic way. I mean that's
test, right? And you write thetest, first it fails and then you
write the code that fulfillsthat and just that and nothing more.
(25:36):
And now that test is, youknow, it's either a spoken record
in your case or you know, it'sa detailed spec markdown file that
you know, describes exactlyhow the software is supposed to be...
exactly how the system issupposed to behave. Which can be
used for two things, right?One is for using it to generate the
(25:58):
code that implements thatbehavior and then more importantly
the code that validates thatbehavior. Right? And you can also
think about splitting yourtests in the same way that you split
training data sets when you'retraining models, which is you know,
you have 80% and then youwithhold 20% and use that for validation
(26:20):
to make sure that your spec,you know, your model is not overfitting
on the 80%. Right? So youcould also do... you could also split
tests up in that way where youcan have, you know, certain part
of the spec which is hiddenfrom the LLM when it's generating
the code. And then it's onlyused for validation to make sure
that it's validating the spec.Right? So there's a pretty interesting
way to structure a smallerspec which is not handed to the implementation
(26:44):
aspect of it, which is apretty cool pack that we've used.
When it comes to supply chainattacks that we have seen, you know,
in the last few months. Imean, you know, they've been rising
at a pretty exponential rate.And so, yeah, I do get that point.
You know, open source isunfortunately in a tricky, tricky
(27:05):
spot. And my take is that it'sgoing to evolve from open source
implementations into opensource standards. So there's going
to be a lot morestandardization where you pull the
standard from, you know, somesort of a GitHub or an open source
repository. And the standardagain has validations and tests and
(27:28):
all those sort of things. Andthen the implementation, the actual
production code that runs inthe hot path in your system is generated
by an AI that you have or yourteam of engineers. And then the open
source is literally just aspec and the set of tests that tell
(27:49):
you that your implementationis watertight and that way you can
defend your supply chain fromany, you know... because unfortunately
the reality is untrusted codeexecution is the biggest threat everywhere
right now, especially withvibe coding. Be that a supply chain
infiltration or you know, bethat a vibe coding surface in your
(28:13):
product where you're allowingcustomers to extend the product by,
you know, giving this like,"Oh, enter a prompt here, now generate
something." It's all untrustedcode. And I feel like it's... my
take is that the way to reignin that untrusted code is by tests
and it's by specs. Oneinteresting,youknow,Iwastalkingtosomebodywhoworksatalargedefense tech
(28:38):
companythatwe allknowofandoneofthe,oneofthethings thathe the
it,you know,heworksatFrontierAI it... you know,
he works at Frontier AI, intheir world... and they have a really
sophisticated real worldsimulation. So it's like a real world
simulation that's built onlike a game engine that, you know,
(29:01):
allows them to, you know,really test how things work in a
versatile setting withoutreally having to engineer anything.
I mean, like, that's a reallycool way to validate software, you
know, as well, like, which isnot... and again, in defense tech,
(29:21):
you're producing real worldthings. So your simulation is really
a physics engine. And now thelaws of physics are, you know, pretty
well known and, you know, wellwritten down for a long time now.
So it's almost easier to builda physics engine. But what does it
mean toIhadtolikebreakdownsomelawsofphysicsforintheobservabilityspace,youknow, whenyouhavenetworkflowlogsthataregoingfromoneplacetoanother,there'sa
(29:52):
if you're an observabilitycompany. Like, if I had Thesize ofthepacket,right?st
Yep.
You don't break that law. Noone ever breaks that law. Right.
Like no networking equipment,firewall, router is ever going to
break that law. They're alwaysgoing to log that. Right? It's never
going to be incorrect. Andthen figuring out those laws that
govern the environment foryour own domain and creating this
(30:18):
simulation realm where you canrun your software and validate it
to the point of a mathematicalproof almost. So like, "Oh, if it
works here, we know it's goingto work in production." That might
be a pretty interesting pathfor platform engineering teams to,
you know, think about and sortof pull the thread on a little bit.
(30:40):
Yeah. One of the things, likekind of going back to you were saying
earlier about like, just likethe different, like smaller tools
that you kind of put together.When I talk to my LLM to do my tests...
and for anybody who doesn'tlike TDD, you're going to recoil
when you hear what I'm aboutto say that I do, but I swear it's
the thing that helps us moveso quick... I speak to it in plain
(31:03):
English because I speak to itin like BDD, like Gherkin, like,
"Given this, win this." Andour tests... our test suite has skewed
that direction. And the thingthat's interesting about it is like
we now have these very like,factual statements, right? It's like,
"When there is a cube set ofdeployments," it's a very English
phrase that we use around theorg all the time, like it's a sentence
(31:25):
that even the sales guyunderstands what I'm saying. And
so the interesting thing isit's so easy to reason about now
because it's like it's a plainEnglish sentence and that ties to
a tiny function that does thatexact thing, right? And so now when
I'm talking to my LLM, it'slike a lot of the tests, it doesn't
even have to write because Ialready have stuff tied to each of
the given wins thens, right?So I get to speak in plain English.
(31:48):
And even our sales guy canwrite like, "Hey, you know, I was
talking to the customer andthey said they need this." It's like,
"Oh, yeah, well, you can talkto this thing in plain English and
it will start buildingfeatures for you." And the crazy
part is like seventy, eighty,ninety percent of the test suite
at any point in time isalready written because we have all
these common phrases. So it'scoming to the point where it's like,
"Hey, let's talk about doingsome flow log stuff." It's like,
(32:11):
oh, we just have to add a rulein plain English and a little function
and it'll write that functionfor us, and it's so easy to reason
about. And I see these peoplewere it's like, "Oh, I've got a 10,000
line PR." It's like, I've got10,000 three line PRs. And it's very
easy to reason about wherealmost anybody in the org can look
at it and go, "Oh, we've addeda new rule that does this." I can
see th code that it producesfor the test. And like, now there's
(32:33):
some logic that's going tofulfill that, right? I don't know,
like, that feels like a reallygood flow. Again, I know that people
that don't like TDD are like,"Ugh, cucumber. Like, I don't want
Cucumber and Gherkin back inhere." But it's so easy to develop
this way, I cannot imaginegoing back. And like, folks might
(32:55):
be hearing like, "Oh, you'renot developing anymore though." It's
like, "No, I'm doing morearchitecture and thought about software
than I ever have."
Right?
vibe the last part of the TDD,right? It's red, green, refactor.
Most people don't get to therefactor part. Like, how do you get
your PM to give you time torefactor your code? It's like, it's
part of the job. Now it's thejob, right? It's like, I see stuff
come in and I'm like, "Oh,this all looks like shit." Like,
(33:16):
now I get to think about like,what does a grander, better architecture
look like for customers on theAPI or even internal developers using
stuff. Like, how do I buildsomething that works well for them?
And it's like I get to focus alot more time getting rid of debt
than I ever have in anyproduct I've ever worked on. It's
really, really interesting.So, coming back to platform engineers,
(33:38):
I'd love to get your thoughtson this because I think the most
underappreciated part ofplatform engineering - it's often
invisible, right? Even a goodDevOps team, good Sysadmins, like
when it's working, nobodynotices you, right? Nobody's like
running around like highfiving this admin and they're like,
(34:00):
"That auto scaler really doesits job well." It's like when it
doesn't, that's when peoplerecognize you. Right? And so like
in this new era of, you know,a lot of code being generated by
agents and vibe coded by endusers, like, how does that change
what we're responsible for asplatform engineers? And stability?
(34:21):
Like, we can have a very...and I know it's kind of two phases
- is the platform engineeringteam vibe coding stuff and then is
the engineers on top of theplatform vive coding stuff - but
I would love to know how yousee the world through the lens of
a platform engineer, wherewe're just getting things that may
be a lot less stable at first,but it's unstable and coming through
(34:42):
at a faster rate... which wecan now apply quality to and get
it stable.
Cucumberoff, Imean,with BDDand Cucumber, I mean, you're bringing
back some fun memories. Rubyon Rails. Man, that stuff is so well
rooted in that... it still isso well rooted in that ecosystem.
TypeScript has taken over theworld these days, but I do miss using
(35:05):
that. I remember the syntaxand those sort of things used to
be a little bit unwieldy withcucumber, but that stuff got figured
out pretty quickly. And now, Imean, it's just, you know, using
natural language to expressyourself and not even bothering with
those, you know, those, thosesyntactical nuances and quirks. And
really, I mean, that becomesyour spec... the guardrail in which
(35:28):
your software has to operate.And I feel when it comes to platform
engineers, you're 100% right.It's pretty asymmetrical. Lik when
the system is working well,when you're doing a fantastic job,
that means nobody's noticingthat the system is working well.
And with vibe coding and withAI, that's exactly the thing that
(35:50):
platform engineering has touphold, right? Like can it sustain
a ten x or even hundred x oreven a thousand x more software running
on the platform? So justscaling it up in a reliable way and
really supporting these newcoding paradigms that are emerging
(36:11):
now. Which is, you know...coding factories is like my favorite
one, right? I mean, where younow have... or development teams
are moving toward a worldwhere instead of writing code...
So, okay, I mean, so if youhad to like back up, right, like
before twenty twenty-one,developers were writing code character
by character, right? Onekeystroke at a time. And then came
(36:36):
AI based ID autocomplete,right? Where you start typing something
and the AI was like, "Oh cool,maybe these are the parameters in
the function that you wanted."And then came prompt driven agentic
decoding, right? Where toolslike Cursor and Claude Code became
popular, where instead ofstarting in the IDE, you now start
on the right hand panel andyou, you know, it's like, "Hey, this
(36:58):
is what I wanted to do." Andthen, you know, it proposes a diff
that you then accept orreject. And now I see more and more
teams going toward a worldwhere they're codifying their practices
and, you know, the thing theyproduce in a set of markdown files
and then creating a systemwhere you just insert the spec from
(37:21):
one end and it produces afully functional pull request from
the other side, right?
Yeah.
Taking into account like allthe quirks of your system and your
enterprise and your companyand all the coding standards and
everything else. And I feellike that entire harness again, that
(37:41):
factory harness, which iscalled the factory pattern, right?
I mean, it's different thanthe factory pattern that I used to
use when writing code way backwhen, but now it's just like, okay,
cool, like you have this, youhave this agentic harness that can
one shot an entire spec into afully functional PR. And that's going
to be the next thing thatplatform engineering has to tackle,
(38:02):
right? Like maintaining thesefactories and maintaining the assembly
lines. And not onlymaintaining, but innovating on that
and you know, making themfaster and you know, creating systems
where these pull requests whenthey get produced... because guess
what happens when you have afactory producing 1,000 pull requests
an hour, right? I mean, that'sa lot of code. Where is that going
(38:25):
to be deployed? How's thatgoing to be validated? So scaling
up not only the productiononline infrastructure, but also everything
around it is like, well, let'screate systems that can deploy PRs
and validate PRs at agentspeed, not at human speed. Right?
So like a thousand PRs an hourshould be deployed, should be validated.
(38:48):
You know, like some sort of asimulation environment. I would expect
that's the next frontier thatplatform engineering has to, you
know, conquer. I mean, there'salso. Sorry, go on.
No, no, I'm just like, thatis. I mean, you're just. Yeah, my
brain's going... It's funnybecause like so much of platform
(39:08):
engineering has always beenabout like the golden paths to production,
right? I can get you somecloud resources easily, or I can
get your Docker Container ontoa K8s cluster, or your functions
into a lambda. Like differentteams got different paths. But like
now it's like if every team...you got a front end team, you got
a backend team, you got acheckout team, you got a payments
team... if all of these teamsare kind of inventing their own little,
(39:31):
you know, guardrails andharnesses everywhere. Like, it's
a lot of rework, right? Ithink you're right. I think you'll
probably see a lot of thatcome into the platform. And that's
interesting because thinkabout... I mean, we're already to
the point on my team whereit's like, the number of PRs that
we produce today is stressful.I was talking to my dad recently.
(39:56):
He was an air trafficcontroller in Guantanamo. So like
a pretty intense job. Like wayback in the day, not like recently.
Sure. Yeah.
I was just like, "I feel liketoday developing is so much more
exhausting than it was a yearago." And I was like, "And it feels
weird because I don't writesoftware anymore." I was like, "I'm
just reading and processing."And he's like, "That's the scope,
(40:18):
dude." And I was like, "Whatdo you mean, the scope?" And the
scope is what they call likethe little radar thing that they
watch. He's like, "That's whatthe entire job of ATC is like, you
do not get to use the flowstate part of your brain whatsoever.
You're constantly analyzingand like using the executive function
side of your brain." And it'sliterally more exhausting because
you don't tend to use thatside of your brain a lot. You're
(40:41):
using the creative side ofyour brain, you're thinking through
things, right? You're havingthese deep flow states versus like,
"A plane is landing, a planeis landing. Two planes are near each
other." And that's what we'redoing with these PRs. It's like,
"These two PRs conflict withmain, like, what does that mean?"
I'm shifting contextconstantly and he's like, "The way
they solve this in ATC isevery two hours you have to take
(41:03):
a break." Because your brainlike after about two hours, like
it just... the executive sidejust plummets, right? And with all
of these changes... it's likethe chang rate is insane. A to
of PR is going through and youknow, to have people overlooking
all of those is hard. It'shard on people, right? And so like,
(41:24):
you know, as we think about,"How do we get value out of AI for
a business?" Like the AIcan't... like the business can't
take all that value becauseit's going to exhaust humans. Right?
Humans got to get some valueout of it too. Like do our hours
change? Do we work less? Do westart to see companies as benefits?
Saying, "Hey, engineers workfor two hours a day and then they
have off until after lunch andthey work for two more hours." Right.
(41:46):
Like, I think the job's reallygoing to change a lot. But you know,
at the same time with allthose PRs coming through, it's like
starting to... again buildingblocks and guardrails. Like maybe
every single PR now that youhave more time, maybe every single
PR does go through k6 and doload testing on it. Maybe every single
one does land in a previewenvironment that has a full suite
(42:07):
of browser tests go across itthat you've never had the time to
do before because you werekind of slaving away over every single
save to a database. Right? Andthat quality work is the thing that
every PM wish you had time todo but you never did because they're
focusing on revenue, right?Like that's the job now. And so it
will be interesting to seemore of that start to land on platform
(42:30):
engineers' plates.
That's such a good comparisonwith ATC. I mean, like that's exactly
how I feel when you... it'sthe shift away from being able to
do deep generative work intolike more reactive work. When we
(42:51):
talk about, well, AI is goingto, you know, ten x or hundred x
a team's productivity,ultimately the team is accountable
for that ten x output, whichmeans it's going to... from first
principles... is going to comeat ten x more cognitive load on that
team. Because I don't see usas an industry getting to a point
where we are okay withshifting that accountability to AI,
(43:12):
right? The team is still...the human team is still going to
be accountable for whateverthe team produces, right? Whether
they use AI or not. So thereis a pretty deep cognitive burden
I feel when it comes toplatform engineering. I mean just
inventing tools and techniquesthat simplify that burden... that
(43:33):
reactive burden... andhelping, you know, that review process
easier. One of the ideas thatwe have talked about at Cribl i
not all code in your code baseis equal. You know, there is certain
code that's important and thatthere's certain code that's maybe
not as important. S likereally splitting it up or you know,
drawing rings around your codelike, "Oh, this is the ring 0, the
(43:57):
kernel." Like in Cribl's casethat would be the score stream processing
engine that can processhundreds of terabytes of data or
transform, you know, hundredsof terabytes of data in an hour.
Lik that's the kernel andthen the ring around that is maybe
like the control plane for thekernel, right? So what sort of stuff
(44:18):
are you putting in?Optimizations, those sort of things,
then kind of like expandingthat. And then at the very edge you
have smaller things like somesettings panel hidden in the UI,
UX somewhere. So likesplitting that between kernel land
and user land, kind of how theLinux system does it and then treating
(44:40):
different grades of PRs. So(a) like a PR should not span across
multiple rings, it should bein its own ring. And then (b) treating
those differently. Right? Imean like if it's PR that introduces
a bug, but then you can... ifthe bug is caught, like you know,
if you... it's easy to remedythat, you know, like you can push
(45:02):
out a fix in ten minute andthere would be no impact versus messing
something up in like the core.You'll probably have like more human
scrutiny.
Yeah, I'm going to give youanother Thoughtworks throwback that
I feel like you're sayinghere, but you're not using the words
- it's domain driven design.Like you're talking about like the
core domain, right? Like thatwas always the thing. It's like it's
so hard. And so many codebases where it's just like not a
(45:24):
monolithic repo, but like amonolithic code base where it's just
like... you know, in Rails youopen the app folder and there's just
seven thousand model inthere, right? But like starting to
organize. And again it's... thi is quality engineering. This
is the stuff we all wished wehad time to do in two thousand fifteen
and never did. And now it'slike thinking about the different
parts of your business thatare critical. This stuff is going
to get... you know, that coredomain, that string transformer...
(45:48):
like that's always going toget humanized. I don't know, maybe
the user settings page islike, "Eh, we don't give a shit.
Like let's just let it merge,like the testing, like let it go.
Like let's focus on makingsure that the secure parts of the
app are secure, the core hotpaths are good. And the domains that
(46:08):
are not the core domain, maybethey don't need as much attention
in guardrails, and we focus onthe part that does." And it's easier
to reason about stuff in thatworld too, right? Because it's like
everything's scoped down tothe little circle that you're working
in. Whoah, baby.
Yeah, yeah, yeah. It'schanging fast. The world is changing
(46:28):
fast. Software engineering,platform engineering, our entire
industry is at the tip of thespear when it comes to AI transformation.
Since code is such a... I meancoding is like the first breakout
case of generative AI, right?I mean like it's the first breakout
success. I don't thinkanything is quite that breakout even.
(46:49):
I mean the next best thing isprobably self driving cars. Those
are in like a handful ofcities, right? Whereas agentic coding
is all around. Like everysoftware engineer, every... like
no matter where you're in theworld is using some form of agentic
coding. At this point we getto decide, you know, or we get to
be the guinea pigs of, youknow, how this is going to play out.
Hey, and we don't... you don'thave to be the guinea pig. This is
(47:12):
the thing that's exciting tome about this that I was like, I
feel like all these like AICEOs, like I feel like they are fucking
terrible. They're great atraising money, but they're terrible
at marketing. Like coming outand being like, "Fifty percent of
you are going to lose yourjobs in the next three years." It's
like, "Hey, you know whypeople throw Molotov cocktails at
you? Like, it's because youaren't making people happy with your
(47:33):
words." B it's... at the same time, it's like we've had a
few of these... like, I don'twant to say revolutions... but these,
like, big changes in, like,software engineering over the past,
you know, twenty, thirtyyears. And we're in the next one
right now, right? And I feellike this is the time where it's
like, you can get a lot offree labor from these subsidized
tokens to make your code basesreally good. And then you can think
(47:55):
a lot more about what does thefuture of engineering look like for
our industry and what does itlook like for our orgs, right? And
that to me is exciting. It'like, it's still engineering work.
It's headier stuff. I thinkit's more interesting problems to
solve than save user todatabase. Like, you've done that
at your last seven jobs indifferent languages. We're at an
(48:17):
exciting time. And I thinkmost people that are afraid of it
are afraid of it because ofthe words that these CEOs are saying,
right? And it's like, I thinkthey're saying that because they
want to raise money again. Ineed more people because I'm making
more code, right? It's like, Ineed more people, right? So I don't
(48:39):
know. It's wild times. We canbe guinea pigs, but I think a lot
of us can also be computerscientists now. Don't be the guinea
pig. Be that, right? Be theguinea pig scientist. I don't know.
Like, let's figure out howthis new world looks because there's
a lot of stuff to figure out.I think CI is going to look a lot
different in the next fewyears. People are starting to talk
about... like, I know a lot ofpeople are pointing to GitHub and
(49:00):
be like, "GitHub's fallingover." It's like, "Dude, GitHub is
getting way more commits andrepos than they've ever had in their
lives. Like, they have a lotof just like, stability to deal with
from just like the onslaughtof data being thrown their way."
Like, you can, you can blameit on core AI all you want, but,
like, they're handling morewrites than most of us, right? So
it's like the world, ourengineering stuff that we have kind
(49:22):
of clung onto since the firstDevOps talk from Flickr, like CI/C
and Git and all this stuff...It's like, this stuff is likely going
to change and it's us who getsto make the decision as to what that
looks like. It's us that hasto get familiar with how these systems
work to be able to make thosedecisions. And I feel like if you're
(49:44):
sitting around like afraid ofLLMs and AI right now, like those
decisions are going to be madefor you, which sucks.
A hundred percent. Right. Imean, everyone has got to be thinking
about how do they become anowner of whatever their domain is.
Right? I mean just startacting like an owner of the problem
that you have, of the problemthat you're trying to solve, and
then, you know, use every AIagentic software tool out there to
(50:08):
help you conquer that and justreally deliver that ROI. Really.
So like if you're anengineer... if you're a front end
engineer, you know that looksdifferent for you, then if you're
a platform engineer, thatlooks different for you. But ultimately
it's just showing relentlessownership of the problem that you're
trying to solve - What areusers looking for? Where your customers
(50:28):
are paying you money for andthen going and really delivering
on that front. Which is a farbetter place to be in than being
a guinea pig.
Yeah. Awesome. Well, I reallyappreciate you coming on the show
today. It was super fun. Wherecan people follow your work and learn
more about what you and yourteam at Cribl are building?
Yeah. So you know, if youfollow me on LinkedIn. Follow me
(50:51):
on Twitter. We're prettyactive on LinkedIn where we post,
you know, all the cool stuffthat we are doing at Cribl. Just
this week we released a bunchof benchmarks on LinkedIn where we
compared privacy sensitivedata scanning models that we build
in-house, that we trainedin-house, and how they stack up with
OpenAI's new privacy filter.So OpenAI released a 1.5-billion
(51:13):
parameter open source model onHugging Face. It's awesome, definitely
go check it out. It can scanand redact sensitive information
from any data that you throwat it. And at Cribl, we have our
own in-house built models thatare way faster than OpenAI's models.
Much more accurate as well. Soour F1 scores are... which is a score
(51:37):
that really tells you aboutprecision and recall when it comes
to AI systems, really howaccurate a model is... we're able
to benchmark our models atmore than ninety percent wherea
OpenAI's model was aboutfifty-seven something percent. And
then our model is also ahundred and thirty-three x faste
than OpenAI's model becausewhat we have been able to do is we
have been able to compressmuch higher accuracy in a really
(52:01):
small model. So you're talkingabout a hundred millio parameter
model that can run on a lot oflogs, a lot of telemetry data, observability
data. Because guess what? Withagentic AI, LLMs are everywhere.
And people are sticking allkinds of things into LLMs these days,
right? Personally addedinformation, health records, Social
(52:21):
Security numbers, phonenumbers. And one of the biggest challenge
that IT teams and securityteams have to conquer is how do you
scrub all of that informationfrom access logs, from audit logs...
a really pressing problemacross the industry. So I'm really
happy that we are leading thecharge on that and developing these
models that enable you toscrub sensitive information so it
(52:42):
doesn't land in places that itshouldn't. You don't get one of those
weird letters in your mailsaying that, "Oh, there's been a
data breach." Or, "You'resigning up for one year free of some
monitoring..."
You know what happens when youhave six Equifax monitorings that
all overlap from all yourbreaches? Nothing. Your data's still
out there.
It's all out there. Right, exactly.
(53:03):
That is funny. We actually...we just added a hook to our stack
that essentially like... Idon't want to say... kin of like
a loud list, like what wecould put into OTel. We caught that...
you know, we lean into OTELpretty heavily. Folks that have listened
to the show before know that Iused to do a lot of what I call the
OTel driven development. Idon't do as much of it anymore. But
(53:25):
like we lean into OTEL prettyheavily. Like, we don't... we actually
don't emit a single log fromour server, we just... everything
goes into OTel. And I'll tellyou what man, LLMs love to put the
entire record into OTelattributes if it can. And so like
we have rules on like thetypes of data that can go in. Like
even stuff that's not... wedon't necessarily deal with PII,
(53:45):
but we deal with other thingsthat... you know, password secrets,
credentials for databases,like all sorts of stuff like that.
And yeah, so it's just like,"Dude, that is important." Like,
yo have a big team... ahundred, two hundred, three hundred
engineers... you are going tostart leaking stuff into logs that
probably should not be therefor sure. That is awesome. Is that
(54:07):
mode a part of Cribl or isthat open sourced as well?
It's right now part of Cribl.It's Cribl Guard, which is a product
we released a few months agoand that's how you access that model.
And we're going to release apaper on arXiv in the next week or
so that will enable you toreproduce that sort of a model on
your site as well. So thatpart will be published.
(54:30):
Awesome. We'll put a link inthe show notes to Cribl Guard and
then if... yeah, when thatpaper's published we'd love to put
that in the show notes as wellfor folks.
Awesome. Yeah, we'd love that.
Yeah. Awesome. Well, thanks somuch for coming on the show today.
Everybody listening, I'll seeyou next time.
Popular Podcasts
Hey Jonas!
Hey Jonas! The official Jonas Brothers podcast. Hosted by Kevin, Joe, and Nick Jonas. It’s the Jonas Brothers you know... musicians, actors, and well, yes, brothers. Now, they’re sharing another side of themselves in the playful, intimate, and irreverent way only they can. Spend time with the Jonas Brothers here and stay a little bit longer for deep conversations like never before.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by Audiochuck Media Company.
Betrayal Weekly
Betrayal Weekly is back for a new season. Every Thursday, Betrayal Weekly shares first-hand accounts of broken trust, shocking deceptions, and the trail of destruction they leave behind. Hosted by Andrea Gunning, this weekly ongoing series digs into real-life stories of betrayal and the aftermath. From stories of double lives to dark discoveries, these are cautionary tales and accounts of resilience against all odds. From the producers of the critically acclaimed Betrayal series, Betrayal Weekly drops new episodes every Thursday. If you would like to share your story, you can reach out to the Betrayal Team by emailing them at betrayalpod@gmail.com and follow us on Instagram at @betrayalpod and @glasspodcasts. Please join our Substack for additional exclusive content, curated book recommendations, and community discussions. Sign up FREE by clicking this link Beyond Betrayal Substack. Join our community dedicated to truth, resilience, and healing. Your voice matters! Be a part of our Betrayal journey on Substack.