March 23, 2026 • 30 mins
Adi Polak talks to Aravind Suresh (OpenAI) about his career in distributed systems and real-time streaming. Arvind’s first job: coding at school. His challenge: turning OpenAI’s fragile Kafka setup into a reliable, multi-region streaming backbone.
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
SPEAKER_02 (00:00):
As long as we are able to build a system, it could
be uh abstraction that makes iteasy for our users, and as long
as it solves a problem, that'sgood.
I've noticed time and again thatsimple systems scale better, any
kind of like clever, complexcode.
Ultimately, in the long run,although if the overall
architecture is simple toexplain to others, will it stand
(00:20):
the test of time?
SPEAKER_01 (00:21):
Hello, I am Adi Polak, and you're listening to
Confluent Developer, where weexplore the fascinating journeys
of software developer tacklingcomplex problems.
In this episode, I'minterviewing Arveen Suresh from
his early days exploringdistributed systems to becoming
a key figure behind open AI,real-time infrastructure.
(00:46):
Arvine shares a story of scale,simplicity, and building the
right abstractions for AI.
His early passion for coding,starting back in school,
eventually led him into theworld of data streaming,
large-scale systems, andenterprise software.
So, what did OpenAI dodifferently to scale their data
(01:06):
streaming platform?
Have a listen.
Hi, Arvind.
I'm so excited to have you onour show.
SPEAKER_02 (01:19):
Hi, Andy.
Uh executive here.
Nice to meet you.
SPEAKER_01 (01:22):
Yeah, well, thank you.
First of all, thank you so muchfor being part of Corinth Nola
Program Committee.
Your insights and comments werefantastic and really helped us
build a good program.
And also, thank you for being aspeaker and sharing your
expertise.
I know, you know, behind thescenes, there's a lot of work
that you put into actuallytaking everything that you know
and turning it into apresentation that people can
(01:45):
learn from.
So thank you for doing all thesegreat things for the community.
SPEAKER_02 (01:49):
Yeah, thank you.
Excited to share our learningsas we uh like whatever we have
done uh at OpenAI.
So it was fun being part of theconference.
SPEAKER_01 (01:58):
That's that's fun.
So I definitely know you.
A lot of our program committee,and maybe some of the folks uh
from Corinth knows you, but noteveryone who are listening to us
today do.
So maybe you want to share alittle bit about yourself.
SPEAKER_02 (02:11):
Sure, yes.
Uh uh I'm Arvind.
I'm currently at OpenAI.
I lead the real-timeinfrastructure team.
So we power the streamingbackbone behind all AI products
like ChatGPT, Soda.
We also help a bunch of researchworkflows too.
So I've been doing distributedsystems and data platform for
like close to eight years now.
(02:32):
Uh and I've been like I've Ifound this space uh sometime.
I can tell you about that in alittle bit, but like the there
was a moment in my life when Ifigured out that distributed
systems was something that I wasreally passionate about, and
I've been like diving deeper, uhbuilding systems that scale
pretty much my entire career.
SPEAKER_01 (02:52):
That's amazing.
I mean, there's so much to knowabout distributed systems.
I always feel like it's it's awhole world that when you start
digging into it, when you're auser of a distributed system,
let's say you're a consumer ofan API, it's it's one angle.
You try to optimize, you have aproduct mindset.
But then when you dive deeperinto the infrastructure, it's
(03:15):
like, oh, things are notconsistent.
What do you do?
SPEAKER_02 (03:21):
Yeah, exactly.
Like uh, I think this is a goodpoint about infra and product,
right?
Like um, I've been part ofinfrastructure teams pretty much
my entire career, and it hasalways been uh I like working
with product teams or likedifferent teams in the company
because ultimately we are hereto may build the foundations
that make products successful.
(03:43):
So we're co-buildinginfrastructure with you.
So in early stages of the teamor the company, you end up
co-building your infrastructurewith the product team, and the
product teams will beco-building infrastructure and
their products with you.
So I really like thatexperience.
That is more fun, understandingtheir use case, partnering where
we can help.
SPEAKER_01 (04:01):
That's amazing.
So I know you've been, you know,you have many years' experience
with data streaming and kind ofdata streaming is is a big
passion and uh and theprofessionalism, you know,
career that you chose.
Um, I'm curious, like, what aresome of the challenges that
you've seen throughout yourcareers that, you know, and
learnings that are people canlearn from you?
SPEAKER_02 (04:22):
Sure.
Um, so let me start with like ummy recent experience at OpenAI.
So the challenge here wasbasically how do you build a
reliable streaming platform whenyou're scaling 10x every six,
seven months?
So your platform scales 10xevery six, seven months,
(04:43):
different sets of use cases,different shapes of workloads
that we support.
Now, how do you make sure uh theplatform is also reliable when
we do all of this, right?
The core challenges uh involvesomething like balancing
velocity and stability with likea such a small team, and how do
you work with users?
How do you hide all theinfrastructure complexities?
(05:05):
Um, so that just to give somecontext about uh OpenAI's use
cases, right?
The streaming infra is kind ofat the heart of OpenAI in like
different workflows.
Like uh one of the core thingsthat we use Stream Infra for is
something called Flywheel.
What that means is um we buildgreat AI models, then we launch
products using those AI models,then you get some usage data
(05:29):
about that product based on whatour users, how our users use
their products.
And insights from these areactually used to build better
models later.
And this kind of loop keepsspinning.
Like you have more models, youget better usage data, then you
get insights, you train bettermodels, and so on and so forth.
And guess what?
The data pipe here is Kafka.
So Kafka is used to store all ofthese events, and you use Plink
(05:52):
a lot of stream processing toget all this data as real time
as possible wherever it'sneeded, right?
Um, and also in all uhdistributed system settings,
let's say experimentation isheavily distributed today.
So the our researchers andengineers would like to know the
latest status of an experimentand see whether it makes sense
to continue that or not.
And today's uh training isheavily distributed, right?
(06:14):
So you have workers talking toeach other, and Kafka has the
data pipe to carry those events,and Flink is used to transform
and join that to basicallyunderstand what to do.
It's kind of bread and butter ofuh researchers and engineers.
And um, I've seen this inaction.
Like without streaming infra, itwas very hard for them to build
things.
With stream infra, it wasactually easier for them to do
their day-to-day.
(06:36):
So, in this background, um, thisis basically kind of the result,
right?
But this is not how it was whenI joined uh the company.
It was like early stages, uh, itwas still a scrappy uh piece of
infrastructure.
Like there was nostandardization around Kafka
clusters at that point.
Like there was no blessed paths.
(06:56):
Uh, Kafka was a single point offailure, uh, and there was a lot
of friction and confusion onlike how to use Kafka.
So this is the state there, andlike it was like close to a year
or two um years of work to getus to this shape.
SPEAKER_01 (07:10):
Wow.
So let me see, you know, collectall the all the pieces of
information.
Essentially, when anyone in theworld decides to use OpenAI
models, uh, we as humans cangive some sort of feedback, like
thumbs up, thumbs down, uh, orhave some sentiment to uh uh to
what we do.
Uh if we're happy, not happy,asking for uh improvements and
(07:32):
so on.
And that essentially becomes afeedback to um to for retraining
or rebuilding the models andimproving them so the the whole
system can become better.
And that's as a feedback turnsinto Kafka events that are being
processed later on with Flink ina streaming engine in real time.
(07:52):
And that's kind of like thebackbone for everything that you
do.
And then throughout, you know,as as you have these uh data
data streaming pipelines,essentially it's not serving
only that feedback loop with themodel, but actually, you know,
also researchers today that wantto run different
experimentation, or anyone whouh you know wants to tap into
(08:13):
what is happening in the systemneeds to have access to that
real-time data, essentially.
Right?
SPEAKER_02 (08:20):
Yes, that that is cut, yes.
SPEAKER_01 (08:22):
Got it, got it.
And then your team has to managethe cluster, right?
Uh manage the large scale ofthis distributed system.
Um you know, I'm guessing that'sthat means you know, a lot of
machines.
Um and then um make sure it runswith the right latency for the
(08:42):
system, right?
SPEAKER_02 (08:44):
Yes, yes, that that is correct.
The the emphasis on reliabilityis actually very high uh because
all of this is mission criticaldata, it's important to us.
So the we promise close to four,four, four and a half nines of
Kafka uh availability.
And also the core uh coreprinciple behind all of this was
(09:05):
uh building reliable systems wasof course important, but another
key principle that we kept inmind is how do we make all of
this easy for our users to use?
Because uh, when I say users, sowe are an infra team.
When I say users, I talk aboutall the other employees,
engineers, researchers at OpenAIwho would use all of this.
So, how do we make thatexperience easy for them?
(09:25):
So the ergonomics part was veryimportant.
Uh, it was like building AGI andbuilding AI systems is hard in
its own way.
People did not want to becomeKafka experts, right?
Uh so how do we build the rightabstractions?
Uh, we made some earlydecisions.
Like, for instance, instead ofexposing Kafka clusters directly
to users, we built layers ofabstractions.
(09:45):
So we built we built a bunch ofproxies surrounding our Kafka
clusters so that we can abstractKafka away from our users.
It helps us to do a lot ofoperations, scalability.
It also helps in uh multiplex.
So our Kafka topologies likeKafka messages get multiplexed
across different clusters anddifferent regions, and that
helps in higher availability.
(10:06):
That way, Kafka would not be asingle point of failure.
It does have some trade-offshere and there, but happy to add
details there.
But this actually simplified uhconsumption of Kafka and like
building workloads on top ofKafka easier for our users.
Because tomorrow we can scale acluster, we can add more nodes,
(10:27):
we can rebalance them, we canmove them from one region to the
other, but having a layer ofindirection in between actually
helped us make those it made allthose operations easier.
SPEAKER_01 (10:37):
Got it.
So if essentially, um just tosee if I understand correctly,
different people in theorganization, these are our
users, different people havedifferent skills and different
expertise.
Like ML researchers, they haveyou know probably the best
knowledge and on how to buildalgorithms and how to improve
them and so on.
Uh, and they have knowledge ofthe tools that they're using,
(10:58):
but we can't really ask them tolearn everything about Flink and
Kafka.
And so we want to give them uhtools that are uh intuitive to
what they do by building a layerthat uh abstract away Kafka and
Flink complexities uh so theycan run faster.
That's it, right?
(11:18):
Okay, got it.
Yes, and what are some of thechallenges that you ran into?
Because I can imagine, you know,people would have all kinds of
requests.
SPEAKER_02 (11:26):
Yes, yeah, that's a good question.
So our challenges here wasmainly around Kafka being a
single point of failure, butthat's actually a very hard
problem.
Like uh if you deploy, let'ssay, Kafka clusters in a single
region, and how do you make howdo you make that work in a
multi-region setup, right?
Like either you can go down thepath of having stretch clusters,
(11:48):
which I've seen in some cases,uh, where you have Kafka itself
distributed across regions.
Uh, there are obviously latencyconsiderations, stability at
scale considerations here.
Um, there are some other uhdesigns I've seen where you have
mirroring from one Kafka clusterto the other, so that in case a
region goes down.
So we were solving about caseswhere even the cases where
(12:10):
regions were going down, westill wanted Kafka to be up.
So even in those cases, um we wehave we basically had like a
bunch of these design options,which were all kind of complex.
Um we didn't know, we werethinking whether to do stretch
clusters or whether to domirroring.
And that's when we decided, canwe just make some
simplifications here?
(12:31):
We thought about uh, okay, wecan have built layers of
abstractions in between.
And let's say uh you multiplexmessages from one uh topic
across multiple Kafka clusters.
And when we do that, youobviously throw out some notions
of like partitioning, ordering,which are kind of very, very
important.
Like uh for folks who come froma traditional Kafka world, they
(12:52):
would be surprised, they wouldfind it odd that we threw away
these core primitives that Kafkabecame famous for, like
partitioning, ordering.
But we realized that a majorityof use cases uh were okay with
not having those guarantees.
They were okay with, let's say,for partitioning, they were okay
with doing that downstream,let's say the Flink app or the
downstream consumer.
And ordering, as you know, is ahard problem in distributed
(13:15):
systems itself, right?
So instead of relying onreliable ordering of events, why
can't you just stamp your eventswith a logical clock and use
that to uh sort in thedownstream?
So we made a bunch of these keytrade-offs and we went back to
our users and talked to them ifall these things make sense to
them.
And they were okay with it.
(13:35):
And that's when we actually wentand built our first version of
Kafka at OpenAI.
It was basically a producer-sideproxy which multiplexes messages
across multiple clusters, whichagain there is a consumer-side
proxy which pulls all of thisfor you and sends it to your end
consumer services.
So this was our first version.
(13:56):
This came in sometime aroundmid-2024.
SPEAKER_01 (13:59):
Amazing.
So essentially, it took a lot ofcomplexities out of Kafka and
some some a lot of uh failureguarantees, I will say.
So it's probably you know,coming.
I'm sure like the person thatcame up with that idea was
everyone looked at it and waslike, Who are you?
Do you know Kafka?
(14:20):
Um, but apparently, you know, itworks because at the end of the
day, you're serving a user, andthese users are okay with these
trade-offs.
SPEAKER_00 (14:28):
Now a quick word from our sponsor.
Confluent Developer the Podcastis brought to you by Confluent
Developer the Website, which haseverything you need as a
developer of data streamingsystems.
And it's completely free.
We've got curriculum, hands-onexercises, executable tutorials,
the online data streamingengineer certification, also
free, a way to find a meetupnear you, those are free.
(14:51):
Everything is there.
I really want you to besuccessful in your journey as a
data streaming engineer, andthis is the site that has what
you need.
Check it out atdeveloper.confluent.io.
That's developer.confluent.io.
Now back to the show.
SPEAKER_01 (15:07):
It's it definitely shows like a flexible mindset of
an engineering, because youknow, many times in our career,
and I my background is also inthe in the um distributed system
infrastructure.
There's like the things thatwe're used to do, the way we
forever did things.
And um being user-obsessed isdefinitely um a value that can
(15:31):
help us build systems that caterto our users versus trying to
answer all the best practices inthe world, I'll put it that way.
Um so that's that's exciting.
I mean, that's that's a biglearning.
SPEAKER_02 (15:44):
Yeah, I totally agree with what you said, right?
Like uh ultimately we arelooking, we are building systems
that solve problems.
Uh, it doesn't matter whether weuse a Kafka or a database or
whatever tools are going tohold.
Uh as long as we are able tobuild a system, it could be uh
an abstraction that makes iteasy for our users to use that
system.
And as long as it solves aproblem, that's good.
SPEAKER_01 (16:07):
Right, right, right.
No, at the end of that's that'swhat we're paid to do, right?
Solve the problem.
So I love it.
Cool.
Well, thank you for sharing.
That's um that's reallyexciting.
I know you build more things,right?
SPEAKER_02 (16:22):
Yes, uh, we built um so immediately after we shipped
this, there were more thissolved all the stateless
consumption from Kafka.
And then after that, uh therewere use cases around stateful
processing.
So that's when like we startedadopting Flink more at OpenAI.
Yeah, and also the some of theproblems are similar across
(16:44):
both.
Like, how do we make sure Flinkapps scale?
How do we make sure seamlesslywith this multi-cluster Kafka?
Like it's it's a differenttopology, right?
If you have Flink working with asingle Kafka cluster, there are
in uh everyone has been doingthat for years, but how would
you make it work with multipleKafka clusters?
What uh how do you we had tobuild uh a custom flink source
(17:08):
for it to work?
Then we had to build a customflink sync, then whether then we
had to invest in control planesbecause it um it's only a matter
of time.
This is also something that Iobserved in the last year or so.
It's only a matter of time untiluh the system that you build,
which is horizontally scalable,after a year or so, you end up
with like a lot of clusters, alarge split of clusters for you
(17:31):
to manage, and it ends up with alot of operational toil, on-call
pages, et cetera, which is whenwe realize the importance of
control panes and importance ofself-healing systems, auto
healing systems.
And that's kind of where uh wewent next.
So we introduced control planesfor all parts of an
infrastructure, including Kafka,our consumer side of Kafka,
(17:55):
Flink, all of these systemsended up having control planes
just to manage the scale ofclusters that we are dealing
with.
SPEAKER_01 (18:02):
Yeah, that's that's smart.
It's um from a certain scale,you have to have it.
I remember uh back, well, it wasuh when I was in Microsoft some
years back, but it was uhpost-acquisition of LinkedIn,
they were building um differentclass different Hadoop clusters,
(18:22):
and then there was a controlplane that managed a couple of
hundreds or thousands of uh miniHadoop clusters just to uh
support the scale.
And um yeah, I believe uh todaywith uh cell architecture, um it
is highly used.
This is uh what we have in inConfluent, they call it the Cora
(18:44):
engine.
There is a paper out there, uh,but we essentially we build cell
architecture so we can supportas many Kafka clusters as
needed.
SPEAKER_02 (18:53):
So I see what is a cell here?
SPEAKER_01 (18:56):
So the cell architecture is essentially it
can be a collection of um well,a Kafka cluster that supports a
collection of things in it.
So you can read about thearticle, it's we you know, it's
it requires it requires lookingat the architecture, but
essentially this is how uh AWS,for example, build S3.
(19:16):
So this is the architecturebehind that.
Um so it's a multi-tenancy,multi-tenancy environment that
keeps that takes care ofscaling, security, uh, and a lot
of other aspects.
And that was a great uh,especially in the recent outage
that was, um, confluence stayedup, which is important because
you don't want to put everythingin on one region.
(19:39):
Uh you want to continuesupporting our users.
Uh so I'm very happy you'rebuilding a control plane and you
have different control planesfor what you're doing.
Uh, and I'm curious because youknow, adding Flink is something
that you know you have to thinkabout.
Like, how do you how do yousimplify Flink for the rest of
the organization?
SPEAKER_02 (19:58):
Yeah, so it started off with building the right um
uh so ultimately, uh yeah,rightly said, Flink has its own
complexities.
Uh like it's it has its ownlearning curve, right?
It's not as easy, like it's easyto talk to Kafka and publish
messages, but it's not so easyto write a Flink app.
So it started off with um usworking closely with users to
(20:21):
explain Flink semantics to themand building um so most of the
projects that ended up havingcomplex Flink apps uh ended up
like a lot of co-building withusers, like working closely with
them.
But from an infra perspective,what we did is we made sure the
Flink control plane handles allof the complexities around
(20:42):
deployments, failovers, rollingback clusters, etc.
Uh, like rolling backdeployments that didn't work,
etc.
And on the user side, we made itvery easy for them to actually
create a Flink app.
So we had like uh toolings thatcan generate all the boilerplate
for you so that it's easy forusers to actually just latch
(21:05):
onto a Kafka topic and consumeuh stream of data from that and
write flinks up.
And uh yeah, go ahead.
SPEAKER_01 (21:14):
So if if I'm a user and I want to create a Flink app
but I know nothing about Flink,um do I need to use Python or
Java, or do I have any specificrequirements of how I write my
code, or there's no code at all,maybe I can ask ChatGPT to
generate the code for me.
SPEAKER_02 (21:35):
That's a good question.
So we are getting there.
So right now, what we do is sowe adopt open is heavily, uh a
lot of services are in Python.
So we also adopted Pyflink.
So the initial version of thiswas a good amount of scaffolded
code that helps users to writeFlink apps.
The second version of it, whichwe are kind of going through
(21:57):
right now, is uh giving usersbetter tools.
To understand their flink jobs.
Like, for instance, coming upwith um like giving tools for
them to understand a healthscore behind their Flink app,
whether they're following allthe best practices and giving
them a score like 80 out of 100,you're losing 20 points because
(22:18):
of this, and these are thingsthat you need to do, etc.
Like you're managing your state.
Another thing we notice that weare currently doing is our Flink
state uh for some of these jobsare becoming very massive.
And when our users end up uhdebugging these apps, they need
to look into their state andinspect.
(22:40):
And actually, it's a very bigchallenge because uh once our
app, if you want to debug yourapp, you need to look into the
state.
And without looking into thestate, you're kind of blocked.
And some of and most of thesetimes, these states' files are
very massive and they're storedin top store.
So we are currently building astate file inspection tool.
So basically, you can debug,look at all the state that's
(23:01):
managed by your Think app,trying to understand what's
going wrong, how to optimizethings further.
Some of these, even subtle bugsin your code, end up creating
giant state files, even thoughyou don't intend to.
Uh, where we believe in the longterm, majority of link apps can
(23:25):
be represented through SQL orFinkSQL or a DSL.
So we're trying to carve thestructure of that DSL out and
have like a managed platformwhere users can just give in a
SQL or just codify their app ina DSL, transformations, joins,
aggregations in a format, andthen we manage the lifecycle of
the app behind the scenes.
(23:46):
So this is the no-code visionthat we have.
We can also integrate AI intoit.
You can have prompt engineering,like you can use natural
language to define your Flinkapp, your Flint app DSL, and
then you hit uh deploy and itgets deployed.
SPEAKER_01 (24:00):
Got it, got it.
So if let's say I'm more on anengineering side, but I'm on
more on the product side, I willbe able to write a SQL and also
have all the tools to inspectwhat is happening in my platform
in case uh you know performanceis not what I want, and I want
to improve it.
So I'll be able to actually havethat uh access to the data.
(24:24):
That's really interestingbecause one of the greatest
challenges with Flink is statemanagement.
And you know, so many people getit wrong for for good reasons
because you have to know so manythings in order to uh make the
right decisions, and it's justagain, like we cannot expect
everyone to know that.
So that's that's reallyexciting.
And the fact that you're usingPython as well with Flink, is
(24:47):
the infrastructure also inPython, or do you have the
infrastructure side and more ofJava world JVM?
SPEAKER_02 (24:53):
Yeah, so uh our infrastructure, at least for
data platform, all ofinfrastructure is mostly in
Java, but the end userabstractions are in Python.
So Kafka client is in Python.
SPEAKER_01 (25:05):
Oh, got it, got it.
Cool.
All right, so it's um yeah, it'show if if there is an error that
comes from the the Java's JVMstack, uh like how do you make
sure the user can understandthat there's uh I'm sure you
kind of you have the tools andyou you think through that, but
uh that's always uh you know oneof the challenges.
(25:27):
Like let's say I'm you know, Inever touched anything related
to JVM in my life, only didPython, um which is not true.
I I I was heavy in uh in the JVMspace, but let's let's pretend
for a second.
Um I wouldn't know what to dowith the Java exception, if I'll
get it.
SPEAKER_02 (25:46):
Yeah, so actually we have faced a problem even before
that.
Um sometimes we noticed um theactual error inside your Flink
app could be multiple layersdeep, could be in some different
file, some different place.
Uh and uh we once we face enoughproblems of this shape, we uh we
practice a workstam to basicallybuild a UI that can just tell
(26:10):
you exactly if there is a Flinkapp failure.
This is what uh uh so basicallydebugging your Flink apps,
getting the right exceptions.
And also uh we are trying to goahead with like uh we built an
AI agent that looks into theseexceptions and with enough
context about your app, it willkind of tell you what the right
next steps are and what weshould do.
(26:30):
Uh and you can keep uh there isa feedback loop here because if
it doesn't work, you just giveus feedback and then we try to
fix the agent.
But uh this is also somethingthat we built.
SPEAKER_01 (26:41):
That's awesome.
I love the usage of AI to assistin engineering work.
SPEAKER_02 (26:46):
Yeah, um I've seen like multiple examples.
Like I think AI agent uhdebugger is one example.
Uh AI assisted uh Flink appcreation through a custom DSL.
That's like another example.
Uh yeah, I'm seeing a lot ofexamples these days, making us
more productive.
SPEAKER_01 (27:04):
Yes, 100%.
It's um I see it how how itimproves everything that we that
we work on.
And um, you know, even it's kindof like uh having a body tells
you it's like, hey, you know,here and there you should take a
look because it might be thatthe error is is you know, here
are three potential reasons whywhy you got this error, start
(27:24):
exploring rather than you knowindividual trying to uh figure
it out on them on their own.
Um so it kind of helps bring thecollective brains of everyone in
the company to uh to solve aproblem without getting everyone
involved necessarily uh in ameeting or in a uh you know
gathering of other Slack orother tools.
(27:45):
So that's uh this is great.
This is great.
So you've you've been working,that's you've been solving a lot
of problems there.
SPEAKER_02 (27:54):
Yeah, um every day is exciting, a lot of scale, and
um a lot of different shapes ofproblems too.
So so uh as I said before, likeuh like solving problems with
the users because every um andtrying to find the right
abstractions that would actuallygeneralize to a more two
(28:14):
different use case shapes.
It's kind of what I do on adaily basis.
SPEAKER_01 (28:17):
That's awesome.
So if you would, you know, togive your younger self or
someone that you mentor that nowis entering engineering, just
you know, finish their degree orwant to enter the data streaming
world, what what would it tellthem?
SPEAKER_02 (28:32):
That's a good question.
Um, a couple of things.
Uh, one is I've noticed time andagain that simple systems scale
better.
You could have different notionsof uh any kind of like clever,
complex core, but uh ultimatelyin the long run, although if the
overall architecture is simpleto explain to others, if uh only
(28:55):
if that system is simple, itwill stand the test of time.
Learn this the hard way a coupleof times.
Uh and the other thing is uh youcould uh the long-term
reliability actually comes fromlike a lot of automation.
This goes back to the controlplanes point which I made
earlier.
Your initial wins andreliability could just come from
(29:17):
a nicely designed uh stack, butin the long term, you would uh
you would notice a lot of manualoperations, you would a lot of
routine operations, rebalancingclusters, provisioning them.
And so the long-term reliabilitywould just come from automation.
So building control planes,self-healing systems.
(29:40):
So these these are some lessonsthat I learned.
But uh understanding detailsbehind every system was actually
important.
It's actually more important tounderstand what it is for and
what it is not for, and gettingthat clarity because that is how
you build the right buildingblocks um it for which other
users can use and build greatsystems.
SPEAKER_01 (30:01):
Love it, love it.
So simplicity, uh understandingthe building blocks behind
things and building uh what'snecessary.
So sometimes you don't have touh over-engineer uh a solution.
Arvin, thank you so much forsharing the different things
that you work on and sharingyour knowledge and expertise.
Uh it was super exciting, and uhyeah, it's uh I'm sure people
(30:24):
would have lots of takeawaysfrom these.
SPEAKER_02 (30:28):
Yeah, thank you.
I mean nice chatting with you.
As long as we are able to build a system, it could
be uh abstraction that makes iteasy for our users, and as long
as it solves a problem, that'sgood.
I've noticed time and again thatsimple systems scale better, any
kind of like clever, complexcode.
Ultimately, in the long run,although if the overall
architecture is simple toexplain to others, will it stand
(00:20):
the test of time?
SPEAKER_01 (00:21):
Hello, I am Adi Polak, and you're listening to
Confluent Developer, where weexplore the fascinating journeys
of software developer tacklingcomplex problems.
In this episode, I'minterviewing Arveen Suresh from
his early days exploringdistributed systems to becoming
a key figure behind open AI,real-time infrastructure.
(00:46):
Arvine shares a story of scale,simplicity, and building the
right abstractions for AI.
His early passion for coding,starting back in school,
eventually led him into theworld of data streaming,
large-scale systems, andenterprise software.
So, what did OpenAI dodifferently to scale their data
(01:06):
streaming platform?
Have a listen.
Hi, Arvind.
I'm so excited to have you onour show.
SPEAKER_02 (01:19):
Hi, Andy.
Uh executive here.
Nice to meet you.
SPEAKER_01 (01:22):
Yeah, well, thank you.
First of all, thank you so muchfor being part of Corinth Nola
Program Committee.
Your insights and comments werefantastic and really helped us
build a good program.
And also, thank you for being aspeaker and sharing your
expertise.
I know, you know, behind thescenes, there's a lot of work
that you put into actuallytaking everything that you know
and turning it into apresentation that people can
(01:45):
learn from.
So thank you for doing all thesegreat things for the community.
SPEAKER_02 (01:49):
Yeah, thank you.
Excited to share our learningsas we uh like whatever we have
done uh at OpenAI.
So it was fun being part of theconference.
SPEAKER_01 (01:58):
That's that's fun.
So I definitely know you.
A lot of our program committee,and maybe some of the folks uh
from Corinth knows you, but noteveryone who are listening to us
today do.
So maybe you want to share alittle bit about yourself.
SPEAKER_02 (02:11):
Sure, yes.
Uh uh I'm Arvind.
I'm currently at OpenAI.
I lead the real-timeinfrastructure team.
So we power the streamingbackbone behind all AI products
like ChatGPT, Soda.
We also help a bunch of researchworkflows too.
So I've been doing distributedsystems and data platform for
like close to eight years now.
(02:32):
Uh and I've been like I've Ifound this space uh sometime.
I can tell you about that in alittle bit, but like the there
was a moment in my life when Ifigured out that distributed
systems was something that I wasreally passionate about, and
I've been like diving deeper, uhbuilding systems that scale
pretty much my entire career.
SPEAKER_01 (02:52):
That's amazing.
I mean, there's so much to knowabout distributed systems.
I always feel like it's it's awhole world that when you start
digging into it, when you're auser of a distributed system,
let's say you're a consumer ofan API, it's it's one angle.
You try to optimize, you have aproduct mindset.
But then when you dive deeperinto the infrastructure, it's
(03:15):
like, oh, things are notconsistent.
What do you do?
SPEAKER_02 (03:21):
Yeah, exactly.
Like uh, I think this is a goodpoint about infra and product,
right?
Like um, I've been part ofinfrastructure teams pretty much
my entire career, and it hasalways been uh I like working
with product teams or likedifferent teams in the company
because ultimately we are hereto may build the foundations
that make products successful.
(03:43):
So we're co-buildinginfrastructure with you.
So in early stages of the teamor the company, you end up
co-building your infrastructurewith the product team, and the
product teams will beco-building infrastructure and
their products with you.
So I really like thatexperience.
That is more fun, understandingtheir use case, partnering where
we can help.
SPEAKER_01 (04:01):
That's amazing.
So I know you've been, you know,you have many years' experience
with data streaming and kind ofdata streaming is is a big
passion and uh and theprofessionalism, you know,
career that you chose.
Um, I'm curious, like, what aresome of the challenges that
you've seen throughout yourcareers that, you know, and
learnings that are people canlearn from you?
SPEAKER_02 (04:22):
Sure.
Um, so let me start with like ummy recent experience at OpenAI.
So the challenge here wasbasically how do you build a
reliable streaming platform whenyou're scaling 10x every six,
seven months?
So your platform scales 10xevery six, seven months,
(04:43):
different sets of use cases,different shapes of workloads
that we support.
Now, how do you make sure uh theplatform is also reliable when
we do all of this, right?
The core challenges uh involvesomething like balancing
velocity and stability with likea such a small team, and how do
you work with users?
How do you hide all theinfrastructure complexities?
(05:05):
Um, so that just to give somecontext about uh OpenAI's use
cases, right?
The streaming infra is kind ofat the heart of OpenAI in like
different workflows.
Like uh one of the core thingsthat we use Stream Infra for is
something called Flywheel.
What that means is um we buildgreat AI models, then we launch
products using those AI models,then you get some usage data
(05:29):
about that product based on whatour users, how our users use
their products.
And insights from these areactually used to build better
models later.
And this kind of loop keepsspinning.
Like you have more models, youget better usage data, then you
get insights, you train bettermodels, and so on and so forth.
And guess what?
The data pipe here is Kafka.
So Kafka is used to store all ofthese events, and you use Plink
(05:52):
a lot of stream processing toget all this data as real time
as possible wherever it'sneeded, right?
Um, and also in all uhdistributed system settings,
let's say experimentation isheavily distributed today.
So the our researchers andengineers would like to know the
latest status of an experimentand see whether it makes sense
to continue that or not.
And today's uh training isheavily distributed, right?
(06:14):
So you have workers talking toeach other, and Kafka has the
data pipe to carry those events,and Flink is used to transform
and join that to basicallyunderstand what to do.
It's kind of bread and butter ofuh researchers and engineers.
And um, I've seen this inaction.
Like without streaming infra, itwas very hard for them to build
things.
With stream infra, it wasactually easier for them to do
their day-to-day.
(06:36):
So, in this background, um, thisis basically kind of the result,
right?
But this is not how it was whenI joined uh the company.
It was like early stages, uh, itwas still a scrappy uh piece of
infrastructure.
Like there was nostandardization around Kafka
clusters at that point.
Like there was no blessed paths.
(06:56):
Uh, Kafka was a single point offailure, uh, and there was a lot
of friction and confusion onlike how to use Kafka.
So this is the state there, andlike it was like close to a year
or two um years of work to getus to this shape.
SPEAKER_01 (07:10):
Wow.
So let me see, you know, collectall the all the pieces of
information.
Essentially, when anyone in theworld decides to use OpenAI
models, uh, we as humans cangive some sort of feedback, like
thumbs up, thumbs down, uh, orhave some sentiment to uh uh to
what we do.
Uh if we're happy, not happy,asking for uh improvements and
(07:32):
so on.
And that essentially becomes afeedback to um to for retraining
or rebuilding the models andimproving them so the the whole
system can become better.
And that's as a feedback turnsinto Kafka events that are being
processed later on with Flink ina streaming engine in real time.
(07:52):
And that's kind of like thebackbone for everything that you
do.
And then throughout, you know,as as you have these uh data
data streaming pipelines,essentially it's not serving
only that feedback loop with themodel, but actually, you know,
also researchers today that wantto run different
experimentation, or anyone whouh you know wants to tap into
(08:13):
what is happening in the systemneeds to have access to that
real-time data, essentially.
Right?
SPEAKER_02 (08:20):
Yes, that that is cut, yes.
SPEAKER_01 (08:22):
Got it, got it.
And then your team has to managethe cluster, right?
Uh manage the large scale ofthis distributed system.
Um you know, I'm guessing that'sthat means you know, a lot of
machines.
Um and then um make sure it runswith the right latency for the
(08:42):
system, right?
SPEAKER_02 (08:44):
Yes, yes, that that is correct.
The the emphasis on reliabilityis actually very high uh because
all of this is mission criticaldata, it's important to us.
So the we promise close to four,four, four and a half nines of
Kafka uh availability.
And also the core uh coreprinciple behind all of this was
(09:05):
uh building reliable systems wasof course important, but another
key principle that we kept inmind is how do we make all of
this easy for our users to use?
Because uh, when I say users, sowe are an infra team.
When I say users, I talk aboutall the other employees,
engineers, researchers at OpenAIwho would use all of this.
So, how do we make thatexperience easy for them?
(09:25):
So the ergonomics part was veryimportant.
Uh, it was like building AGI andbuilding AI systems is hard in
its own way.
People did not want to becomeKafka experts, right?
Uh so how do we build the rightabstractions?
Uh, we made some earlydecisions.
Like, for instance, instead ofexposing Kafka clusters directly
to users, we built layers ofabstractions.
(09:45):
So we built we built a bunch ofproxies surrounding our Kafka
clusters so that we can abstractKafka away from our users.
It helps us to do a lot ofoperations, scalability.
It also helps in uh multiplex.
So our Kafka topologies likeKafka messages get multiplexed
across different clusters anddifferent regions, and that
helps in higher availability.
(10:06):
That way, Kafka would not be asingle point of failure.
It does have some trade-offshere and there, but happy to add
details there.
But this actually simplified uhconsumption of Kafka and like
building workloads on top ofKafka easier for our users.
Because tomorrow we can scale acluster, we can add more nodes,
(10:27):
we can rebalance them, we canmove them from one region to the
other, but having a layer ofindirection in between actually
helped us make those it made allthose operations easier.
SPEAKER_01 (10:37):
Got it.
So if essentially, um just tosee if I understand correctly,
different people in theorganization, these are our
users, different people havedifferent skills and different
expertise.
Like ML researchers, they haveyou know probably the best
knowledge and on how to buildalgorithms and how to improve
them and so on.
Uh, and they have knowledge ofthe tools that they're using,
(10:58):
but we can't really ask them tolearn everything about Flink and
Kafka.
And so we want to give them uhtools that are uh intuitive to
what they do by building a layerthat uh abstract away Kafka and
Flink complexities uh so theycan run faster.
That's it, right?
(11:18):
Okay, got it.
Yes, and what are some of thechallenges that you ran into?
Because I can imagine, you know,people would have all kinds of
requests.
SPEAKER_02 (11:26):
Yes, yeah, that's a good question.
So our challenges here wasmainly around Kafka being a
single point of failure, butthat's actually a very hard
problem.
Like uh if you deploy, let'ssay, Kafka clusters in a single
region, and how do you make howdo you make that work in a
multi-region setup, right?
Like either you can go down thepath of having stretch clusters,
(11:48):
which I've seen in some cases,uh, where you have Kafka itself
distributed across regions.
Uh, there are obviously latencyconsiderations, stability at
scale considerations here.
Um, there are some other uhdesigns I've seen where you have
mirroring from one Kafka clusterto the other, so that in case a
region goes down.
So we were solving about caseswhere even the cases where
(12:10):
regions were going down, westill wanted Kafka to be up.
So even in those cases, um we wehave we basically had like a
bunch of these design options,which were all kind of complex.
Um we didn't know, we werethinking whether to do stretch
clusters or whether to domirroring.
And that's when we decided, canwe just make some
simplifications here?
(12:31):
We thought about uh, okay, wecan have built layers of
abstractions in between.
And let's say uh you multiplexmessages from one uh topic
across multiple Kafka clusters.
And when we do that, youobviously throw out some notions
of like partitioning, ordering,which are kind of very, very
important.
Like uh for folks who come froma traditional Kafka world, they
(12:52):
would be surprised, they wouldfind it odd that we threw away
these core primitives that Kafkabecame famous for, like
partitioning, ordering.
But we realized that a majorityof use cases uh were okay with
not having those guarantees.
They were okay with, let's say,for partitioning, they were okay
with doing that downstream,let's say the Flink app or the
downstream consumer.
And ordering, as you know, is ahard problem in distributed
(13:15):
systems itself, right?
So instead of relying onreliable ordering of events, why
can't you just stamp your eventswith a logical clock and use
that to uh sort in thedownstream?
So we made a bunch of these keytrade-offs and we went back to
our users and talked to them ifall these things make sense to
them.
And they were okay with it.
(13:35):
And that's when we actually wentand built our first version of
Kafka at OpenAI.
It was basically a producer-sideproxy which multiplexes messages
across multiple clusters, whichagain there is a consumer-side
proxy which pulls all of thisfor you and sends it to your end
consumer services.
So this was our first version.
(13:56):
This came in sometime aroundmid-2024.
SPEAKER_01 (13:59):
Amazing.
So essentially, it took a lot ofcomplexities out of Kafka and
some some a lot of uh failureguarantees, I will say.
So it's probably you know,coming.
I'm sure like the person thatcame up with that idea was
everyone looked at it and waslike, Who are you?
Do you know Kafka?
(14:20):
Um, but apparently, you know, itworks because at the end of the
day, you're serving a user, andthese users are okay with these
trade-offs.
SPEAKER_00 (14:28):
Now a quick word from our sponsor.
Confluent Developer the Podcastis brought to you by Confluent
Developer the Website, which haseverything you need as a
developer of data streamingsystems.
And it's completely free.
We've got curriculum, hands-onexercises, executable tutorials,
the online data streamingengineer certification, also
free, a way to find a meetupnear you, those are free.
(14:51):
Everything is there.
I really want you to besuccessful in your journey as a
data streaming engineer, andthis is the site that has what
you need.
Check it out atdeveloper.confluent.io.
That's developer.confluent.io.
Now back to the show.
SPEAKER_01 (15:07):
It's it definitely shows like a flexible mindset of
an engineering, because youknow, many times in our career,
and I my background is also inthe in the um distributed system
infrastructure.
There's like the things thatwe're used to do, the way we
forever did things.
And um being user-obsessed isdefinitely um a value that can
(15:31):
help us build systems that caterto our users versus trying to
answer all the best practices inthe world, I'll put it that way.
Um so that's that's exciting.
I mean, that's that's a biglearning.
SPEAKER_02 (15:44):
Yeah, I totally agree with what you said, right?
Like uh ultimately we arelooking, we are building systems
that solve problems.
Uh, it doesn't matter whether weuse a Kafka or a database or
whatever tools are going tohold.
Uh as long as we are able tobuild a system, it could be uh
an abstraction that makes iteasy for our users to use that
system.
And as long as it solves aproblem, that's good.
SPEAKER_01 (16:07):
Right, right, right.
No, at the end of that's that'swhat we're paid to do, right?
Solve the problem.
So I love it.
Cool.
Well, thank you for sharing.
That's um that's reallyexciting.
I know you build more things,right?
SPEAKER_02 (16:22):
Yes, uh, we built um so immediately after we shipped
this, there were more thissolved all the stateless
consumption from Kafka.
And then after that, uh therewere use cases around stateful
processing.
So that's when like we startedadopting Flink more at OpenAI.
Yeah, and also the some of theproblems are similar across
(16:44):
both.
Like, how do we make sure Flinkapps scale?
How do we make sure seamlesslywith this multi-cluster Kafka?
Like it's it's a differenttopology, right?
If you have Flink working with asingle Kafka cluster, there are
in uh everyone has been doingthat for years, but how would
you make it work with multipleKafka clusters?
What uh how do you we had tobuild uh a custom flink source
(17:08):
for it to work?
Then we had to build a customflink sync, then whether then we
had to invest in control planesbecause it um it's only a matter
of time.
This is also something that Iobserved in the last year or so.
It's only a matter of time untiluh the system that you build,
which is horizontally scalable,after a year or so, you end up
with like a lot of clusters, alarge split of clusters for you
(17:31):
to manage, and it ends up with alot of operational toil, on-call
pages, et cetera, which is whenwe realize the importance of
control panes and importance ofself-healing systems, auto
healing systems.
And that's kind of where uh wewent next.
So we introduced control planesfor all parts of an
infrastructure, including Kafka,our consumer side of Kafka,
(17:55):
Flink, all of these systemsended up having control planes
just to manage the scale ofclusters that we are dealing
with.
SPEAKER_01 (18:02):
Yeah, that's that's smart.
It's um from a certain scale,you have to have it.
I remember uh back, well, it wasuh when I was in Microsoft some
years back, but it was uhpost-acquisition of LinkedIn,
they were building um differentclass different Hadoop clusters,
(18:22):
and then there was a controlplane that managed a couple of
hundreds or thousands of uh miniHadoop clusters just to uh
support the scale.
And um yeah, I believe uh todaywith uh cell architecture, um it
is highly used.
This is uh what we have in inConfluent, they call it the Cora
(18:44):
engine.
There is a paper out there, uh,but we essentially we build cell
architecture so we can supportas many Kafka clusters as
needed.
SPEAKER_02 (18:53):
So I see what is a cell here?
SPEAKER_01 (18:56):
So the cell architecture is essentially it
can be a collection of um well,a Kafka cluster that supports a
collection of things in it.
So you can read about thearticle, it's we you know, it's
it requires it requires lookingat the architecture, but
essentially this is how uh AWS,for example, build S3.
(19:16):
So this is the architecturebehind that.
Um so it's a multi-tenancy,multi-tenancy environment that
keeps that takes care ofscaling, security, uh, and a lot
of other aspects.
And that was a great uh,especially in the recent outage
that was, um, confluence stayedup, which is important because
you don't want to put everythingin on one region.
(19:39):
Uh you want to continuesupporting our users.
Uh so I'm very happy you'rebuilding a control plane and you
have different control planesfor what you're doing.
Uh, and I'm curious because youknow, adding Flink is something
that you know you have to thinkabout.
Like, how do you how do yousimplify Flink for the rest of
the organization?
SPEAKER_02 (19:58):
Yeah, so it started off with building the right um
uh so ultimately, uh yeah,rightly said, Flink has its own
complexities.
Uh like it's it has its ownlearning curve, right?
It's not as easy, like it's easyto talk to Kafka and publish
messages, but it's not so easyto write a Flink app.
So it started off with um usworking closely with users to
(20:21):
explain Flink semantics to themand building um so most of the
projects that ended up havingcomplex Flink apps uh ended up
like a lot of co-building withusers, like working closely with
them.
But from an infra perspective,what we did is we made sure the
Flink control plane handles allof the complexities around
(20:42):
deployments, failovers, rollingback clusters, etc.
Uh, like rolling backdeployments that didn't work,
etc.
And on the user side, we made itvery easy for them to actually
create a Flink app.
So we had like uh toolings thatcan generate all the boilerplate
for you so that it's easy forusers to actually just latch
(21:05):
onto a Kafka topic and consumeuh stream of data from that and
write flinks up.
And uh yeah, go ahead.
SPEAKER_01 (21:14):
So if if I'm a user and I want to create a Flink app
but I know nothing about Flink,um do I need to use Python or
Java, or do I have any specificrequirements of how I write my
code, or there's no code at all,maybe I can ask ChatGPT to
generate the code for me.
SPEAKER_02 (21:35):
That's a good question.
So we are getting there.
So right now, what we do is sowe adopt open is heavily, uh a
lot of services are in Python.
So we also adopted Pyflink.
So the initial version of thiswas a good amount of scaffolded
code that helps users to writeFlink apps.
The second version of it, whichwe are kind of going through
(21:57):
right now, is uh giving usersbetter tools.
To understand their flink jobs.
Like, for instance, coming upwith um like giving tools for
them to understand a healthscore behind their Flink app,
whether they're following allthe best practices and giving
them a score like 80 out of 100,you're losing 20 points because
(22:18):
of this, and these are thingsthat you need to do, etc.
Like you're managing your state.
Another thing we notice that weare currently doing is our Flink
state uh for some of these jobsare becoming very massive.
And when our users end up uhdebugging these apps, they need
to look into their state andinspect.
(22:40):
And actually, it's a very bigchallenge because uh once our
app, if you want to debug yourapp, you need to look into the
state.
And without looking into thestate, you're kind of blocked.
And some of and most of thesetimes, these states' files are
very massive and they're storedin top store.
So we are currently building astate file inspection tool.
So basically, you can debug,look at all the state that's
(23:01):
managed by your Think app,trying to understand what's
going wrong, how to optimizethings further.
Some of these, even subtle bugsin your code, end up creating
giant state files, even thoughyou don't intend to.
Uh, where we believe in the longterm, majority of link apps can
(23:25):
be represented through SQL orFinkSQL or a DSL.
So we're trying to carve thestructure of that DSL out and
have like a managed platformwhere users can just give in a
SQL or just codify their app ina DSL, transformations, joins,
aggregations in a format, andthen we manage the lifecycle of
the app behind the scenes.
(23:46):
So this is the no-code visionthat we have.
We can also integrate AI intoit.
You can have prompt engineering,like you can use natural
language to define your Flinkapp, your Flint app DSL, and
then you hit uh deploy and itgets deployed.
SPEAKER_01 (24:00):
Got it, got it.
So if let's say I'm more on anengineering side, but I'm on
more on the product side, I willbe able to write a SQL and also
have all the tools to inspectwhat is happening in my platform
in case uh you know performanceis not what I want, and I want
to improve it.
So I'll be able to actually havethat uh access to the data.
(24:24):
That's really interestingbecause one of the greatest
challenges with Flink is statemanagement.
And you know, so many people getit wrong for for good reasons
because you have to know so manythings in order to uh make the
right decisions, and it's justagain, like we cannot expect
everyone to know that.
So that's that's reallyexciting.
And the fact that you're usingPython as well with Flink, is
(24:47):
the infrastructure also inPython, or do you have the
infrastructure side and more ofJava world JVM?
SPEAKER_02 (24:53):
Yeah, so uh our infrastructure, at least for
data platform, all ofinfrastructure is mostly in
Java, but the end userabstractions are in Python.
So Kafka client is in Python.
SPEAKER_01 (25:05):
Oh, got it, got it.
Cool.
All right, so it's um yeah, it'show if if there is an error that
comes from the the Java's JVMstack, uh like how do you make
sure the user can understandthat there's uh I'm sure you
kind of you have the tools andyou you think through that, but
uh that's always uh you know oneof the challenges.
(25:27):
Like let's say I'm you know, Inever touched anything related
to JVM in my life, only didPython, um which is not true.
I I I was heavy in uh in the JVMspace, but let's let's pretend
for a second.
Um I wouldn't know what to dowith the Java exception, if I'll
get it.
SPEAKER_02 (25:46):
Yeah, so actually we have faced a problem even before
that.
Um sometimes we noticed um theactual error inside your Flink
app could be multiple layersdeep, could be in some different
file, some different place.
Uh and uh we once we face enoughproblems of this shape, we uh we
practice a workstam to basicallybuild a UI that can just tell
(26:10):
you exactly if there is a Flinkapp failure.
This is what uh uh so basicallydebugging your Flink apps,
getting the right exceptions.
And also uh we are trying to goahead with like uh we built an
AI agent that looks into theseexceptions and with enough
context about your app, it willkind of tell you what the right
next steps are and what weshould do.
(26:30):
Uh and you can keep uh there isa feedback loop here because if
it doesn't work, you just giveus feedback and then we try to
fix the agent.
But uh this is also somethingthat we built.
SPEAKER_01 (26:41):
That's awesome.
I love the usage of AI to assistin engineering work.
SPEAKER_02 (26:46):
Yeah, um I've seen like multiple examples.
Like I think AI agent uhdebugger is one example.
Uh AI assisted uh Flink appcreation through a custom DSL.
That's like another example.
Uh yeah, I'm seeing a lot ofexamples these days, making us
more productive.
SPEAKER_01 (27:04):
Yes, 100%.
It's um I see it how how itimproves everything that we that
we work on.
And um, you know, even it's kindof like uh having a body tells
you it's like, hey, you know,here and there you should take a
look because it might be thatthe error is is you know, here
are three potential reasons whywhy you got this error, start
(27:24):
exploring rather than you knowindividual trying to uh figure
it out on them on their own.
Um so it kind of helps bring thecollective brains of everyone in
the company to uh to solve aproblem without getting everyone
involved necessarily uh in ameeting or in a uh you know
gathering of other Slack orother tools.
(27:45):
So that's uh this is great.
This is great.
So you've you've been working,that's you've been solving a lot
of problems there.
SPEAKER_02 (27:54):
Yeah, um every day is exciting, a lot of scale, and
um a lot of different shapes ofproblems too.
So so uh as I said before, likeuh like solving problems with
the users because every um andtrying to find the right
abstractions that would actuallygeneralize to a more two
(28:14):
different use case shapes.
It's kind of what I do on adaily basis.
SPEAKER_01 (28:17):
That's awesome.
So if you would, you know, togive your younger self or
someone that you mentor that nowis entering engineering, just
you know, finish their degree orwant to enter the data streaming
world, what what would it tellthem?
SPEAKER_02 (28:32):
That's a good question.
Um, a couple of things.
Uh, one is I've noticed time andagain that simple systems scale
better.
You could have different notionsof uh any kind of like clever,
complex core, but uh ultimatelyin the long run, although if the
overall architecture is simpleto explain to others, if uh only
(28:55):
if that system is simple, itwill stand the test of time.
Learn this the hard way a coupleof times.
Uh and the other thing is uh youcould uh the long-term
reliability actually comes fromlike a lot of automation.
This goes back to the controlplanes point which I made
earlier.
Your initial wins andreliability could just come from
(29:17):
a nicely designed uh stack, butin the long term, you would uh
you would notice a lot of manualoperations, you would a lot of
routine operations, rebalancingclusters, provisioning them.
And so the long-term reliabilitywould just come from automation.
So building control planes,self-healing systems.
(29:40):
So these these are some lessonsthat I learned.
But uh understanding detailsbehind every system was actually
important.
It's actually more important tounderstand what it is for and
what it is not for, and gettingthat clarity because that is how
you build the right buildingblocks um it for which other
users can use and build greatsystems.
SPEAKER_01 (30:01):
Love it, love it.
So simplicity, uh understandingthe building blocks behind
things and building uh what'snecessary.
So sometimes you don't have touh over-engineer uh a solution.
Arvin, thank you so much forsharing the different things
that you work on and sharingyour knowledge and expertise.
Uh it was super exciting, and uhyeah, it's uh I'm sure people
(30:24):
would have lots of takeawaysfrom these.
SPEAKER_02 (30:28):
Yeah, thank you.
I mean nice chatting with you.
Popular Podcasts
Betrayal Weekly
Betrayal Weekly is back for a new season. Every Thursday, Betrayal Weekly shares first-hand accounts of broken trust, shocking deceptions, and the trail of destruction they leave behind. Hosted by Andrea Gunning, this weekly ongoing series digs into real-life stories of betrayal and the aftermath. From stories of double lives to dark discoveries, these are cautionary tales and accounts of resilience against all odds. From the producers of the critically acclaimed Betrayal series, Betrayal Weekly drops new episodes every Thursday. If you would like to share your story, you can reach out to the Betrayal Team by emailing them at betrayalpod@gmail.com and follow us on Instagram at @betrayalpod and @glasspodcasts. Please join our Substack for additional exclusive content, curated book recommendations, and community discussions. Sign up FREE by clicking this link Beyond Betrayal Substack. Join our community dedicated to truth, resilience, and healing. Your voice matters! Be a part of our Betrayal journey on Substack.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com