March 9, 2026 • 34 mins
Adi Polak talks to Sage Pierce (Indeed) about his career in software engineering and event-driven architectures. Sage’s first job: Java Swing development at a Department of Defense–affiliated research lab. His challenge: working at Indeed on event-driven views and IMI to join data across domains in a polyglot microservices world.
Sage's Atleon project: https://github.com/atleon
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
SPEAKER_02 (00:00):
A lot of the hardest lessons that I had to learn had
to deal with quote unquotescaling myself up.
You have to learn to stop doingthings yourself.
Even though it's hard and I'msaying it's hard, you shouldn't
be afraid of it.
SPEAKER_00 (00:11):
Hello there everyone.
I'm Andipalak and welcome toConfluent Developer, where we
explored the wild, weird, andwonderful journeys of software
developers building systems thathopefully don't wake up at 3
a.m.
This episode I'm interviewingSage Pierce from hacking on Java
Swing apps inside a top secretdefense lab.
(00:35):
Yes, swing, it still exists, tobecoming a principal engineer
obsessed with domain-drivendesign.
Sage has seen things, monoliththings, microservices things,
page and go burr things.
His journey takes us fromcalling sidegs and governance
security clearance all the wayto event-driven architecture,
(01:00):
scaling through people, and eveninventing a new streaming
pattern like IMI, all whilediscovering the universal truth
of engineering.
If your name is in Gitplame,you're on call.
We talk growth, leadership,streaming data, and how to avoid
building yet another distributedheader ball.
(01:21):
Let's get to it.
SPEAKER_02 (01:31):
Hey Addie.
Yeah, excited to talk talk techand talk talk streaming.
SPEAKER_00 (01:35):
I know you've been recently promoted, so
congratulations.
Kind of like crossing the chasmin between uh staff engineer to
principal engineer.
It's it's a big deal.
Um I don't know if engineersknow it at home, but it's kind
of like when you reach a seniorlevel, it could it could be like
you know, a finite level forsome people, for some folks.
And getting anything beyondthat, it's definitely um um
(01:59):
very, very, you know, it showsthat one's is really eager to
learn and grow and uh do whatneeds to be done in order to
continue pushing forward.
So I'm curious if you have kindof you know learnings or things
you had to learn or you know,de-learn also forget, put it
that way.
SPEAKER_02 (02:17):
Yeah, uh honestly, all of the above.
I think every person's uh everyevery engineer's journey to the
levels of staff engineer andprincipal and then technical
fellow after that.
I know there are differentterminologies at companies.
But every every engineer'sjourney to those levels, I
think, is is unique.
Everybody has their ownexperiences and a unique story
(02:38):
to tell about that.
Um so what I what I can tell youis every everything everything
about my journey is very uhunique to me, I think.
Um I can tell you that I I'vebeen at I've I've been at this
job of software engineering.
I've I've I've been doing it forabout 15 years.
I got my uh start um in 2010.
(03:00):
Um and and since then, you know,the similar to all engineers,
it's just been a long series ofof lessons and learning and
growing.
And as far as the transitionfrom uh sort of a junior or
senior level into staff and andprincipal goes, uh for me, the
the lessons, a lot of thehardest lessons that I had to
(03:24):
learn had to deal with uh quoteunquote scaling myself up.
Um and that's that's a quotefrom uh one of my one of my
managers that um I still workwith.
Um he's not not currently mymanager, but he's somebody I
look up to a lot.
And um one of the first thingshe told me as I was getting
further on into my career wasyou have to learn to stop doing
(03:46):
things yourself.
I am I am an engineer that I Ireally, really enjoy getting
into the weeds, solving theproblem, not just not just
saying what the problem is andthen saying what a good solution
is.
I get really focused on if I ifI know what the right solution
is, I'm it it takes it it takesa lot of willpower to not just
(04:11):
go and do that thing.
Um but as I found out over time,as you as you build more and
more things, people startlooking to you as the person of
support.
You become the whether you likeit or not, empirically, you
become the first person thatgets contacted when something
you build built breaks, even ifyou've tried to hand that thing
(04:32):
off to another team.
Um you you are if if your nameis is in the get blame, you're
the you're the person that getsthat gets tagged first.
Uh so one of the hardest lessonsI've learned in the transition
from senior engineer to staffengineer to principal engineer
is is focusing less on doing thework, doing what I what I used
(04:55):
to consider the work, which iswriting the code, and more
focusing on how can I teach theengineers around me uh to do
that thing and and help coachthem up when they do something
different that uh differentlythan I would have done it.
Um and if I think they're thatthe way I would have done it is
(05:15):
better, I try to explain whythat is.
And sometimes I'm not alwaysright, and that's even another
opportunity for me to grow whenthere are disagreements and I
learn that you know the way thatI would have done it wasn't the
best way to do it, and thathappens as well.
So it's a it's it has a lot todo with learning from others and
teaching others um as as youmove up rather than just doing
(05:37):
the work yourself.
SPEAKER_00 (05:38):
Yeah, it's um you know, this is gonna stick with
me though for a while.
It's like scaling through othersand learning to scale yourself
in in the organization, knowingthat um our knowledge is finite.
We only have 24 hours a day, nomatter what we do.
Uh, you know, we can have otherresources, infinite other
resources, but time um is reallyas finite in uh yeah, so this is
(06:03):
a this is a really interestinglearning.
I wonder if you had likesomething distinctive between uh
staff and principal that you canthink of.
SPEAKER_02 (06:14):
Yeah, for sure.
I and so I obviously youmentioned I I was promoted last
year and that promotion happenedwhile I've while I've been at
Indeed.
Um so I've been at Indeed goingon five years now.
Um the the biggest difference tome between staff and principal
generically speaking is is thistransition from working, being a
(06:36):
tech lead for a group ofengineers, and instead being a
tech lead that knows how tocoordinate amongst different
groups of engineers in differentorganizations.
So I'll give I'll give you anexample.
This ties a little bit into uhthe talk I'm giving uh next
week.
Um I'm I'm a I'm sort of adomain-driven design junkie.
(06:57):
Um you can see some of mydomain-driven design books uh
behind me on my shelf.
Um and when I I usedomain-driven design and domains
in particular when talking aboutdifferent teams and what they're
responsible for and how thosedomains and those teams talk to
each other.
At indeed, uh, the current teamthat I'm a principal engineer
(07:18):
for has to deal with managingthe ingestion and management of
uh intents to hire.
So we we maintain the datastorage and data flow of how
employers say I'm I'm lookingfor a job to hire, or looking
for a person to hire for a job.
And then there are there areother domains that are
responsible for some data thatis completely orthogonal to the
(07:42):
hiring intent.
Um, that might be informationabout the companies themselves.
It might be other domains thatare responsible for deriving
data from other data thatemployers give us.
Um moving into the position ofprincipal engineer requires much
more of not just being a techlead for all of the services
(08:04):
that the subset of teams thatare in a particular domain are
working on, but thenunderstanding how that domain
relates to other domains thatother teams are responsible for.
Um and not just uh internalizingthe communication or the
necessities of communicationbetween those domains, but also
(08:25):
working with the members oneither team and either
organization that are that aredesigning and building out those
domains.
SPEAKER_00 (08:32):
Cool.
And also I'm guessing likefinding the dependencies and um
trying to unblock.
Yeah, there's it's fascinatinguh going from staff to
principal.
I'm sure it's a lot of work inuh Code of Steel for finding the
ways to uh to do it uheffectively efficiently.
SPEAKER_02 (08:49):
Yeah, I I do I do feel compelled to tell you a
part of my story in particular.
Um I did not want to be promotedto principal engineer for for
quite a while.
Um my manager at the time, uh Ithink we we first we first
broached the subject of beingpromoted from staff to principal
(09:10):
uh about two years before itactually happened for me.
Um and the I'll be honest, themain reason I didn't want to be
promoted was that I was I I knewthen, as I know now, that I very
much like doing the coding work,right?
Um but I I uh I saw my mycolleagues that were already
principals at that level, theyspent much more of their time in
(09:34):
meetings.
And I I was already spending asignificant percentage of my
time in meetings, and when Iwould see the calendars of my
principal engineer colleagues,and I would see 50, 60, 70
percent of their week blockedoff for meetings, that just
didn't appeal to me.
So growing into principal, um, Ihad to really get my manager and
(09:54):
uh uh the leadership apparatusto agree that if I if this was
going to happen for me, I wantedto do it at my own pace.
I wanted to uh maintain aconnection to the actual code
being delivered, uh, as well astaking on those extra
responsibilities of uh scalingmyself out through others.
SPEAKER_00 (10:15):
Yeah, you know, it's it's amazing to me how um we
have more authority to definethe way if we're willing to uh
first of all identify what arethe things that we we enjoy and
we want to do, and second thingbe able to communicate it in a
way that um other sides canrelate and understand.
And um it's um yeah, I think youdid that because you were able
(10:39):
to kind of shape uh your role inthe way that uh where you can
continue to enjoy it um and alsosatisfy the uh the rest of the
leadership and company needs, uhput it that way.
So congrats, that's superexciting.
Um I you know, we're here totalk about challenges, we're
(11:00):
here to talk about work.
Uh but I was curious, do youremember your first ever job?
SPEAKER_02 (11:07):
Oh yeah, absolutely.
Um yeah, so I mentioned I got Igot started about 15 years ago.
So 15 years ago, um I I got ajob while I was in college.
Um I was at the University ofTexas.
Um go Longhorns.
I'm a big Longhorn fan.
You can see my Gilder Bear hasgot a longhorn on it.
Um so I I had I knew when I wasin college that I didn't really
(11:33):
know for certain in college thatI was gonna be a software
engineer when I got out.
Um my major was electrical andcomputer engineering.
I was really into renewableenergy, um, but also
microprocessing.
Um so the programming I wasdoing was really kind of systems
level, uh, embedded processinglevel uh in school.
So I didn't really uh I didn'tknow that I would be real into
(11:55):
high-level programming, kind ofwhat I'm doing now with JV and
languages and whatnot.
Um but I I uh knew I needed toget a job because I was I was
low on cash and I was a poorcollege student.
Um and I applied for an openposition at a research lab on a
research campus affiliated withUT.
(12:16):
And the name of that researchlab was called Applied Research
Laboratories.
And they they hired me as uhinitially as a software
technician.
Um that was just a fancy way ofsaying I was a uh a student uh
that was being paid part-time touh work at the research lab.
Um and this research lab hadfederal contracts uh with the
(12:39):
Department of Defense.
So there's a little bit about mywork that I can't that I backed
in that I can't, I'm notsupposed to talk about, even
though it was like more than 10years ago.
And I'm pretty sure I don't Idon't think I was really working
on anything that was classified,but I had to go get a they made
me get a government clearanceand whatnot.
Um, but that was just because Iwas around classified things.
(13:00):
Um, but that job uh was I didn'tknow it at the time, um, but
that job lacked every singlebell and whistle related to
software engineering that youcould imagine.
It being a research lab that wasaffiliated with uh or that had
federal contracts and needed tobe security compliant meant that
(13:22):
they were very, very picky aboutthe the tools and technologies
that they allowed you to use.
Um so my my first developmentenvironment was a bare bones
Linux uh laptop, or not laptop,it was a it was a tower actually
at first, and they gave me alaptop.
Um and that uh computer, Iwasn't I I wasn't allowed to
(13:45):
take that computer home.
I had to work on it, I had tostay at the office.
Um and my coding environment wasthere was no IDE, it was a a
shell script and a uh graphicaltext editor called GEdit.
(14:06):
And I had to learn, I I had tolearn a lot of things that I
take for granted now, like howhow do I compile Java locally,
um, how do I run thatapplication?
And the applications I wasworking on, and I don't know if
this isn't sensitive oranything, but I was working on
applications that helped uhhelped uh defense contractors
(14:26):
monitor uh the state of our GPSsatellites.
Um so they were they were Javaapplications written with swing.
Um and I believe swing is stillavailable as uh an API in Java.
I don't think anybody uses it.
I don't think they should.
But everything I was working onat the time had to do with Java
(14:47):
Swing development.
SPEAKER_00 (14:48):
Yeah, maybe those you know, Java jobs that you
created are still around andrunning somewhere, monitoring
everything.
I can only use it.
If it works, you know, don'tchange it unless there are new
requirements.
Um but that that's that's reallysuper cool how you did the
transition from uh from thatworld into software and kind of
working on uh yeah, working umalso working at the governance.
(15:14):
It's always interesting becausethere's so many cool projects
that unfortunately people cannotshare.
Uh but a lot of very coolthings.
SPEAKER_01 (15:22):
Now a quick word from our sponsor.
Confluent developer the podcastis brought to you by Confluent
Developer the website, which haseverything you need as a
developer of data streamingsystems.
And it's completely free.
We've got curriculum, hands-onexercises, executable tutorials,
the online data streamingengineer certification, also
free, a way to find a meetupnear you, those are free.
(15:45):
Everything is there.
I really want you to besuccessful in your journey as a
data streaming engineer, andthis is the site that has what
you need.
Check it out atdeveloper.confluent.io.
That's developer.confluent.io.
Now back to the show.
SPEAKER_00 (16:01):
Okay, let's do a jump forward.
There's a lot of, you know,challenges, software challenges,
architecture challenges, um, oras you mentioned, like domain,
how do you even define likeowners?
How do you split it?
How do you how do you create theright interface and so on?
Um maybe you want to share somerecent challenge that you worked
on or you're still working onand some learnings around that.
SPEAKER_02 (16:23):
Yeah, so the it's this is I I I'm gonna take this
question, I'm gonna uh move it,bleed it into kind of what this
this overarching problem thathas permeated the last 10 or so
years of my professional life.
And that challenge, I I refer tothis challenge as event-driven
views.
(16:44):
And the the the setup for thischallenge is is pretty I I my
guess is it's pretty um uhcommon to a lot of people that
have been in tech for anysignificant amount of time,
especially any anybody that'sworked at companies um that
started with large monolithicstructures and found out that it
doesn't scale well.
(17:05):
Um so uh the the problemstatement starts is that starts
like this.
Most really successful techproducts, software products, at
least prior to five years agoand in the previous couple of
decades, they usually startedreally quickly with somebody
creating a database with a bituh a nice schema and then
(17:27):
setting up a service in front ofit and then demonstrating value
and then launching that toproduction.
And then as they wanted to addnew things, they would just add
new things to that service, addmore data to the database, and
then as the as the product mighthave grown, they would have had
people, other internal uses ofconsuming that database to run
things like analytics.
(17:48):
Maybe they attach some changedata capture to the database so
that they get real-time uh datapropagation downstream.
By and large, what you find outis if the company has a long
life, has a has a longer thannon-trivial life and they keep
building on that initialstructure, what they end up with
is a giant mess of a databaseand and monolithic service and
(18:12):
monolithic architecture thatbecomes difficult to work on,
right?
We've we've heard uh monolithmight as well be a four-letter
word in the world of uh softwareengineering and software
architecture.
Um I think everybody's familiarwith with the pain that those
kind of monoliths can solve orcan uh induce.
(18:34):
Um so what ends what ends uphappening by and large, and this
was really popular throughoutthe 2010s while I was uh
progressing from junior, senior,uh, and a staff software
engineer, uh, was the approachto what the the approach for a
solution for monolithicarchitectures is start breaking
it down into microservices.
(18:55):
So microservices, in my view,should imply uh at least a
little bit of knowledge andbackground on domain-driven
design, because you can't havemicroservices kind of spawn from
the world of domain-drivendesign.
Um so uh uh having having anunderstanding of domain-driven
design and uh leading thatleading into creating
(19:17):
microservices was prettyimportant for me.
Um and I was fortunate at my atmy second job that I had, uh
they were uh pretty adamant thatthey spend a good amount of
resources on up leveling theirengineers.
And one of the topics that wewould talk about uh was
domain-driven design.
So I had a I had a pretty gooduh good academic start on
(19:39):
learning about why why you breakup monoliths into microservices
and how you use domain-drivendesign to do that.
Um and it wasn't too long afterthat that I stumbled into this
problem that kept on coming upof okay, we've broken that we've
broken this monolith intomicroservices using
(20:00):
domain-driven design.
And now the problem is we haveuse cases like analytics, or we
have use cases that are domainsthat are span other domains that
we've identified.
We need we need a domain thataggregates data from domain A
and domain B, and we need tokeep an entity uh up to date in
(20:25):
real time based on changes tothese other domains.
It used to be the case if youhad a monolithic architecture,
all that data was in one place.
You could just, it was easy,create create that view off of
the data that spans multiplebounded contexts, it's all
munged together in the samedatabase anyway.
Seems like a prettystraightforward thing to do.
(20:45):
Obviously, you can't really dothat with uh with uh
microservice architecture.
And and even doing that even ina microservice architecture
implies you have some form of anevent-driven, uh an event-driven
design baked into yourarchitecture as well.
Because how can you, how doesany downstream domain know about
(21:07):
data changes in an upstreamdomain if that domain isn't
producing events?
SPEAKER_00 (21:12):
Right.
SPEAKER_02 (21:13):
So we so we've already talked about like three
three pretty large concepts thathave entire books dedicated to
them domain-driven design,microservice architecture, and
event-driven, event-drivendesign.
Um and the those all those alllead to this, are all a
component of this problem thatkept coming up of we need to be
(21:34):
able to do something interestingwith a view of data that spans
multiple bounded contexts.
That could be analytics, itcould be indexing into a search
engine for multivariable search,it could be its own domain that
other people want toincrementally operate on, but
keep that data replicated fromthe other, keep certain data
(21:56):
replicated from other domains.
There's a whole wide gamut ofuse cases.
SPEAKER_00 (22:03):
Let me just see if I can I can I f I follow up
everything you said.
So essentially you you have uhyou broke into Monolith, right?
Uh you turn it intomicroservices.
SPEAKER_02 (22:12):
Each one of them has their own you know streaming
data from place to place and umuh serving uh whatever
requirements uh they areresponsible and and owns um and
suddenly you get morerequirements that are now more
kind of like data focused likeyou have analytics you have
different services that needs toaggregate uh information right
(22:33):
and this is where kind of likenew challenges uh appears in in
the system okay cool that'sexactly right and the the extra
compounding factors that make itmore that make it more difficult
is you find out that if when youone of the benefits of
microservices and domain drivendesign is you can separate
(22:54):
things but as those thingsbecome owned by different teams
those teams have differentpreferences one team might want
to use a relational database orit might make sense for their
their domain architecture to usea non-relational database.
Another team might be usingKafka to produce their events
but they might be using an AWSnative solution so they might be
producing events to SQS.
(23:16):
Or and one of the things that II find really common is that
given that Kafka is rel is isrelatively young compared to
message buses in general thesome of the older companies and
tech stacks will use somethinglike RabbitMQ or some uh any any
older uh message bustechnologies and what the the
(23:39):
compounding factor there is wellany solution for an event driven
view that you need has to beable to account for this
polyglot infrastructuredifferent databases different
message buses and whatnot isdifferent right so the tech
stack is different so what whatwhat's what do we do?
(24:00):
Yeah absolutely so um this thethe the nature of a polyglot
architecture and then the thenature of the eventual
consistency of an event of anevent driven system those all
lead to a unique set of problemsif you're trying to solve this
this come up with a solution forthis abstract need to come up to
(24:20):
see a view an up-to-datereal-time view um of the data
that you care about that needsto be joined across domains um
one of the things I touch on andI'll touch on in my talk next
week is this sounds reallysimilar to like GraphQL right to
like query federation I havedata across multiple domains
that I need to join on demandand and be able to view that
(24:43):
data.
Where event driven views differsfrom something like query
federation or GraphQL is thatevent driven views is much more
backend oriented.
You want to you you know theexact services you need to hit
so it's not it's not it's notexactly federated and there's a
lot of throughput and latencyconstraints that you'd like to
(25:05):
account for that aren'tnecessarily addressed in
GraphQL.
And in addition not justquerying the data but notifying
your downstream consumers whenthat view that you're keeping
has changed becomes not justdesirable but a requirement in a
lot of cases because a lot ofdomains will care about when the
join data changes not just theindependent data from the
(25:27):
upstream domains change.
So that is it's a long-windedway to explain the the entire
domain of this uh this problemof event driven views um which
is what I've been working oncoming up with I've come up with
helped come up with severalsolutions um in the in the past
uh decade or so um and the oneI'm presenting on next week has
(25:49):
to do with this uh streamtopology that I that I have
termed ingest materialize indexum so that that is the that is
in the name of my presentationnext week which um I shortened
that to IMI um and IMI is allabout coming up with a simple a
maximally simple solutionattempting to be maximally uh
(26:13):
simple uh solution uh for eventdriven views cool can you give
us like a brief uh you know thisis how we build it absolutely
yeah so uh uh IMI ingestmaterialize index it's I name it
that because it's based off ofthese three uh three stream
stage each each of those wordsmaps to a stream stage so the
(26:37):
way IMI starts off is you ingestevents from any any uh producers
of events and those those eventsshould indicate um typically
ideally they should be domainevents but not everybody has
those sometimes they're just rawCDC events but these events
should indicate when data thatyou might care about viewing has
(26:59):
changed so that ingest stage isresponsible for consuming
whatever the raw schema isupstream that you don't control
because the producers controlthat schema consuming that those
events and then using the datain those events to relate the
foreign data the foreign keys inthat data to the view keys that
(27:21):
need to be reprocessed orrematerialized.
So in ingestion you have thisresponsibility of correlation
and then ultimately once thatcorrelation is done you produce
commands.
So this is I'm CQRS commandquery responsibility segregation
is another concept that's bakedinto this not just the problem
(27:43):
but also the solution.
So ingestion is responsible forcorrelating the changes in
foreign domains to the domain ofyour view and then generating
commands to reprocess that view.
That feeds into the second stageof IMI materialization and in
materialization what happens isyou take these commands and in
(28:08):
our in all of our cases we applymicro batching um which is take
in elements or take t durationof commands whichever one comes
first and then you take thatbatch of commands and you
dereference as usually the wordthat we call we dereference all
of the data that's uh related tothe view that you want to
(28:32):
generate and then we feed thatblob that aggregated blob of raw
data to a generation phase andthen that generation phase
produces all the views that needthe current view the current
state of all the views that needto be indexed and those get put
onto another Kafka topic andthat feeds into the the third
(28:52):
stage which is index so each ofthose views can be indexed to
any number of destinationsincluding a database where you
can look up the data on demandor you can have CDC from that
database which we also do whichlets you let your downstream
clients know when that data ischanged.
SPEAKER_00 (29:12):
Cool.
SPEAKER_02 (29:13):
So I'm guessing like you know thinking through there
are some requirements that thissystem has to comply to like I'm
thinking maybe um exactly onceright if anything changes you
know there's some command oroperator something happened to
the data and we are sendingthose events as changes we need
to make sure it actually arrivesat the destination right yeah so
(29:35):
it it ends up being the case uhwith all of our use cases that
we don't care about exactly oncewe just care about at least once
um so that that is our minimumlevel of guarantee you will get
you will get these changes willflow through the system at least
once um the the other thing Iwanted to call out um before we
(29:58):
move on to uh before we move onto the next thing is the the
architecture itself I'm I'm oneof the things I'm happiest about
with it it's number one it'sabstract so it can be applied to
several different uh streamingmethodologies um but it requires
a very minimum set of resourceswhich is one of our requirements
we don't want to we would liketo not have to replicate our
(30:21):
data all over the place just tomake it compatible with a given
tool um and the solution the thebare solution that we that uh
IMI proposes is a singleapplication with at least three
stream processes in it and thentwo Kafka topics so that
absolutely minimizes the numberof resources you get we we get
(30:44):
at least once processingespecially using Kafka as uh the
intermediary between the streamstages um and we get robustness
um because we know we won't uhwe have we have uh a library
that we use it then this is alsoa part of my presentation um a
bit of a passion project thatI've been working on called
(31:05):
Atleon um and that's a it's alightweight stream processing
framework that gives you theguarantee of at least once
processing regardless of whatinfrastructure you're consuming
from or producing to that'sreally cool well I'm very I'm
very excited for your talk andif anyone is watching it
afterwards it's gonna beavailable on YouTube as well so
we can always catch up with withthat um I'm curious is the is
(31:28):
this library is ever going to beopen sourced or do you think
about that?
Yeah it is open source oh it isopen source yep you can check it
out at atleon.io very very cooland the the name the name is uh
comes from at least once at Leonum so that's uh I've been it's
(31:48):
it's been a project that I'vebeen working on for going on
through might be four years nowwow okay that's uh I mean some
of the best projects are theones that we work on for a long
time in my opinion so very coolall right any like learning so
far that you want to sharebefore we're wrapping things up
(32:11):
from you know thinking about thesystem designing architecture
pitching uh the system alsogetting buy-ins from some folks
as well I'm sure you've beenworking you know across the
company with multiple teams Ithink my biggest lesson with
with with all all of this stuffconcerning uh not just event
driven views but but streamprocessing in general it's super
(32:34):
super powerful but there therethere is a pretty big learning
curve when you're going everyeverybody starts out their their
first project is uh or theirfirst like business project is
usually create a a rest endpointor maybe a grpc endpoint and it
it's really intuitive to thinkrequest response right moving
(32:56):
into the world and and this isthe challenge that I I face a
lot when designing asynchronoussystems and teaching other
engineers about asynchronous uhsystems and stream processing is
you don't really know what'sgonna happen after you fire an
event off.
You know you you agree to acontract and you produce it and
then the things that happenafter that not only not only are
(33:19):
you not really sure anybody'sgonna do anything after you fire
that event off but it some ofthe things that that happen
might be non-intuitive.
So it it's it's definitely I Ifind asynchronous processing to
be extremely powerful a very funproblem space um but I but a
(33:39):
pretty a pretty high learningcurve but a learning curve
that's definitely worth itbecause it it definitely moves
you to the the next level ofbeing able to think and design
systems that scale and eventhough even though it's hard and
I'm saying it's hard shouldn'tbe afraid of it.
Dive in and uh learn learn whatit's all about.
SPEAKER_00 (34:00):
Yeah just do it it's the future cool sage thank you
so much for joining me today andsharing all your expertise and
again congrats on the promotionand uh you know if this uh
people watching after currentwill share the link uh and as
well to the YouTube uh videothat will be uploaded after the
event so everyone can uh youknow go back and and listen and
(34:23):
learn uh from the recordings.
SPEAKER_02 (34:26):
Sounds good.
SPEAKER_00 (34:27):
And of course it was a pleasure
A lot of the hardest lessons that I had to learn had
to deal with quote unquotescaling myself up.
You have to learn to stop doingthings yourself.
Even though it's hard and I'msaying it's hard, you shouldn't
be afraid of it.
SPEAKER_00 (00:11):
Hello there everyone.
I'm Andipalak and welcome toConfluent Developer, where we
explored the wild, weird, andwonderful journeys of software
developers building systems thathopefully don't wake up at 3
a.m.
This episode I'm interviewingSage Pierce from hacking on Java
Swing apps inside a top secretdefense lab.
(00:35):
Yes, swing, it still exists, tobecoming a principal engineer
obsessed with domain-drivendesign.
Sage has seen things, monoliththings, microservices things,
page and go burr things.
His journey takes us fromcalling sidegs and governance
security clearance all the wayto event-driven architecture,
(01:00):
scaling through people, and eveninventing a new streaming
pattern like IMI, all whilediscovering the universal truth
of engineering.
If your name is in Gitplame,you're on call.
We talk growth, leadership,streaming data, and how to avoid
building yet another distributedheader ball.
(01:21):
Let's get to it.
SPEAKER_02 (01:31):
Hey Addie.
Yeah, excited to talk talk techand talk talk streaming.
SPEAKER_00 (01:35):
I know you've been recently promoted, so
congratulations.
Kind of like crossing the chasmin between uh staff engineer to
principal engineer.
It's it's a big deal.
Um I don't know if engineersknow it at home, but it's kind
of like when you reach a seniorlevel, it could it could be like
you know, a finite level forsome people, for some folks.
And getting anything beyondthat, it's definitely um um
(01:59):
very, very, you know, it showsthat one's is really eager to
learn and grow and uh do whatneeds to be done in order to
continue pushing forward.
So I'm curious if you have kindof you know learnings or things
you had to learn or you know,de-learn also forget, put it
that way.
SPEAKER_02 (02:17):
Yeah, uh honestly, all of the above.
I think every person's uh everyevery engineer's journey to the
levels of staff engineer andprincipal and then technical
fellow after that.
I know there are differentterminologies at companies.
But every every engineer'sjourney to those levels, I
think, is is unique.
Everybody has their ownexperiences and a unique story
(02:38):
to tell about that.
Um so what I what I can tell youis every everything everything
about my journey is very uhunique to me, I think.
Um I can tell you that I I'vebeen at I've I've been at this
job of software engineering.
I've I've I've been doing it forabout 15 years.
I got my uh start um in 2010.
(03:00):
Um and and since then, you know,the similar to all engineers,
it's just been a long series ofof lessons and learning and
growing.
And as far as the transitionfrom uh sort of a junior or
senior level into staff and andprincipal goes, uh for me, the
the lessons, a lot of thehardest lessons that I had to
(03:24):
learn had to deal with uh quoteunquote scaling myself up.
Um and that's that's a quotefrom uh one of my one of my
managers that um I still workwith.
Um he's not not currently mymanager, but he's somebody I
look up to a lot.
And um one of the first thingshe told me as I was getting
further on into my career wasyou have to learn to stop doing
(03:46):
things yourself.
I am I am an engineer that I Ireally, really enjoy getting
into the weeds, solving theproblem, not just not just
saying what the problem is andthen saying what a good solution
is.
I get really focused on if I ifI know what the right solution
is, I'm it it takes it it takesa lot of willpower to not just
(04:11):
go and do that thing.
Um but as I found out over time,as you as you build more and
more things, people startlooking to you as the person of
support.
You become the whether you likeit or not, empirically, you
become the first person thatgets contacted when something
you build built breaks, even ifyou've tried to hand that thing
(04:32):
off to another team.
Um you you are if if your nameis is in the get blame, you're
the you're the person that getsthat gets tagged first.
Uh so one of the hardest lessonsI've learned in the transition
from senior engineer to staffengineer to principal engineer
is is focusing less on doing thework, doing what I what I used
(04:55):
to consider the work, which iswriting the code, and more
focusing on how can I teach theengineers around me uh to do
that thing and and help coachthem up when they do something
different that uh differentlythan I would have done it.
Um and if I think they're thatthe way I would have done it is
(05:15):
better, I try to explain whythat is.
And sometimes I'm not alwaysright, and that's even another
opportunity for me to grow whenthere are disagreements and I
learn that you know the way thatI would have done it wasn't the
best way to do it, and thathappens as well.
So it's a it's it has a lot todo with learning from others and
teaching others um as as youmove up rather than just doing
(05:37):
the work yourself.
SPEAKER_00 (05:38):
Yeah, it's um you know, this is gonna stick with
me though for a while.
It's like scaling through othersand learning to scale yourself
in in the organization, knowingthat um our knowledge is finite.
We only have 24 hours a day, nomatter what we do.
Uh, you know, we can have otherresources, infinite other
resources, but time um is reallyas finite in uh yeah, so this is
(06:03):
a this is a really interestinglearning.
I wonder if you had likesomething distinctive between uh
staff and principal that you canthink of.
SPEAKER_02 (06:14):
Yeah, for sure.
I and so I obviously youmentioned I I was promoted last
year and that promotion happenedwhile I've while I've been at
Indeed.
Um so I've been at Indeed goingon five years now.
Um the the biggest difference tome between staff and principal
generically speaking is is thistransition from working, being a
(06:36):
tech lead for a group ofengineers, and instead being a
tech lead that knows how tocoordinate amongst different
groups of engineers in differentorganizations.
So I'll give I'll give you anexample.
This ties a little bit into uhthe talk I'm giving uh next
week.
Um I'm I'm a I'm sort of adomain-driven design junkie.
(06:57):
Um you can see some of mydomain-driven design books uh
behind me on my shelf.
Um and when I I usedomain-driven design and domains
in particular when talking aboutdifferent teams and what they're
responsible for and how thosedomains and those teams talk to
each other.
At indeed, uh, the current teamthat I'm a principal engineer
(07:18):
for has to deal with managingthe ingestion and management of
uh intents to hire.
So we we maintain the datastorage and data flow of how
employers say I'm I'm lookingfor a job to hire, or looking
for a person to hire for a job.
And then there are there areother domains that are
responsible for some data thatis completely orthogonal to the
(07:42):
hiring intent.
Um, that might be informationabout the companies themselves.
It might be other domains thatare responsible for deriving
data from other data thatemployers give us.
Um moving into the position ofprincipal engineer requires much
more of not just being a techlead for all of the services
(08:04):
that the subset of teams thatare in a particular domain are
working on, but thenunderstanding how that domain
relates to other domains thatother teams are responsible for.
Um and not just uh internalizingthe communication or the
necessities of communicationbetween those domains, but also
(08:25):
working with the members oneither team and either
organization that are that aredesigning and building out those
domains.
SPEAKER_00 (08:32):
Cool.
And also I'm guessing likefinding the dependencies and um
trying to unblock.
Yeah, there's it's fascinatinguh going from staff to
principal.
I'm sure it's a lot of work inuh Code of Steel for finding the
ways to uh to do it uheffectively efficiently.
SPEAKER_02 (08:49):
Yeah, I I do I do feel compelled to tell you a
part of my story in particular.
Um I did not want to be promotedto principal engineer for for
quite a while.
Um my manager at the time, uh Ithink we we first we first
broached the subject of beingpromoted from staff to principal
(09:10):
uh about two years before itactually happened for me.
Um and the I'll be honest, themain reason I didn't want to be
promoted was that I was I I knewthen, as I know now, that I very
much like doing the coding work,right?
Um but I I uh I saw my mycolleagues that were already
principals at that level, theyspent much more of their time in
(09:34):
meetings.
And I I was already spending asignificant percentage of my
time in meetings, and when Iwould see the calendars of my
principal engineer colleagues,and I would see 50, 60, 70
percent of their week blockedoff for meetings, that just
didn't appeal to me.
So growing into principal, um, Ihad to really get my manager and
(09:54):
uh uh the leadership apparatusto agree that if I if this was
going to happen for me, I wantedto do it at my own pace.
I wanted to uh maintain aconnection to the actual code
being delivered, uh, as well astaking on those extra
responsibilities of uh scalingmyself out through others.
SPEAKER_00 (10:15):
Yeah, you know, it's it's amazing to me how um we
have more authority to definethe way if we're willing to uh
first of all identify what arethe things that we we enjoy and
we want to do, and second thingbe able to communicate it in a
way that um other sides canrelate and understand.
And um it's um yeah, I think youdid that because you were able
(10:39):
to kind of shape uh your role inthe way that uh where you can
continue to enjoy it um and alsosatisfy the uh the rest of the
leadership and company needs, uhput it that way.
So congrats, that's superexciting.
Um I you know, we're here totalk about challenges, we're
(11:00):
here to talk about work.
Uh but I was curious, do youremember your first ever job?
SPEAKER_02 (11:07):
Oh yeah, absolutely.
Um yeah, so I mentioned I got Igot started about 15 years ago.
So 15 years ago, um I I got ajob while I was in college.
Um I was at the University ofTexas.
Um go Longhorns.
I'm a big Longhorn fan.
You can see my Gilder Bear hasgot a longhorn on it.
Um so I I had I knew when I wasin college that I didn't really
(11:33):
know for certain in college thatI was gonna be a software
engineer when I got out.
Um my major was electrical andcomputer engineering.
I was really into renewableenergy, um, but also
microprocessing.
Um so the programming I wasdoing was really kind of systems
level, uh, embedded processinglevel uh in school.
So I didn't really uh I didn'tknow that I would be real into
(11:55):
high-level programming, kind ofwhat I'm doing now with JV and
languages and whatnot.
Um but I I uh knew I needed toget a job because I was I was
low on cash and I was a poorcollege student.
Um and I applied for an openposition at a research lab on a
research campus affiliated withUT.
(12:16):
And the name of that researchlab was called Applied Research
Laboratories.
And they they hired me as uhinitially as a software
technician.
Um that was just a fancy way ofsaying I was a uh a student uh
that was being paid part-time touh work at the research lab.
Um and this research lab hadfederal contracts uh with the
(12:39):
Department of Defense.
So there's a little bit about mywork that I can't that I backed
in that I can't, I'm notsupposed to talk about, even
though it was like more than 10years ago.
And I'm pretty sure I don't Idon't think I was really working
on anything that was classified,but I had to go get a they made
me get a government clearanceand whatnot.
Um, but that was just because Iwas around classified things.
(13:00):
Um, but that job uh was I didn'tknow it at the time, um, but
that job lacked every singlebell and whistle related to
software engineering that youcould imagine.
It being a research lab that wasaffiliated with uh or that had
federal contracts and needed tobe security compliant meant that
(13:22):
they were very, very picky aboutthe the tools and technologies
that they allowed you to use.
Um so my my first developmentenvironment was a bare bones
Linux uh laptop, or not laptop,it was a it was a tower actually
at first, and they gave me alaptop.
Um and that uh computer, Iwasn't I I wasn't allowed to
(13:45):
take that computer home.
I had to work on it, I had tostay at the office.
Um and my coding environment wasthere was no IDE, it was a a
shell script and a uh graphicaltext editor called GEdit.
(14:06):
And I had to learn, I I had tolearn a lot of things that I
take for granted now, like howhow do I compile Java locally,
um, how do I run thatapplication?
And the applications I wasworking on, and I don't know if
this isn't sensitive oranything, but I was working on
applications that helped uhhelped uh defense contractors
(14:26):
monitor uh the state of our GPSsatellites.
Um so they were they were Javaapplications written with swing.
Um and I believe swing is stillavailable as uh an API in Java.
I don't think anybody uses it.
I don't think they should.
But everything I was working onat the time had to do with Java
(14:47):
Swing development.
SPEAKER_00 (14:48):
Yeah, maybe those you know, Java jobs that you
created are still around andrunning somewhere, monitoring
everything.
I can only use it.
If it works, you know, don'tchange it unless there are new
requirements.
Um but that that's that's reallysuper cool how you did the
transition from uh from thatworld into software and kind of
working on uh yeah, working umalso working at the governance.
(15:14):
It's always interesting becausethere's so many cool projects
that unfortunately people cannotshare.
Uh but a lot of very coolthings.
SPEAKER_01 (15:22):
Now a quick word from our sponsor.
Confluent developer the podcastis brought to you by Confluent
Developer the website, which haseverything you need as a
developer of data streamingsystems.
And it's completely free.
We've got curriculum, hands-onexercises, executable tutorials,
the online data streamingengineer certification, also
free, a way to find a meetupnear you, those are free.
(15:45):
Everything is there.
I really want you to besuccessful in your journey as a
data streaming engineer, andthis is the site that has what
you need.
Check it out atdeveloper.confluent.io.
That's developer.confluent.io.
Now back to the show.
SPEAKER_00 (16:01):
Okay, let's do a jump forward.
There's a lot of, you know,challenges, software challenges,
architecture challenges, um, oras you mentioned, like domain,
how do you even define likeowners?
How do you split it?
How do you how do you create theright interface and so on?
Um maybe you want to share somerecent challenge that you worked
on or you're still working onand some learnings around that.
SPEAKER_02 (16:23):
Yeah, so the it's this is I I I'm gonna take this
question, I'm gonna uh move it,bleed it into kind of what this
this overarching problem thathas permeated the last 10 or so
years of my professional life.
And that challenge, I I refer tothis challenge as event-driven
views.
(16:44):
And the the the setup for thischallenge is is pretty I I my
guess is it's pretty um uhcommon to a lot of people that
have been in tech for anysignificant amount of time,
especially any anybody that'sworked at companies um that
started with large monolithicstructures and found out that it
doesn't scale well.
(17:05):
Um so uh the the problemstatement starts is that starts
like this.
Most really successful techproducts, software products, at
least prior to five years agoand in the previous couple of
decades, they usually startedreally quickly with somebody
creating a database with a bituh a nice schema and then
(17:27):
setting up a service in front ofit and then demonstrating value
and then launching that toproduction.
And then as they wanted to addnew things, they would just add
new things to that service, addmore data to the database, and
then as the as the product mighthave grown, they would have had
people, other internal uses ofconsuming that database to run
things like analytics.
(17:48):
Maybe they attach some changedata capture to the database so
that they get real-time uh datapropagation downstream.
By and large, what you find outis if the company has a long
life, has a has a longer thannon-trivial life and they keep
building on that initialstructure, what they end up with
is a giant mess of a databaseand and monolithic service and
(18:12):
monolithic architecture thatbecomes difficult to work on,
right?
We've we've heard uh monolithmight as well be a four-letter
word in the world of uh softwareengineering and software
architecture.
Um I think everybody's familiarwith with the pain that those
kind of monoliths can solve orcan uh induce.
(18:34):
Um so what ends what ends uphappening by and large, and this
was really popular throughoutthe 2010s while I was uh
progressing from junior, senior,uh, and a staff software
engineer, uh, was the approachto what the the approach for a
solution for monolithicarchitectures is start breaking
it down into microservices.
(18:55):
So microservices, in my view,should imply uh at least a
little bit of knowledge andbackground on domain-driven
design, because you can't havemicroservices kind of spawn from
the world of domain-drivendesign.
Um so uh uh having having anunderstanding of domain-driven
design and uh leading thatleading into creating
(19:17):
microservices was prettyimportant for me.
Um and I was fortunate at my atmy second job that I had, uh
they were uh pretty adamant thatthey spend a good amount of
resources on up leveling theirengineers.
And one of the topics that wewould talk about uh was
domain-driven design.
So I had a I had a pretty gooduh good academic start on
(19:39):
learning about why why you breakup monoliths into microservices
and how you use domain-drivendesign to do that.
Um and it wasn't too long afterthat that I stumbled into this
problem that kept on coming upof okay, we've broken that we've
broken this monolith intomicroservices using
(20:00):
domain-driven design.
And now the problem is we haveuse cases like analytics, or we
have use cases that are domainsthat are span other domains that
we've identified.
We need we need a domain thataggregates data from domain A
and domain B, and we need tokeep an entity uh up to date in
(20:25):
real time based on changes tothese other domains.
It used to be the case if youhad a monolithic architecture,
all that data was in one place.
You could just, it was easy,create create that view off of
the data that spans multiplebounded contexts, it's all
munged together in the samedatabase anyway.
Seems like a prettystraightforward thing to do.
(20:45):
Obviously, you can't really dothat with uh with uh
microservice architecture.
And and even doing that even ina microservice architecture
implies you have some form of anevent-driven, uh an event-driven
design baked into yourarchitecture as well.
Because how can you, how doesany downstream domain know about
(21:07):
data changes in an upstreamdomain if that domain isn't
producing events?
SPEAKER_00 (21:12):
Right.
SPEAKER_02 (21:13):
So we so we've already talked about like three
three pretty large concepts thathave entire books dedicated to
them domain-driven design,microservice architecture, and
event-driven, event-drivendesign.
Um and the those all those alllead to this, are all a
component of this problem thatkept coming up of we need to be
(21:34):
able to do something interestingwith a view of data that spans
multiple bounded contexts.
That could be analytics, itcould be indexing into a search
engine for multivariable search,it could be its own domain that
other people want toincrementally operate on, but
keep that data replicated fromthe other, keep certain data
(21:56):
replicated from other domains.
There's a whole wide gamut ofuse cases.
SPEAKER_00 (22:03):
Let me just see if I can I can I f I follow up
everything you said.
So essentially you you have uhyou broke into Monolith, right?
Uh you turn it intomicroservices.
SPEAKER_02 (22:12):
Each one of them has their own you know streaming
data from place to place and umuh serving uh whatever
requirements uh they areresponsible and and owns um and
suddenly you get morerequirements that are now more
kind of like data focused likeyou have analytics you have
different services that needs toaggregate uh information right
(22:33):
and this is where kind of likenew challenges uh appears in in
the system okay cool that'sexactly right and the the extra
compounding factors that make itmore that make it more difficult
is you find out that if when youone of the benefits of
microservices and domain drivendesign is you can separate
(22:54):
things but as those thingsbecome owned by different teams
those teams have differentpreferences one team might want
to use a relational database orit might make sense for their
their domain architecture to usea non-relational database.
Another team might be usingKafka to produce their events
but they might be using an AWSnative solution so they might be
producing events to SQS.
(23:16):
Or and one of the things that II find really common is that
given that Kafka is rel is isrelatively young compared to
message buses in general thesome of the older companies and
tech stacks will use somethinglike RabbitMQ or some uh any any
older uh message bustechnologies and what the the
(23:39):
compounding factor there is wellany solution for an event driven
view that you need has to beable to account for this
polyglot infrastructuredifferent databases different
message buses and whatnot isdifferent right so the tech
stack is different so what whatwhat's what do we do?
(24:00):
Yeah absolutely so um this thethe the nature of a polyglot
architecture and then the thenature of the eventual
consistency of an event of anevent driven system those all
lead to a unique set of problemsif you're trying to solve this
this come up with a solution forthis abstract need to come up to
(24:20):
see a view an up-to-datereal-time view um of the data
that you care about that needsto be joined across domains um
one of the things I touch on andI'll touch on in my talk next
week is this sounds reallysimilar to like GraphQL right to
like query federation I havedata across multiple domains
that I need to join on demandand and be able to view that
(24:43):
data.
Where event driven views differsfrom something like query
federation or GraphQL is thatevent driven views is much more
backend oriented.
You want to you you know theexact services you need to hit
so it's not it's not it's notexactly federated and there's a
lot of throughput and latencyconstraints that you'd like to
(25:05):
account for that aren'tnecessarily addressed in
GraphQL.
And in addition not justquerying the data but notifying
your downstream consumers whenthat view that you're keeping
has changed becomes not justdesirable but a requirement in a
lot of cases because a lot ofdomains will care about when the
join data changes not just theindependent data from the
(25:27):
upstream domains change.
So that is it's a long-windedway to explain the the entire
domain of this uh this problemof event driven views um which
is what I've been working oncoming up with I've come up with
helped come up with severalsolutions um in the in the past
uh decade or so um and the oneI'm presenting on next week has
(25:49):
to do with this uh streamtopology that I that I have
termed ingest materialize indexum so that that is the that is
in the name of my presentationnext week which um I shortened
that to IMI um and IMI is allabout coming up with a simple a
maximally simple solutionattempting to be maximally uh
(26:13):
simple uh solution uh for eventdriven views cool can you give
us like a brief uh you know thisis how we build it absolutely
yeah so uh uh IMI ingestmaterialize index it's I name it
that because it's based off ofthese three uh three stream
stage each each of those wordsmaps to a stream stage so the
(26:37):
way IMI starts off is you ingestevents from any any uh producers
of events and those those eventsshould indicate um typically
ideally they should be domainevents but not everybody has
those sometimes they're just rawCDC events but these events
should indicate when data thatyou might care about viewing has
(26:59):
changed so that ingest stage isresponsible for consuming
whatever the raw schema isupstream that you don't control
because the producers controlthat schema consuming that those
events and then using the datain those events to relate the
foreign data the foreign keys inthat data to the view keys that
(27:21):
need to be reprocessed orrematerialized.
So in ingestion you have thisresponsibility of correlation
and then ultimately once thatcorrelation is done you produce
commands.
So this is I'm CQRS commandquery responsibility segregation
is another concept that's bakedinto this not just the problem
(27:43):
but also the solution.
So ingestion is responsible forcorrelating the changes in
foreign domains to the domain ofyour view and then generating
commands to reprocess that view.
That feeds into the second stageof IMI materialization and in
materialization what happens isyou take these commands and in
(28:08):
our in all of our cases we applymicro batching um which is take
in elements or take t durationof commands whichever one comes
first and then you take thatbatch of commands and you
dereference as usually the wordthat we call we dereference all
of the data that's uh related tothe view that you want to
(28:32):
generate and then we feed thatblob that aggregated blob of raw
data to a generation phase andthen that generation phase
produces all the views that needthe current view the current
state of all the views that needto be indexed and those get put
onto another Kafka topic andthat feeds into the the third
(28:52):
stage which is index so each ofthose views can be indexed to
any number of destinationsincluding a database where you
can look up the data on demandor you can have CDC from that
database which we also do whichlets you let your downstream
clients know when that data ischanged.
SPEAKER_00 (29:12):
Cool.
SPEAKER_02 (29:13):
So I'm guessing like you know thinking through there
are some requirements that thissystem has to comply to like I'm
thinking maybe um exactly onceright if anything changes you
know there's some command oroperator something happened to
the data and we are sendingthose events as changes we need
to make sure it actually arrivesat the destination right yeah so
(29:35):
it it ends up being the case uhwith all of our use cases that
we don't care about exactly oncewe just care about at least once
um so that that is our minimumlevel of guarantee you will get
you will get these changes willflow through the system at least
once um the the other thing Iwanted to call out um before we
(29:58):
move on to uh before we move onto the next thing is the the
architecture itself I'm I'm oneof the things I'm happiest about
with it it's number one it'sabstract so it can be applied to
several different uh streamingmethodologies um but it requires
a very minimum set of resourceswhich is one of our requirements
we don't want to we would liketo not have to replicate our
(30:21):
data all over the place just tomake it compatible with a given
tool um and the solution the thebare solution that we that uh
IMI proposes is a singleapplication with at least three
stream processes in it and thentwo Kafka topics so that
absolutely minimizes the numberof resources you get we we get
(30:44):
at least once processingespecially using Kafka as uh the
intermediary between the streamstages um and we get robustness
um because we know we won't uhwe have we have uh a library
that we use it then this is alsoa part of my presentation um a
bit of a passion project thatI've been working on called
(31:05):
Atleon um and that's a it's alightweight stream processing
framework that gives you theguarantee of at least once
processing regardless of whatinfrastructure you're consuming
from or producing to that'sreally cool well I'm very I'm
very excited for your talk andif anyone is watching it
afterwards it's gonna beavailable on YouTube as well so
we can always catch up with withthat um I'm curious is the is
(31:28):
this library is ever going to beopen sourced or do you think
about that?
Yeah it is open source oh it isopen source yep you can check it
out at atleon.io very very cooland the the name the name is uh
comes from at least once at Leonum so that's uh I've been it's
(31:48):
it's been a project that I'vebeen working on for going on
through might be four years nowwow okay that's uh I mean some
of the best projects are theones that we work on for a long
time in my opinion so very coolall right any like learning so
far that you want to sharebefore we're wrapping things up
(32:11):
from you know thinking about thesystem designing architecture
pitching uh the system alsogetting buy-ins from some folks
as well I'm sure you've beenworking you know across the
company with multiple teams Ithink my biggest lesson with
with with all all of this stuffconcerning uh not just event
driven views but but streamprocessing in general it's super
(32:34):
super powerful but there therethere is a pretty big learning
curve when you're going everyeverybody starts out their their
first project is uh or theirfirst like business project is
usually create a a rest endpointor maybe a grpc endpoint and it
it's really intuitive to thinkrequest response right moving
(32:56):
into the world and and this isthe challenge that I I face a
lot when designing asynchronoussystems and teaching other
engineers about asynchronous uhsystems and stream processing is
you don't really know what'sgonna happen after you fire an
event off.
You know you you agree to acontract and you produce it and
then the things that happenafter that not only not only are
(33:19):
you not really sure anybody'sgonna do anything after you fire
that event off but it some ofthe things that that happen
might be non-intuitive.
So it it's it's definitely I Ifind asynchronous processing to
be extremely powerful a very funproblem space um but I but a
(33:39):
pretty a pretty high learningcurve but a learning curve
that's definitely worth itbecause it it definitely moves
you to the the next level ofbeing able to think and design
systems that scale and eventhough even though it's hard and
I'm saying it's hard shouldn'tbe afraid of it.
Dive in and uh learn learn whatit's all about.
SPEAKER_00 (34:00):
Yeah just do it it's the future cool sage thank you
so much for joining me today andsharing all your expertise and
again congrats on the promotionand uh you know if this uh
people watching after currentwill share the link uh and as
well to the YouTube uh videothat will be uploaded after the
event so everyone can uh youknow go back and and listen and
(34:23):
learn uh from the recordings.
SPEAKER_02 (34:26):
Sounds good.
SPEAKER_00 (34:27):
And of course it was a pleasure
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Bobby Bones Show
Listen to 'The Bobby Bones Show' by downloading the daily full replay.