May 18, 2026 • 40 mins
Adi Polak talks to Russell Spitzer (Snowflake) about his career in open source data infrastructure. Russell’s first job: software engineer in test at DataStax. His challenge: making Apache Iceberg ready for AI and streaming.
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
SPEAKER_01 (00:00):
What will the future even be of having all these
different coding languages in aworld where the LOM can write
all of them pretty muchequivalently well?
It's fine if a vibe code projectadds like 10 extra classes when
only one was required.
For something like iceberg, weneed to keep it tight.
We can't turn it into ahairball.
SPEAKER_02 (00:17):
Hello, I am Adip Polak with Confluent Developer,
and I invite you to join ustoday to my conversation with
Rysel Speitzer.
Rysel is a principal softwareengineer working for Snowflake,
and he is also a thought leaderin the industry and supported
and contributed to many opensource projects.
So sit down, grab your favoritedrink, and join us for a
(00:39):
fascinating conversation on howAI shapes our distributed data
infrastructure, and what aresome of exciting features that
are coming out specifically foriceberg?
It can make your life a lot moreeasier when thinking about
unstructured data, referencingother file formats from your
catalog or from your tables, andgenerally making AI workloads
(01:03):
accessible, available, and fast.
Yes.
Also fast because we need totake care of streaming
architecture.
Hey Russell.
SPEAKER_01 (01:19):
Hey Addy.
SPEAKER_02 (01:21):
So good to see you.
It's been a while, and I knowyou've been busy recently.
So what's what's happening withiceberg?
SPEAKER_01 (01:31):
Yeah, I mean iceberg is going through a huge
transition time right now, asI'm I'm sure everyone is, right?
We we've come to this new worldwhere this LLM-based AI stuff
and agentic things iseverywhere.
It's part of everything, uh,part of coding, part of
reviewing, part of productmanagement.
And iceberg is changing to kindof meet that need.
Uh one of the cool things thatwe're working on right now is
(01:53):
Iceberg format version v4.
And the two big targets for V4of Iceberg are to make it a
great solution for AI use casesand for streaming use cases.
And this is really showing thehistory of the project, right?
It started as this format thatwas really oriented towards this
(02:13):
kind of hive replacement, trenoreplacement, data warehouse
replacement.
And that has a really differentcharacteristic from streaming
workloads, as I know you'reextremely familiar with, and
these new AI workloads, whichare coming in with some
multimodal things, we're seeingvector stores, we have all kinds
of different uh challenges thatwe need to kind of address.
And one of the coolest thingsabout working in open source,
(02:36):
especially on something likeiceberg, is we can take ideas
from all over the community, allover the industry, and integrate
them into something thathopefully ends up working for
everybody.
So I'm very excited right nowabout Iceberg V4.
SPEAKER_02 (02:49):
Cool.
Yeah, it's an exciting time tobe working on the infrastructure
for sure.
I think it's kind of coming backnow with AI.
We're realizing that, hey, datainfrastructure does matter in
the age of AI and we do need uhspecific, you have specific, we
have now specific requirementsfrom our infrastructure, whether
(03:10):
it be analytics or a streamingum space, but we do need other
stuff.
So what are the new things in inv4 that enables us to run AI and
also whether some of the thingsyou know we should have in it?
SPEAKER_01 (03:28):
Yeah, I mean, so the things that we have pretty much
consensus on now for v4.
And since we're a communityproject and we don't we don't
actually have a v4 until thewhole community gets together
and votes and says, that's whatv4 is.
Everything I'm saying is prettymuch what is the current state
of the project.
But the biggest things we'redoing for AI enablement here is
(03:49):
we are adopting a method ofbasically going uh vertical with
your changes to a table.
So previously, the way we alwayslike to think about tables is
you know, we've basically gotthis relational table, and every
time you add a new row, you'rejust adding things onto the
bottom, right?
And then iceberg later added theability to do acquit uh delete
files and uh delete vectors, andthat'll just let you knock out
(04:12):
rows.
But again, these are horizontalcuts to your table.
What we didn't have is theability to divide your table up
vertically.
So basically saying thesecolumns are in one file, these
columns are in another file, andthen the ability to replace a
column within a file withoutdeleting it.
So basically, previously, everytime you wanted to get rid of a
column, you had to rewrite theentire data file and all of the
(04:35):
other columns that were adjacentto it.
Uh, this is really bad for a lotof AI use cases where you're
essentially maybe adding a newfeature every day.
And that feature is essentiallyjust an extremely long uh set of
vectors or set of doubles orfloats or whatever.
It's it's usually this maybesparse, maybe dense, but
(04:56):
columnar addition to your table.
So basically, instead of our oldtables where we added at the
bottom, we are now addingsideways.
So as we go sideways, we need toadjust the table format to
handle that sort of thing.
And this is, you know, uh anapproach that's been taken by a
few other projects, and we'reexcited to kind of make it into
a first-class feature withinApache Iceberg.
(05:16):
Um, the other big discussionthat we keep having is how are
we going to better deal with themultimodal aspect of a lot of
these table formats?
And there's there's two bigapproaches that I think are kind
of going being being worked onin conjunction between the
Apache Parquet community, theApache Iceberg community, and a
(05:36):
lot of other communities, wherewe're kind of trying to figure
out is this a file formatresponsibility or is this a
table format responsibility?
Because there's there's twomodes, at least I think, of
going about handling that.
And one is we embed ourmultimodal, our non-structured
data directly in the same kindof data files that we are
(05:57):
storing, everything else.
So this is an approach that Ithink like Lance takes, and uh I
think some other formats arealso trying to do this this kind
of approach.
Um and that's one thing you cando.
We could do the same thing withparquet, and we're kind of
looking at parquet about does itmake sense to improve the way
we're storing things so that wecan do something similar, and
(06:17):
then we can use again within thetable format the ability to
store columns separately to putlarger objects inside of
completely different files orsomething like that.
The other approach, which Ithink is really interesting, is
actually just using a referencewithin your relational table to
uh unstructured data that is nota part of the row structure
(06:38):
itself.
So this is basically saying, whyare we even putting this data,
which we know is actually verycompressible and very, you know,
good in its current formats?
Like a PNG is pretty great as aPNG file.
You can zip a thousand of themtogether and get pretty good
behavior if they're all prettysimilar in structure.
Why do we want to move that intoa structured file that has a lot
(07:00):
of other things in it?
So there's a question of whetherwe could just store references
to those files within arelational table and then have
the table format be responsiblefor the lifecycle management.
So, like whether or not youactually have credentials and
access to get to those externalfiles or when to remove them
because there's no longer anyreferences to them in the table
and things like that.
(07:20):
So there's two directions Ithink that Iceberg is going
there.
So we're we're obviously workingvery closely with the people in
the parquet community to try tomake sure that we have support
for storing this kind of data.
And also we're approaching thatexternal reference idea as well
to kind of see if that is theright way to go uh at the same
(07:40):
time.
Because, you know, I thinkthere's there's different, there
might be different fits fordifferent types of multimodal
data.
SPEAKER_02 (07:48):
Yeah, no, it's um it makes it really interesting.
It makes a really interestingproblem.
I remember back in the days, andand maybe that's a little bit
old, but if you know, we heldreferences inside a table for
for years, right?
And the always the challenge waswho manages that life cycle to
the way you that you mentioned,because it's um there is a
(08:09):
question of transaction andatomicity of that transaction
that we need to think about.
Like now the file got changed.
How long does it take tomanifest inside the table that
holds the reference that thatfile now got changed?
And which side is responsible uhfor that atomicity to happen?
(08:33):
And how do we pair these twosystems and perhaps these even
two tables together in order tomake validate that that actually
the case?
Um Yeah.
SPEAKER_01 (08:44):
I mean that's why I think it's really like a table
format level problem.
Like if we can define that atthis multi-file level, then I
think we we can actually goforward pretty confidently and
say like these are expectedbehaviors, and then we can have
engines and everyone kind ofwork within that framework.
Because I I know that we we didthe same thing back in the day,
and a lot of jobs I've had, andlike if everyone's doing it on
(09:07):
their own, that's the problem,right?
Because we end up with a ton ofdivergent things and none of
them work together because theway you do it doesn't work the
way I'm doing it, and you end upwith atomicity and consistency
issues and broken guarantees.
SPEAKER_02 (09:20):
So yeah.
And from my experience workingwith iceberg, and I also
reviewed a couple of reallyreally great books from from you
know, great authors from ourcommunity.
I had the opportunity tocontribute there as well, kind
of like more behind the scenesas more of a technical reviewer.
Um, there is the notion ofbranches in iceberg today.
SPEAKER_01 (09:43):
Yeah, I mean, iceberg already has a lot of
those components we would evenwant to do this sort of thing,
where we're where you want tojust build a completely separate
branch, do all of your testingthere, and maybe do your
management of files, and thenonly merge it back in when
you're like, I'm sure the statehere is good.
I mean, I haven't written up anyideas on how we're actually
(10:04):
going to do that for thesereference types.
Because right now, you know, thequestion is what should be in a
reference type, where should wedefine that?
Um, we've had some proposalsthat maybe we should even put
the base definition of what thatstructure looks like in parquet,
just so that we have a kind ofglobal definition that isn't as
table-biased, and then we canput our rules around it inside
(10:26):
of iceberg.
SPEAKER_02 (10:28):
Right, right.
Well, you know, and if I look atKubernetes world, if I look at
uh things like Kafka, uh RocksDBas a source of truth is always
uh you know a solid idea anddirection.
But you know, I I'd be curiousto follow along and see how that
works.
SPEAKER_00 (10:47):
Now a quick word from our sponsor.
Confluent developer the podcastis brought to you by Confluent
Developer the Website, which haseverything you need as a
developer of data streamingsystems.
And it's completely free.
We've got curriculum, hands-onexercises, executable tutorials,
the online data streamingengineer certification, also
free.
A way to find a meetup near you,those are free.
(11:10):
Everything is there.
I really want you to besuccessful in your journey as a
data streaming engineer, andthis is the site that has what
you need.
Check it out atdeveloper.confluent.io.
That's developer.confluent.io.
Now back to the show.
SPEAKER_02 (11:27):
Yeah, and also that there is the question, uh I
think you touched about it alittle bit, of streaming,
because adding another layer ofatomicity and adding in perhaps
another source of truth databasethat is strongly consistent um
adds latency.
SPEAKER_01 (11:48):
Yeah.
I'm really hoping we don't haveto do a second system to make
this work.
I think we we may design it withlimitations that make it so that
you we just don't need that.
Um, you know, do immutabilityand and other stuff like that,
where you know, there is onestate of the table, the files
have to be in the same state fortheir entire lifetime.
You may not change the state ofa file, offsets should always be
(12:10):
true, you know, that that kindof stuff.
For streaming, though, withiniceberg v4, the things that
we're excited about is we'rereally trying to break down the
amount of time it takes to do acommit.
And that's one of the other bigv4 changes, is we're changing
the structure of the metadatatree.
So it uses almost all the sameideas, but pretty much anywhere
where we couldn't cachesomething before because we were
(12:33):
changing it so often, we'rewe're breaking that and making
it more immutable, more lasting.
So basically, we looked at allof these ideas that we've been
applying to our data files, likethe ability to knock out a row
in a file without deleting it,and the ability now to replace a
column in a file withoutdeleting that whole file.
We're like, well, why don't wetake those approaches and apply
(12:54):
them to the metadata layer aswell?
So when you do that, suddenlywe're we're basically gonna set
it up, and and I don't want togo too deep in the technical
details of this, but there'll beone file that's basically your
entrance into the rest of themetadata tree that will be
completely overwritten on everycommit.
And we're gonna keep that verysmall.
So something like 400 kilobytesis what I keep throwing around,
(13:15):
because on our tests we saw thatyou know, doing IO to S3, you
basically have no differencebetween like four kilobytes and
400 kilobytes.
I mean, there's a slightdifference as you as you go up
and up, but you know, if ifyou're going to open a handle to
S3, you might as well uh giveread that many bytes.
Anyway, so you you read thatmany bytes back and you're gonna
replace that every time.
(13:35):
And when you add new files,you'll add them directly into
that layer.
So the way you used to have todo it is if you wanted to add a
new file, you had to also writeanother file that a manifest
file that it would refer to.
And then the data file would beunderneath that.
And that meant that when youwere creating a snapshot, you
had this serial process wherefirst you made this file that
(13:56):
had a listing of the data file,and then you made another file
that referenced that file.
And because that's you know aserial process, you basically
had to pause each time toactually do a commit.
Now you'll basically say, I'monly overriding that top-level
file, which hopefully you havecached in a memory or like a you
know something like that.
And you quickly change it, youwrite that new file, and you're
(14:19):
done.
So for a streaming use case now,you should have only one IO
operation required to do yourcommit each time, as opposed to
two that have to happen inserial.
So hopefully that helps a lot.
And then the other thing we'redoing, right, is that because
this will always be in thistiny, tiny layer up at the top
of this metadata tree, thatshould also be a pretty low
latency operation, as opposed tohaving to do some more drastic
(14:43):
restructuring.
One good example of this is ifyou wanted to ever delete
something before out of aniceberg table, you basically had
a manifest that had an entrythat said this file exists.
To remove that entry, you had todelete that manifest file.
You had to completely rewriteit.
Because if it still existed, youwould always have this little
entry that says that fileexists.
(15:04):
So we would have to delete itand write a new version of it.
Now, since we're using thoseabilities that we had on data
files at the metadata layer, Ican just say I'm taking that
delete and I'm applying it to amanifest file and just have a
little marker.
So basically a delete vectorthat says that row doesn't
exist.
So we can basically, withoutdisrupting your cache, that file
(15:24):
is still cacheable, you canstill keep it in memory.
You can mark things as deleted.
So suddenly deletes are way, wayfaster than they used to be at
this metadata commit layer.
So those are the kind of thingswe're pulling together.
There's a lot more details onhow we're kind of structuring
this, but the end goal is tohave that commit process be fast
enough that you can start doingvery rapid commits to an iceberg
(15:46):
table if you really want to.
SPEAKER_02 (15:48):
That's awesome.
I love the idea of caching, andit uh reminds me a little bit of
uh Hadoop world back in the daysand some of the latency
challenges uh we had with uhwith HDFS and uh and yarn and
it's the rest of the zoo.
SPEAKER_01 (16:02):
Uh yeah, I mean we don't want to do I.
That's the story of all of thesetechnologies, right?
Is we're just trying to avoidI.O.
all the time.
It's what can you not read?
What can you not write?
What can you cache and not haveto access again?
SPEAKER_02 (16:20):
Right.
We've learned as as an industry,as an open source community for
sure.
And I think AI pushes us to forthat as well.
I mean, latency is waiting forsolutions like cloud code or you
know uh other other solutionslike vertex AI and uh Gemini and
(16:45):
and so on, waiting for them toactually uh give us back a
response starts to be you knowfrustrating.
It's like I gave you a prompt,can you can you make it happen?
SPEAKER_01 (16:57):
This is this is actually one of the reasons that
I of all of the technologiesthat I think people are working
on right now, I feel like tableformats will probably be the
least changed because they're atthis layer where you need to
interopt with them.
And if you have to, like, Imean, right now we're in a great
time, right?
You can basically say, pleasewrite me an interop layer
between these two type fileformats, or write me something
(17:19):
that converts this query intoanother type of query.
But the problem is theunderlying data structure is
still held by whatever itsoriginal limitations are, and
all those layers that you buildin between just make it worse.
So we need to have theseinteroperable formats at the
bottom of everything, becauseotherwise, regardless of how
fast your models are or howsmart they are, they still will
(17:39):
have that translation cost orthat modification cost.
And the only way to make itbetter is to fix the I.O.
at the at the very bottom layerof it.
So I think it's definitelyinteresting time.
SPEAKER_02 (17:54):
Yeah, and and that requires, you know, you know,
solid engineering, a lot of thework that you do with the open
source community and uh bringingdifferent people from different
organizations together.
I believe this is where whenpeople ask me, like, what's why
community, right, why opensource?
I always say this is this is thepower.
It's like you um it's a hive ofminds that comes from across uh
(18:20):
you know the industry and makedecisions and discuss the
challenges and uh go very deepinto the technology and the
nuances of you know the exactformat, the solutions, what do
we need, and what's gonna propelus forward and um be a solution
that it's kind of solves theproblem rather than a company
(18:41):
specific that sometimes can beyou know very siloed.
SPEAKER_01 (18:45):
Yeah, I mean that's that's the danger with working
on you know a company specificproject is even even if you have
the best intentioned engineers,there's so much pressure from
you know the the folks with thedollar signs and the C titles,
right?
We're like, you know what, butcouldn't we just delay this?
Or you know, shouldn't that beonly available to our users?
Or oh, if we didn't do that,wouldn't it make it harder for
(19:07):
our competitors to use the samething?
And when you're in a true opensource governance, and I mean
that's one of the things I loveabout the iceberg community, is
you just can't do that.
Because even if one person says,hey, I'd really love to do this
thing that would make it harderfor everyone else, you'll get
nine people who will say, No,that will make it harder for me
in my job.
So I'm not gonna support you indoing that.
You better have a really goodtechnical reason why we should
(19:30):
all take on that burden.
Um, and it the reason can't beit'll make you more money in Q1.
SPEAKER_02 (19:36):
So Yeah, it's I think as engineers you want to
solve the problem.
Um and sometimes organizationsproblem across the board, it's
like, hey, we we want to makemoney.
And then the question is like,how do we how do we support that
motion in the uh while stillsolving the real the real
engineering problems?
SPEAKER_01 (19:57):
Yeah, I mean I like again, that's another thing I
kind of like about the formats,is that the only the real value
of iceberg at its core is thisinteroperability and this open
governance layer.
So all the people in it keepthat as like a very strong focus
because there's really no pointin supporting iceberg if you're
the only one who supportsiceberg.
(20:17):
We we want to support icebergbecause everyone supports
iceberg, which means that youknow the the pitch I do all the
time internally is I'm like,every person who's using iceberg
is a zero friction customer toSnowflake, potentially, because
they are using the four method.
We can integrate directly.
There's no ETL, there's noconvergence, whatever tools they
(20:38):
have can keep working while theytry POCs and things like that.
And every other vendor, I think,sees it kind of similarly.
This is the open playing field,which means that everyone can
try whatever they want.
And of course, from an end userperspective, I really love it
because that makes everyonecompete to make sure that this
layer really works well and isreally compatible.
And I think the the generalindustry benefits a ton from
(21:02):
having this like communityproject, similar to, you know,
like HTTP standard and thingslike that, where like just
having an industry standardwhere people can all agree
benefits end users way more thanit benefits the end the
companies, but they can alsotake advantage of that because
you know everyone who'scompatible can build up whatever
their offering is on top of it.
SPEAKER_02 (21:23):
Yeah, yeah, I love that.
And it's so true.
It's faster adoption as well asfaster POC.
Imagine if a customer now wantsto see it's you know, if they
they want to buy a new softwaresolution, right?
We do it all the time, weevaluate software.
We want to make sure we canactually build that POC
relatively fast and see thevalue fast, if you know, if we
(21:47):
can.
And and if that's not the caseand it's too challenging, then
we flag it.
Hey, there's gonna be a steeplearning curves, here are the
integration challenges.
This is something that we expectyou know the software company to
solve.
uh for us to integrate with thatwith them.
Um and then it's it becomes kindof like a question of back and
forth and if it's the bestsolution for us.
And of course there are othercomp competitions, right?
(22:10):
The competitors on the marketthat might have a better,
simpler to integrate, simpler torun a POC solution.
It's gonna take us as a customerand this is uh it's a huge huge
power for for open sourcebecause um if we can enable that
(22:32):
then we have a faster path towin.
So that's true.
Cool.
So we talked a little bit aboutAI and some of the AI workloads
and some of the requirementsespecially around referencing
when discussing you knowunstructured data and then we
need to dive and kind of bringuh unstructured data um that is
(22:52):
compressed and that you knowpart of the table is kind of
like a binary nasty binary uhformat um and we talked about a
little bit transactionabilityand optimicity and so on and a
little bit about streaming.
I know you know you mentionedthere is um icebergs of support
(23:14):
bringing in your own catalog aswell right there's a dedicated
um API and there's a dedicatedspec that you can work against
uh as a developer to bring yourcatalog and recently I've been
seeing more and more uh aboutsome of the catalogs that you've
been working on uh and I'd loveI'd love to learn more and also
(23:36):
you know how you see thatconnects with maybe the new
format that you know you'redeveloping.
SPEAKER_01 (23:42):
Well I mean the the iceberg rest specification is
also growing a ton right now andas you mentioned it's basically
the the lingua franca we've madefor integrating with an iceberg
catalog which is a completelyseparate service we basically
define this is how you will talkto a catalog for an iceberg
table this is how you will getback metadata this is how you
will propose new table updatesall the all that kind of stuff
(24:04):
and what's really great abouthaving this API so well defined
is that now it's just reallyeasy to move your catalogs uh
your your tables from onecatalog to another so in the old
days before we had this youbasically were like I'll pick
hive and you pick hive and youconfigure everything
specifically for hive all ofyour clients use hive specific
(24:25):
code hive specific versionthings and what if you had a an
issue you really had no way tocontrol anything because
everyone is talking a hivespecific language to your
catalog and your Hive Metastoredoesn't really know it's dealing
with iceberg tables.
So there's no way todifferentiate what's happening
or tell the client like oh no Ithink you're using a bad version
(24:46):
of this iceberg library becauseyou're using a Hive library and
all you have is this thriftrequest.
So moving to the rest suddenlyallowed everyone to to put more
information in their in theirrequests and have all of these
catalogs have this front endthat says okay I'm dealing with
an iceberg client oh this is abad version I'm just not going
to accept requests from you.
(25:06):
I'm gonna send back an errormessage that you need to
upgrade.
Or you can take requests and youcan say well I have all my
commit logic now internallybecause in the old days
basically the the the clientwould stage the commit using the
catalog's own specific API.
So for example for Hive youwould say Hive I need you to
lock this table reference fromthe client and then on the
(25:29):
client you would check to seewhether you had a valid commit
and then you would then swap thetable metadata using an alter
command and then unlock thetable with a direct hive API
again.
So it's all very hive specificand it's all done on this
client.
So the clients can propagateweird errors if your client
doesn't know how to interactwith the underlying storage.
(25:51):
So this is why I really love theREST API, because it puts a
layer in between the client andthe actual persistence for your
table entities which means thatit's it's much easier to
centralize fixes for this sideof thing or stop badly behaved
clients from doing something anddisrupting your whole setup.
It also of course completelydetaches authentication from all
(26:13):
of the backing storages so youcan have whatever auth you want.
You know if you're using HMSbefore you basically had to use
Kerberos right like that wasyour that was going to be your
security system sorry that'sjust it but if you talk with a
REST client right it's whateverthat catalog decides to expose.
So you know we have a lot ofOAuth things so I work on Apache
(26:34):
Polaris which is a project thatuh uh a while back was was
started by Snowflake and theythey basically as I was hired
and started working on icebergthere were like we need to be
more open source we can'tcontrol these things we need to
have an open community that'sbehind them so they took the
Polaris project they donated toApache and now we're just one of
(26:55):
many contributors to thisproject which you know because
it has the REST API iscompatible with iceberg clients
all over the place and providesall kinds of awesome features
and integrations with all kindsof identity providers and
offlayers.
So you know it's it's reallygreat to kind of be in this
space now where if you want toyou can use Polaris as your
(27:17):
catalog.
Or if you want to switch you cango find another producer of an
Apache iceberg catalog thatservices the REST API and your
switch is actually pretty easybecause the client is exactly
the same.
It's an iceberg REST client.
So again a lot of cool benefitsthat come to any random user and
it's nice to just be able tokeep working on open source even
(27:38):
at that uh part of the stack.
SPEAKER_02 (27:41):
Yeah it's it's pretty amazing catalogs has been
I you know forever kind of likethe war of catalogs and who
builds a better catalog and uhI'm I'm curious because you did
mention Hive and HMS.
How does Polaris and you know weall had our fair share of uh of
working with Hive and fixingtables and and the references
(28:03):
and the and and so on.
How would you say Polaris isdifferent today?
SPEAKER_01 (28:12):
So I mean Polaris was built kind of for this new
iceberg Rest world as like afirst class citizen.
I I do want to say HiveMetastore also is a has a first
class iceberg REST API rightnow.
So you don't even have to talkto Hive using the Hive client.
You can actually talk to it withthe REST API.
The biggest differences are theway Polaris is built is
(28:32):
basically structured aroundiceberg metadata as a first
class construct.
You know it basically assumesthat you're going to talk to it,
you're gonna talk to this RESTAPI and your security, your auth
and all of these are basicallypluggable components into that
system.
So you can decide whether you'regoing to use you know OAuth 2 or
something or some other layer ofsecurity as opposed to in the
(28:54):
Hive world where it's verystrictly tied into the Hadoop
ecosystem.
So you know what's what'ssupported is what's in Hadoop.
And because of that you knowlike I said Polaris can support
Ved did credentials and supportvarious cloud providers with
pretty pretty easily and um oneof the other cool things is we
(29:14):
also are able to because wedidn't assume right that the
catalog actually understands howto communicate with these tables
right you basically surface upthis iceberg metadata whereas in
the Hive Metastore we have inputformats that are like bound to
every single table.
So there's actually like literalcode you are supposed to read a
table with stored into thecatalog and you have to know
(29:36):
that you can load that class andeverything.
But we've got support for awhole bunch of other table
formats now in Polaris.
So you can actually use Lancetables in Polaris you can use
hoodie tables.
I'm forgetting what we justadded but um you know there's
all that cool stuff.
And then the pluggability alsoexists on the backend.
So Polaris itself has apersistence layer and we made
(29:58):
that persistence layer pluggableas well.
So you can choose any relationaldatabase you want to be the
persistence layer or you can usesomething like Mongo or you can
use BigQuery like you can nowhook that up to NoSQL things as
well and have that be thebackend for your your system.
So it's it's really nice there.
So I I'd say the biggestdifferences there are those kind
(30:19):
of pluggability layers thatdidn't really exist within the
original Hive code and of coursealmost no connection to the
Hadoop ecosystem so nodependencies on Kerberos or or
strong ties to Java.
SPEAKER_02 (30:32):
That's really cool.
No strong ties to Java it's uhthat's a really good way to uh
to segment that beyond you knowthe the the fact that we have a
REST API actually now that youmention it I have two questions
uh I hope you're ready let'sstart with the strong ties to
Java um a lot of the datacommunity and data practitioners
(30:55):
right whether they're your dataengineers a data science a data
analyst uh analytics and there'sso many titles today um kind of
starting using more Python rightin a lot of resolutions in the
data space are actually werewere built with you know JVM in
mind and and and Java and sothat uh interoperability between
(31:17):
those two frameworks especiallyaround performance right like
how do we build that um wasalways uh a little bit of tricky
it's like you don't want torewrite your pipelines in Java
and yet again you do want to getthe highest performance possible
um and the fact that youmentioned that I think that's
(31:40):
that's a really good val addedvalue to Polaris as a catalog
because uh uh there's a lot offriction there.
SPEAKER_01 (31:49):
Um I wonder you know if there's more things that you
know made the community to makethis decision beyond what I mean
I doubt that was one of the mainreasons behind the REST API in
the first place right likethere's there was almost no way
to get a Python client withsecurity to work properly with a
Hive purely Hive interface rightbecause you you had to get the
(32:11):
basically you have to pull inall of Kerberos to make that
work properly.
And that just didn't exist.
And the moment we have the RESTAPI up suddenly you could write
a Go client a Python client a Cclient a Rust client a Java
client and they don't have toknow anything about Java to work
properly with the catalog.
So I mean you'll see right nowthe Apache project for Iceberg
(32:34):
has all of those implementationscurrently being worked on.
And it's really exciting to meit and one thing I've seen
recently too and I don't knowwhat if we're we're we aren't
going to see this for a whilebut a lot of folks have told me
and I've seen some benchmarksthat prove this that the the
LLMs and all the coding agentsare actually really good at
writing Go because of howopinionated Go is.
(32:56):
So the the difference there isthat when you write Python right
there's many ways to writesomething right there's there's
many many right ways to do a lotof things in Python.
You've got list comprehensionsand map functions and all of
these like different structures.
But Go does not have that sameflexibility.
So some rank rankings I've seenhave shown that these agents are
(33:17):
are way better at writing Gothan they are at writing
anything else.
So I'm wondering if we'llactually see a big push for more
Go libraries.
I mean it also doesn't reallymatter because it's it's really
a difference in token cost,right?
On how long you let it churn andlike write things and have to
fix its own mistakes.
But once we get to the pointwhere LLMs are are commoditized
(33:40):
and we actually are playing thetrue true cost for all of this
I'm sure that's the we'll we'llget a big push into to working
with more efficient languagesfor LLMs even if they are not
the favorite languages ofhumans.
SPEAKER_02 (33:52):
Yeah that's a really interesting observation that I
must agree with it's the cost oftokens as uh you know like
leaders like Jensen said it'syou know engineers we're gonna
start negotiating that in ourcontracts like how many tokens
am I gonna get uh as part of mycompensation package so I can
actually you know perform at thelevel I want to perform for my
(34:14):
boss right yeah I think I thinkalmost no one is really is
really leaning hard on what thatcost is going to be because I
again I do think we're at awe're at a point now where uh
VCs are like paying a lot of ourtoken costs right now.
SPEAKER_01 (34:29):
Like your our tokens I think are very subsidized.
And we will get to a point wherethat will not be the case and
we'll probably get this hugepushback to be like why are you
using why are you using youryour engine to write C code when
you could be writing Go code oror Rust code which we know the
engine we know the LLM is betterat more efficient at and can do
for a cheaper token cost.
(34:50):
It'll be much harder to justifylanguage decisions.
I mean I I've wondered a lot onwhat will the future even be of
of having all these differentcoding languages in a world
where the LLM can write all ofthem pretty much equivalently
well with these slight costdifferences.
You know it it'll it'll be aninteresting world we'll we'll be
(35:10):
in because like we could haveLLMs just write assembly like
directly if we wanted them tobut we obviously that's probably
not the most cost efficientthing and I think that's what
will will end up being the thefinal trade-off is what language
is the cheapest to writeefficient code in.
SPEAKER_02 (35:27):
Right token economy for engineers maybe should you
know maybe you should write abook about that could be
interesting.
SPEAKER_01 (35:35):
Everything is churning so much right now that
it's hard for me to say I meanbut I I definitely think that
that it will come up in the nextnext few years people will start
I mean I I couldn't I wouldn'tbe surprised if someone writes
even a a language specificallyfor LLMs to write that they can
do more efficiently or has theright kind of you know built-in
(35:58):
checks that let it do one shotsbetter right like that's that's
really the goal.
You want it to be able to dothis with the smallest token
cost.
And anytime it writes somethingwhere there's a possible syntax
error like that's that's a badlanguage for an LLM to write
code in.
SPEAKER_02 (36:11):
Right and and that happens a lot.
I had another question.
So we has we have the REST APIfor uh for working with iceberg
and you know mcp uh is kind ofon the rise I wonder if this is
something the community lookedinto it's like having a
dedicated mcp server instead ofuh you know writing dedicated
(36:35):
you know working directly withthe Rust API.
SPEAKER_01 (36:37):
So the Apache Polaris community actually
already did and built a MCPserver to go along with Polaris.
So if you want to I believe it'sin our tools repository now so
you can basically stand it up atthe same time and have your
Polaris catalog directly MCPenabled out of the box if you
want.
So it's definitely somethingwe're thinking about from the
server perspective.
(36:57):
As like an iceberg projectitself there you know we don't
really have any servercomponents there.
So there's nothing really there.
But we actually we just mergedour our first agents md file so
I was kind of excited about thatbecause I think we were a little
slow to the game to to addingthat we we've been seeing the AI
PRs, we've been seeing the AIissues and uh you know we're
(37:20):
entering a world where I thinkpeople do realize have to
realize that they need stillneed a human in the loop.
I mean I all the buzz is great,right?
But there's a lot of times whenAI will do things in ways that
don't make sense to a humanreviewer and you need a human to
check a lot of these thingsespecially on a code base where
we can't have it growindefinitely in the way that I
(37:42):
think a lot of vibe codedprojects can grow.
You know like that it's fine ifa vibe coded project adds like
10 extra classes when only onewas required.
But for something like icebergwe need to keep it tight not
only so that we can keep workingon it and understand it but so
that LLMs can continue workingon it understand it.
We can't turn it had turn itinto a hairball which makes it
(38:03):
impossible to work on.
So you know we're trying totrying to work on that.
SPEAKER_02 (38:08):
That's that's a solid point and I see it across
the open source community withuh specifically Kafka and Spark
and Flink and so um and thinkingabout this token economy at the
end of the day it's it's gonnacost us now it's relatively
accessible I would say um butyou know who knows what will
(38:29):
happen a couple of days monthsyears down the road yeah I can't
make economic predictions butyou know people are spending a
lot of money right now and theyneed to make a lot more money
back.
SPEAKER_01 (38:41):
So that money's got to come from somebody and it's
probably gonna be us.
SPEAKER_02 (38:45):
Yes 100% all right thank you so much um is there
anything you know we have a lotof people listening in that
thinking about their careersthat they're in the data REI
space in the open source spaceand so on.
If you would to look back andgive them you know one or two
pieces of career advice whatwhat would that be?
SPEAKER_01 (39:09):
I mean this this is a hard time to give career
advice because I feel likeeverything is is shifting again.
But the thing that has served mewell and and again this is a big
your mileage may vary sort ofthing is I've I've always tried
to take whatever job or whateveropportunity will get me closer
to the thing that I actuallywant to do.
(39:29):
So if you look at my LinkedInyou'll see I started out in as a
software engineer and test atData Stacks because that was the
first position the firstinterview I got at a company
that was doing distributedcomputing that would talk to me,
would even let me in the frontdoor.
And I did not care.
As long as someone would get mea little bit closer to
(39:50):
distributed computing I wouldtake that job and I applied all
over the place and that was whatwhat got me in the door and
that's my my same advice I thinkto to almost everyone I talk to
who's entering computer scienceis you you got to just take
whatever opportunity you canthat gets you one step closer.
Don't be afraid of not going allthe way to what you want to do
in your very first job or yourvery next project.
(40:13):
Just make sure that it's it'sone step closer to the thing you
want to do.
SPEAKER_02 (40:18):
I love it.
I love it.
Russell thank you so much alwaysa pleasure speaking with you I
always you know get out of theseconversations with tons of
insights and of course morequestions uh uh and and learning
so really appreciate you takingthe time to uh jump on this call
of course it's been great thanksso much for time
What will the future even be of having all these
different coding languages in aworld where the LOM can write
all of them pretty muchequivalently well?
It's fine if a vibe code projectadds like 10 extra classes when
only one was required.
For something like iceberg, weneed to keep it tight.
We can't turn it into ahairball.
SPEAKER_02 (00:17):
Hello, I am Adip Polak with Confluent Developer,
and I invite you to join ustoday to my conversation with
Rysel Speitzer.
Rysel is a principal softwareengineer working for Snowflake,
and he is also a thought leaderin the industry and supported
and contributed to many opensource projects.
So sit down, grab your favoritedrink, and join us for a
(00:39):
fascinating conversation on howAI shapes our distributed data
infrastructure, and what aresome of exciting features that
are coming out specifically foriceberg?
It can make your life a lot moreeasier when thinking about
unstructured data, referencingother file formats from your
catalog or from your tables, andgenerally making AI workloads
(01:03):
accessible, available, and fast.
Yes.
Also fast because we need totake care of streaming
architecture.
Hey Russell.
SPEAKER_01 (01:19):
Hey Addy.
SPEAKER_02 (01:21):
So good to see you.
It's been a while, and I knowyou've been busy recently.
So what's what's happening withiceberg?
SPEAKER_01 (01:31):
Yeah, I mean iceberg is going through a huge
transition time right now, asI'm I'm sure everyone is, right?
We we've come to this new worldwhere this LLM-based AI stuff
and agentic things iseverywhere.
It's part of everything, uh,part of coding, part of
reviewing, part of productmanagement.
And iceberg is changing to kindof meet that need.
Uh one of the cool things thatwe're working on right now is
(01:53):
Iceberg format version v4.
And the two big targets for V4of Iceberg are to make it a
great solution for AI use casesand for streaming use cases.
And this is really showing thehistory of the project, right?
It started as this format thatwas really oriented towards this
(02:13):
kind of hive replacement, trenoreplacement, data warehouse
replacement.
And that has a really differentcharacteristic from streaming
workloads, as I know you'reextremely familiar with, and
these new AI workloads, whichare coming in with some
multimodal things, we're seeingvector stores, we have all kinds
of different uh challenges thatwe need to kind of address.
And one of the coolest thingsabout working in open source,
(02:36):
especially on something likeiceberg, is we can take ideas
from all over the community, allover the industry, and integrate
them into something thathopefully ends up working for
everybody.
So I'm very excited right nowabout Iceberg V4.
SPEAKER_02 (02:49):
Cool.
Yeah, it's an exciting time tobe working on the infrastructure
for sure.
I think it's kind of coming backnow with AI.
We're realizing that, hey, datainfrastructure does matter in
the age of AI and we do need uhspecific, you have specific, we
have now specific requirementsfrom our infrastructure, whether
(03:10):
it be analytics or a streamingum space, but we do need other
stuff.
So what are the new things in inv4 that enables us to run AI and
also whether some of the thingsyou know we should have in it?
SPEAKER_01 (03:28):
Yeah, I mean, so the things that we have pretty much
consensus on now for v4.
And since we're a communityproject and we don't we don't
actually have a v4 until thewhole community gets together
and votes and says, that's whatv4 is.
Everything I'm saying is prettymuch what is the current state
of the project.
But the biggest things we'redoing for AI enablement here is
(03:49):
we are adopting a method ofbasically going uh vertical with
your changes to a table.
So previously, the way we alwayslike to think about tables is
you know, we've basically gotthis relational table, and every
time you add a new row, you'rejust adding things onto the
bottom, right?
And then iceberg later added theability to do acquit uh delete
files and uh delete vectors, andthat'll just let you knock out
(04:12):
rows.
But again, these are horizontalcuts to your table.
What we didn't have is theability to divide your table up
vertically.
So basically saying thesecolumns are in one file, these
columns are in another file, andthen the ability to replace a
column within a file withoutdeleting it.
So basically, previously, everytime you wanted to get rid of a
column, you had to rewrite theentire data file and all of the
(04:35):
other columns that were adjacentto it.
Uh, this is really bad for a lotof AI use cases where you're
essentially maybe adding a newfeature every day.
And that feature is essentiallyjust an extremely long uh set of
vectors or set of doubles orfloats or whatever.
It's it's usually this maybesparse, maybe dense, but
(04:56):
columnar addition to your table.
So basically, instead of our oldtables where we added at the
bottom, we are now addingsideways.
So as we go sideways, we need toadjust the table format to
handle that sort of thing.
And this is, you know, uh anapproach that's been taken by a
few other projects, and we'reexcited to kind of make it into
a first-class feature withinApache Iceberg.
(05:16):
Um, the other big discussionthat we keep having is how are
we going to better deal with themultimodal aspect of a lot of
these table formats?
And there's there's two bigapproaches that I think are kind
of going being being worked onin conjunction between the
Apache Parquet community, theApache Iceberg community, and a
(05:36):
lot of other communities, wherewe're kind of trying to figure
out is this a file formatresponsibility or is this a
table format responsibility?
Because there's there's twomodes, at least I think, of
going about handling that.
And one is we embed ourmultimodal, our non-structured
data directly in the same kindof data files that we are
(05:57):
storing, everything else.
So this is an approach that Ithink like Lance takes, and uh I
think some other formats arealso trying to do this this kind
of approach.
Um and that's one thing you cando.
We could do the same thing withparquet, and we're kind of
looking at parquet about does itmake sense to improve the way
we're storing things so that wecan do something similar, and
(06:17):
then we can use again within thetable format the ability to
store columns separately to putlarger objects inside of
completely different files orsomething like that.
The other approach, which Ithink is really interesting, is
actually just using a referencewithin your relational table to
uh unstructured data that is nota part of the row structure
(06:38):
itself.
So this is basically saying, whyare we even putting this data,
which we know is actually verycompressible and very, you know,
good in its current formats?
Like a PNG is pretty great as aPNG file.
You can zip a thousand of themtogether and get pretty good
behavior if they're all prettysimilar in structure.
Why do we want to move that intoa structured file that has a lot
(07:00):
of other things in it?
So there's a question of whetherwe could just store references
to those files within arelational table and then have
the table format be responsiblefor the lifecycle management.
So, like whether or not youactually have credentials and
access to get to those externalfiles or when to remove them
because there's no longer anyreferences to them in the table
and things like that.
(07:20):
So there's two directions Ithink that Iceberg is going
there.
So we're we're obviously workingvery closely with the people in
the parquet community to try tomake sure that we have support
for storing this kind of data.
And also we're approaching thatexternal reference idea as well
to kind of see if that is theright way to go uh at the same
(07:40):
time.
Because, you know, I thinkthere's there's different, there
might be different fits fordifferent types of multimodal
data.
SPEAKER_02 (07:48):
Yeah, no, it's um it makes it really interesting.
It makes a really interestingproblem.
I remember back in the days, andand maybe that's a little bit
old, but if you know, we heldreferences inside a table for
for years, right?
And the always the challenge waswho manages that life cycle to
the way you that you mentioned,because it's um there is a
(08:09):
question of transaction andatomicity of that transaction
that we need to think about.
Like now the file got changed.
How long does it take tomanifest inside the table that
holds the reference that thatfile now got changed?
And which side is responsible uhfor that atomicity to happen?
(08:33):
And how do we pair these twosystems and perhaps these even
two tables together in order tomake validate that that actually
the case?
Um Yeah.
SPEAKER_01 (08:44):
I mean that's why I think it's really like a table
format level problem.
Like if we can define that atthis multi-file level, then I
think we we can actually goforward pretty confidently and
say like these are expectedbehaviors, and then we can have
engines and everyone kind ofwork within that framework.
Because I I know that we we didthe same thing back in the day,
and a lot of jobs I've had, andlike if everyone's doing it on
(09:07):
their own, that's the problem,right?
Because we end up with a ton ofdivergent things and none of
them work together because theway you do it doesn't work the
way I'm doing it, and you end upwith atomicity and consistency
issues and broken guarantees.
SPEAKER_02 (09:20):
So yeah.
And from my experience workingwith iceberg, and I also
reviewed a couple of reallyreally great books from from you
know, great authors from ourcommunity.
I had the opportunity tocontribute there as well, kind
of like more behind the scenesas more of a technical reviewer.
Um, there is the notion ofbranches in iceberg today.
SPEAKER_01 (09:43):
Yeah, I mean, iceberg already has a lot of
those components we would evenwant to do this sort of thing,
where we're where you want tojust build a completely separate
branch, do all of your testingthere, and maybe do your
management of files, and thenonly merge it back in when
you're like, I'm sure the statehere is good.
I mean, I haven't written up anyideas on how we're actually
(10:04):
going to do that for thesereference types.
Because right now, you know, thequestion is what should be in a
reference type, where should wedefine that?
Um, we've had some proposalsthat maybe we should even put
the base definition of what thatstructure looks like in parquet,
just so that we have a kind ofglobal definition that isn't as
table-biased, and then we canput our rules around it inside
(10:26):
of iceberg.
SPEAKER_02 (10:28):
Right, right.
Well, you know, and if I look atKubernetes world, if I look at
uh things like Kafka, uh RocksDBas a source of truth is always
uh you know a solid idea anddirection.
But you know, I I'd be curiousto follow along and see how that
works.
SPEAKER_00 (10:47):
Now a quick word from our sponsor.
Confluent developer the podcastis brought to you by Confluent
Developer the Website, which haseverything you need as a
developer of data streamingsystems.
And it's completely free.
We've got curriculum, hands-onexercises, executable tutorials,
the online data streamingengineer certification, also
free.
A way to find a meetup near you,those are free.
(11:10):
Everything is there.
I really want you to besuccessful in your journey as a
data streaming engineer, andthis is the site that has what
you need.
Check it out atdeveloper.confluent.io.
That's developer.confluent.io.
Now back to the show.
SPEAKER_02 (11:27):
Yeah, and also that there is the question, uh I
think you touched about it alittle bit, of streaming,
because adding another layer ofatomicity and adding in perhaps
another source of truth databasethat is strongly consistent um
adds latency.
SPEAKER_01 (11:48):
Yeah.
I'm really hoping we don't haveto do a second system to make
this work.
I think we we may design it withlimitations that make it so that
you we just don't need that.
Um, you know, do immutabilityand and other stuff like that,
where you know, there is onestate of the table, the files
have to be in the same state fortheir entire lifetime.
You may not change the state ofa file, offsets should always be
(12:10):
true, you know, that that kindof stuff.
For streaming, though, withiniceberg v4, the things that
we're excited about is we'rereally trying to break down the
amount of time it takes to do acommit.
And that's one of the other bigv4 changes, is we're changing
the structure of the metadatatree.
So it uses almost all the sameideas, but pretty much anywhere
where we couldn't cachesomething before because we were
(12:33):
changing it so often, we'rewe're breaking that and making
it more immutable, more lasting.
So basically, we looked at allof these ideas that we've been
applying to our data files, likethe ability to knock out a row
in a file without deleting it,and the ability now to replace a
column in a file withoutdeleting that whole file.
We're like, well, why don't wetake those approaches and apply
(12:54):
them to the metadata layer aswell?
So when you do that, suddenlywe're we're basically gonna set
it up, and and I don't want togo too deep in the technical
details of this, but there'll beone file that's basically your
entrance into the rest of themetadata tree that will be
completely overwritten on everycommit.
And we're gonna keep that verysmall.
So something like 400 kilobytesis what I keep throwing around,
(13:15):
because on our tests we saw thatyou know, doing IO to S3, you
basically have no differencebetween like four kilobytes and
400 kilobytes.
I mean, there's a slightdifference as you as you go up
and up, but you know, if ifyou're going to open a handle to
S3, you might as well uh giveread that many bytes.
Anyway, so you you read thatmany bytes back and you're gonna
replace that every time.
(13:35):
And when you add new files,you'll add them directly into
that layer.
So the way you used to have todo it is if you wanted to add a
new file, you had to also writeanother file that a manifest
file that it would refer to.
And then the data file would beunderneath that.
And that meant that when youwere creating a snapshot, you
had this serial process wherefirst you made this file that
(13:56):
had a listing of the data file,and then you made another file
that referenced that file.
And because that's you know aserial process, you basically
had to pause each time toactually do a commit.
Now you'll basically say, I'monly overriding that top-level
file, which hopefully you havecached in a memory or like a you
know something like that.
And you quickly change it, youwrite that new file, and you're
(14:19):
done.
So for a streaming use case now,you should have only one IO
operation required to do yourcommit each time, as opposed to
two that have to happen inserial.
So hopefully that helps a lot.
And then the other thing we'redoing, right, is that because
this will always be in thistiny, tiny layer up at the top
of this metadata tree, thatshould also be a pretty low
latency operation, as opposed tohaving to do some more drastic
(14:43):
restructuring.
One good example of this is ifyou wanted to ever delete
something before out of aniceberg table, you basically had
a manifest that had an entrythat said this file exists.
To remove that entry, you had todelete that manifest file.
You had to completely rewriteit.
Because if it still existed, youwould always have this little
entry that says that fileexists.
(15:04):
So we would have to delete itand write a new version of it.
Now, since we're using thoseabilities that we had on data
files at the metadata layer, Ican just say I'm taking that
delete and I'm applying it to amanifest file and just have a
little marker.
So basically a delete vectorthat says that row doesn't
exist.
So we can basically, withoutdisrupting your cache, that file
(15:24):
is still cacheable, you canstill keep it in memory.
You can mark things as deleted.
So suddenly deletes are way, wayfaster than they used to be at
this metadata commit layer.
So those are the kind of thingswe're pulling together.
There's a lot more details onhow we're kind of structuring
this, but the end goal is tohave that commit process be fast
enough that you can start doingvery rapid commits to an iceberg
(15:46):
table if you really want to.
SPEAKER_02 (15:48):
That's awesome.
I love the idea of caching, andit uh reminds me a little bit of
uh Hadoop world back in the daysand some of the latency
challenges uh we had with uhwith HDFS and uh and yarn and
it's the rest of the zoo.
SPEAKER_01 (16:02):
Uh yeah, I mean we don't want to do I.
That's the story of all of thesetechnologies, right?
Is we're just trying to avoidI.O.
all the time.
It's what can you not read?
What can you not write?
What can you cache and not haveto access again?
SPEAKER_02 (16:20):
Right.
We've learned as as an industry,as an open source community for
sure.
And I think AI pushes us to forthat as well.
I mean, latency is waiting forsolutions like cloud code or you
know uh other other solutionslike vertex AI and uh Gemini and
(16:45):
and so on, waiting for them toactually uh give us back a
response starts to be you knowfrustrating.
It's like I gave you a prompt,can you can you make it happen?
SPEAKER_01 (16:57):
This is this is actually one of the reasons that
I of all of the technologiesthat I think people are working
on right now, I feel like tableformats will probably be the
least changed because they're atthis layer where you need to
interopt with them.
And if you have to, like, Imean, right now we're in a great
time, right?
You can basically say, pleasewrite me an interop layer
between these two type fileformats, or write me something
(17:19):
that converts this query intoanother type of query.
But the problem is theunderlying data structure is
still held by whatever itsoriginal limitations are, and
all those layers that you buildin between just make it worse.
So we need to have theseinteroperable formats at the
bottom of everything, becauseotherwise, regardless of how
fast your models are or howsmart they are, they still will
(17:39):
have that translation cost orthat modification cost.
And the only way to make itbetter is to fix the I.O.
at the at the very bottom layerof it.
So I think it's definitelyinteresting time.
SPEAKER_02 (17:54):
Yeah, and and that requires, you know, you know,
solid engineering, a lot of thework that you do with the open
source community and uh bringingdifferent people from different
organizations together.
I believe this is where whenpeople ask me, like, what's why
community, right, why opensource?
I always say this is this is thepower.
It's like you um it's a hive ofminds that comes from across uh
(18:20):
you know the industry and makedecisions and discuss the
challenges and uh go very deepinto the technology and the
nuances of you know the exactformat, the solutions, what do
we need, and what's gonna propelus forward and um be a solution
that it's kind of solves theproblem rather than a company
(18:41):
specific that sometimes can beyou know very siloed.
SPEAKER_01 (18:45):
Yeah, I mean that's that's the danger with working
on you know a company specificproject is even even if you have
the best intentioned engineers,there's so much pressure from
you know the the folks with thedollar signs and the C titles,
right?
We're like, you know what, butcouldn't we just delay this?
Or you know, shouldn't that beonly available to our users?
Or oh, if we didn't do that,wouldn't it make it harder for
(19:07):
our competitors to use the samething?
And when you're in a true opensource governance, and I mean
that's one of the things I loveabout the iceberg community, is
you just can't do that.
Because even if one person says,hey, I'd really love to do this
thing that would make it harderfor everyone else, you'll get
nine people who will say, No,that will make it harder for me
in my job.
So I'm not gonna support you indoing that.
You better have a really goodtechnical reason why we should
(19:30):
all take on that burden.
Um, and it the reason can't beit'll make you more money in Q1.
SPEAKER_02 (19:36):
So Yeah, it's I think as engineers you want to
solve the problem.
Um and sometimes organizationsproblem across the board, it's
like, hey, we we want to makemoney.
And then the question is like,how do we how do we support that
motion in the uh while stillsolving the real the real
engineering problems?
SPEAKER_01 (19:57):
Yeah, I mean I like again, that's another thing I
kind of like about the formats,is that the only the real value
of iceberg at its core is thisinteroperability and this open
governance layer.
So all the people in it keepthat as like a very strong focus
because there's really no pointin supporting iceberg if you're
the only one who supportsiceberg.
(20:17):
We we want to support icebergbecause everyone supports
iceberg, which means that youknow the the pitch I do all the
time internally is I'm like,every person who's using iceberg
is a zero friction customer toSnowflake, potentially, because
they are using the four method.
We can integrate directly.
There's no ETL, there's noconvergence, whatever tools they
(20:38):
have can keep working while theytry POCs and things like that.
And every other vendor, I think,sees it kind of similarly.
This is the open playing field,which means that everyone can
try whatever they want.
And of course, from an end userperspective, I really love it
because that makes everyonecompete to make sure that this
layer really works well and isreally compatible.
And I think the the generalindustry benefits a ton from
(21:02):
having this like communityproject, similar to, you know,
like HTTP standard and thingslike that, where like just
having an industry standardwhere people can all agree
benefits end users way more thanit benefits the end the
companies, but they can alsotake advantage of that because
you know everyone who'scompatible can build up whatever
their offering is on top of it.
SPEAKER_02 (21:23):
Yeah, yeah, I love that.
And it's so true.
It's faster adoption as well asfaster POC.
Imagine if a customer now wantsto see it's you know, if they
they want to buy a new softwaresolution, right?
We do it all the time, weevaluate software.
We want to make sure we canactually build that POC
relatively fast and see thevalue fast, if you know, if we
(21:47):
can.
And and if that's not the caseand it's too challenging, then
we flag it.
Hey, there's gonna be a steeplearning curves, here are the
integration challenges.
This is something that we expectyou know the software company to
solve.
uh for us to integrate with thatwith them.
Um and then it's it becomes kindof like a question of back and
forth and if it's the bestsolution for us.
And of course there are othercomp competitions, right?
(22:10):
The competitors on the marketthat might have a better,
simpler to integrate, simpler torun a POC solution.
It's gonna take us as a customerand this is uh it's a huge huge
power for for open sourcebecause um if we can enable that
(22:32):
then we have a faster path towin.
So that's true.
Cool.
So we talked a little bit aboutAI and some of the AI workloads
and some of the requirementsespecially around referencing
when discussing you knowunstructured data and then we
need to dive and kind of bringuh unstructured data um that is
(22:52):
compressed and that you knowpart of the table is kind of
like a binary nasty binary uhformat um and we talked about a
little bit transactionabilityand optimicity and so on and a
little bit about streaming.
I know you know you mentionedthere is um icebergs of support
(23:14):
bringing in your own catalog aswell right there's a dedicated
um API and there's a dedicatedspec that you can work against
uh as a developer to bring yourcatalog and recently I've been
seeing more and more uh aboutsome of the catalogs that you've
been working on uh and I'd loveI'd love to learn more and also
(23:36):
you know how you see thatconnects with maybe the new
format that you know you'redeveloping.
SPEAKER_01 (23:42):
Well I mean the the iceberg rest specification is
also growing a ton right now andas you mentioned it's basically
the the lingua franca we've madefor integrating with an iceberg
catalog which is a completelyseparate service we basically
define this is how you will talkto a catalog for an iceberg
table this is how you will getback metadata this is how you
will propose new table updatesall the all that kind of stuff
(24:04):
and what's really great abouthaving this API so well defined
is that now it's just reallyeasy to move your catalogs uh
your your tables from onecatalog to another so in the old
days before we had this youbasically were like I'll pick
hive and you pick hive and youconfigure everything
specifically for hive all ofyour clients use hive specific
(24:25):
code hive specific versionthings and what if you had a an
issue you really had no way tocontrol anything because
everyone is talking a hivespecific language to your
catalog and your Hive Metastoredoesn't really know it's dealing
with iceberg tables.
So there's no way todifferentiate what's happening
or tell the client like oh no Ithink you're using a bad version
(24:46):
of this iceberg library becauseyou're using a Hive library and
all you have is this thriftrequest.
So moving to the rest suddenlyallowed everyone to to put more
information in their in theirrequests and have all of these
catalogs have this front endthat says okay I'm dealing with
an iceberg client oh this is abad version I'm just not going
to accept requests from you.
(25:06):
I'm gonna send back an errormessage that you need to
upgrade.
Or you can take requests and youcan say well I have all my
commit logic now internallybecause in the old days
basically the the the clientwould stage the commit using the
catalog's own specific API.
So for example for Hive youwould say Hive I need you to
lock this table reference fromthe client and then on the
(25:29):
client you would check to seewhether you had a valid commit
and then you would then swap thetable metadata using an alter
command and then unlock thetable with a direct hive API
again.
So it's all very hive specificand it's all done on this
client.
So the clients can propagateweird errors if your client
doesn't know how to interactwith the underlying storage.
(25:51):
So this is why I really love theREST API, because it puts a
layer in between the client andthe actual persistence for your
table entities which means thatit's it's much easier to
centralize fixes for this sideof thing or stop badly behaved
clients from doing something anddisrupting your whole setup.
It also of course completelydetaches authentication from all
(26:13):
of the backing storages so youcan have whatever auth you want.
You know if you're using HMSbefore you basically had to use
Kerberos right like that wasyour that was going to be your
security system sorry that'sjust it but if you talk with a
REST client right it's whateverthat catalog decides to expose.
So you know we have a lot ofOAuth things so I work on Apache
(26:34):
Polaris which is a project thatuh uh a while back was was
started by Snowflake and theythey basically as I was hired
and started working on icebergthere were like we need to be
more open source we can'tcontrol these things we need to
have an open community that'sbehind them so they took the
Polaris project they donated toApache and now we're just one of
(26:55):
many contributors to thisproject which you know because
it has the REST API iscompatible with iceberg clients
all over the place and providesall kinds of awesome features
and integrations with all kindsof identity providers and
offlayers.
So you know it's it's reallygreat to kind of be in this
space now where if you want toyou can use Polaris as your
(27:17):
catalog.
Or if you want to switch you cango find another producer of an
Apache iceberg catalog thatservices the REST API and your
switch is actually pretty easybecause the client is exactly
the same.
It's an iceberg REST client.
So again a lot of cool benefitsthat come to any random user and
it's nice to just be able tokeep working on open source even
(27:38):
at that uh part of the stack.
SPEAKER_02 (27:41):
Yeah it's it's pretty amazing catalogs has been
I you know forever kind of likethe war of catalogs and who
builds a better catalog and uhI'm I'm curious because you did
mention Hive and HMS.
How does Polaris and you know weall had our fair share of uh of
working with Hive and fixingtables and and the references
(28:03):
and the and and so on.
How would you say Polaris isdifferent today?
SPEAKER_01 (28:12):
So I mean Polaris was built kind of for this new
iceberg Rest world as like afirst class citizen.
I I do want to say HiveMetastore also is a has a first
class iceberg REST API rightnow.
So you don't even have to talkto Hive using the Hive client.
You can actually talk to it withthe REST API.
The biggest differences are theway Polaris is built is
(28:32):
basically structured aroundiceberg metadata as a first
class construct.
You know it basically assumesthat you're going to talk to it,
you're gonna talk to this RESTAPI and your security, your auth
and all of these are basicallypluggable components into that
system.
So you can decide whether you'regoing to use you know OAuth 2 or
something or some other layer ofsecurity as opposed to in the
(28:54):
Hive world where it's verystrictly tied into the Hadoop
ecosystem.
So you know what's what'ssupported is what's in Hadoop.
And because of that you knowlike I said Polaris can support
Ved did credentials and supportvarious cloud providers with
pretty pretty easily and um oneof the other cool things is we
(29:14):
also are able to because wedidn't assume right that the
catalog actually understands howto communicate with these tables
right you basically surface upthis iceberg metadata whereas in
the Hive Metastore we have inputformats that are like bound to
every single table.
So there's actually like literalcode you are supposed to read a
table with stored into thecatalog and you have to know
(29:36):
that you can load that class andeverything.
But we've got support for awhole bunch of other table
formats now in Polaris.
So you can actually use Lancetables in Polaris you can use
hoodie tables.
I'm forgetting what we justadded but um you know there's
all that cool stuff.
And then the pluggability alsoexists on the backend.
So Polaris itself has apersistence layer and we made
(29:58):
that persistence layer pluggableas well.
So you can choose any relationaldatabase you want to be the
persistence layer or you can usesomething like Mongo or you can
use BigQuery like you can nowhook that up to NoSQL things as
well and have that be thebackend for your your system.
So it's it's really nice there.
So I I'd say the biggestdifferences there are those kind
(30:19):
of pluggability layers thatdidn't really exist within the
original Hive code and of coursealmost no connection to the
Hadoop ecosystem so nodependencies on Kerberos or or
strong ties to Java.
SPEAKER_02 (30:32):
That's really cool.
No strong ties to Java it's uhthat's a really good way to uh
to segment that beyond you knowthe the the fact that we have a
REST API actually now that youmention it I have two questions
uh I hope you're ready let'sstart with the strong ties to
Java um a lot of the datacommunity and data practitioners
(30:55):
right whether they're your dataengineers a data science a data
analyst uh analytics and there'sso many titles today um kind of
starting using more Python rightin a lot of resolutions in the
data space are actually werewere built with you know JVM in
mind and and and Java and sothat uh interoperability between
(31:17):
those two frameworks especiallyaround performance right like
how do we build that um wasalways uh a little bit of tricky
it's like you don't want torewrite your pipelines in Java
and yet again you do want to getthe highest performance possible
um and the fact that youmentioned that I think that's
(31:40):
that's a really good val addedvalue to Polaris as a catalog
because uh uh there's a lot offriction there.
SPEAKER_01 (31:49):
Um I wonder you know if there's more things that you
know made the community to makethis decision beyond what I mean
I doubt that was one of the mainreasons behind the REST API in
the first place right likethere's there was almost no way
to get a Python client withsecurity to work properly with a
Hive purely Hive interface rightbecause you you had to get the
(32:11):
basically you have to pull inall of Kerberos to make that
work properly.
And that just didn't exist.
And the moment we have the RESTAPI up suddenly you could write
a Go client a Python client a Cclient a Rust client a Java
client and they don't have toknow anything about Java to work
properly with the catalog.
So I mean you'll see right nowthe Apache project for Iceberg
(32:34):
has all of those implementationscurrently being worked on.
And it's really exciting to meit and one thing I've seen
recently too and I don't knowwhat if we're we're we aren't
going to see this for a whilebut a lot of folks have told me
and I've seen some benchmarksthat prove this that the the
LLMs and all the coding agentsare actually really good at
writing Go because of howopinionated Go is.
(32:56):
So the the difference there isthat when you write Python right
there's many ways to writesomething right there's there's
many many right ways to do a lotof things in Python.
You've got list comprehensionsand map functions and all of
these like different structures.
But Go does not have that sameflexibility.
So some rank rankings I've seenhave shown that these agents are
(33:17):
are way better at writing Gothan they are at writing
anything else.
So I'm wondering if we'llactually see a big push for more
Go libraries.
I mean it also doesn't reallymatter because it's it's really
a difference in token cost,right?
On how long you let it churn andlike write things and have to
fix its own mistakes.
But once we get to the pointwhere LLMs are are commoditized
(33:40):
and we actually are playing thetrue true cost for all of this
I'm sure that's the we'll we'llget a big push into to working
with more efficient languagesfor LLMs even if they are not
the favorite languages ofhumans.
SPEAKER_02 (33:52):
Yeah that's a really interesting observation that I
must agree with it's the cost oftokens as uh you know like
leaders like Jensen said it'syou know engineers we're gonna
start negotiating that in ourcontracts like how many tokens
am I gonna get uh as part of mycompensation package so I can
actually you know perform at thelevel I want to perform for my
(34:14):
boss right yeah I think I thinkalmost no one is really is
really leaning hard on what thatcost is going to be because I
again I do think we're at awe're at a point now where uh
VCs are like paying a lot of ourtoken costs right now.
SPEAKER_01 (34:29):
Like your our tokens I think are very subsidized.
And we will get to a point wherethat will not be the case and
we'll probably get this hugepushback to be like why are you
using why are you using youryour engine to write C code when
you could be writing Go code oror Rust code which we know the
engine we know the LLM is betterat more efficient at and can do
for a cheaper token cost.
(34:50):
It'll be much harder to justifylanguage decisions.
I mean I I've wondered a lot onwhat will the future even be of
of having all these differentcoding languages in a world
where the LLM can write all ofthem pretty much equivalently
well with these slight costdifferences.
You know it it'll it'll be aninteresting world we'll we'll be
(35:10):
in because like we could haveLLMs just write assembly like
directly if we wanted them tobut we obviously that's probably
not the most cost efficientthing and I think that's what
will will end up being the thefinal trade-off is what language
is the cheapest to writeefficient code in.
SPEAKER_02 (35:27):
Right token economy for engineers maybe should you
know maybe you should write abook about that could be
interesting.
SPEAKER_01 (35:35):
Everything is churning so much right now that
it's hard for me to say I meanbut I I definitely think that
that it will come up in the nextnext few years people will start
I mean I I couldn't I wouldn'tbe surprised if someone writes
even a a language specificallyfor LLMs to write that they can
do more efficiently or has theright kind of you know built-in
(35:58):
checks that let it do one shotsbetter right like that's that's
really the goal.
You want it to be able to dothis with the smallest token
cost.
And anytime it writes somethingwhere there's a possible syntax
error like that's that's a badlanguage for an LLM to write
code in.
SPEAKER_02 (36:11):
Right and and that happens a lot.
I had another question.
So we has we have the REST APIfor uh for working with iceberg
and you know mcp uh is kind ofon the rise I wonder if this is
something the community lookedinto it's like having a
dedicated mcp server instead ofuh you know writing dedicated
(36:35):
you know working directly withthe Rust API.
SPEAKER_01 (36:37):
So the Apache Polaris community actually
already did and built a MCPserver to go along with Polaris.
So if you want to I believe it'sin our tools repository now so
you can basically stand it up atthe same time and have your
Polaris catalog directly MCPenabled out of the box if you
want.
So it's definitely somethingwe're thinking about from the
server perspective.
(36:57):
As like an iceberg projectitself there you know we don't
really have any servercomponents there.
So there's nothing really there.
But we actually we just mergedour our first agents md file so
I was kind of excited about thatbecause I think we were a little
slow to the game to to addingthat we we've been seeing the AI
PRs, we've been seeing the AIissues and uh you know we're
(37:20):
entering a world where I thinkpeople do realize have to
realize that they need stillneed a human in the loop.
I mean I all the buzz is great,right?
But there's a lot of times whenAI will do things in ways that
don't make sense to a humanreviewer and you need a human to
check a lot of these thingsespecially on a code base where
we can't have it growindefinitely in the way that I
(37:42):
think a lot of vibe codedprojects can grow.
You know like that it's fine ifa vibe coded project adds like
10 extra classes when only onewas required.
But for something like icebergwe need to keep it tight not
only so that we can keep workingon it and understand it but so
that LLMs can continue workingon it understand it.
We can't turn it had turn itinto a hairball which makes it
(38:03):
impossible to work on.
So you know we're trying totrying to work on that.
SPEAKER_02 (38:08):
That's that's a solid point and I see it across
the open source community withuh specifically Kafka and Spark
and Flink and so um and thinkingabout this token economy at the
end of the day it's it's gonnacost us now it's relatively
accessible I would say um butyou know who knows what will
(38:29):
happen a couple of days monthsyears down the road yeah I can't
make economic predictions butyou know people are spending a
lot of money right now and theyneed to make a lot more money
back.
SPEAKER_01 (38:41):
So that money's got to come from somebody and it's
probably gonna be us.
SPEAKER_02 (38:45):
Yes 100% all right thank you so much um is there
anything you know we have a lotof people listening in that
thinking about their careersthat they're in the data REI
space in the open source spaceand so on.
If you would to look back andgive them you know one or two
pieces of career advice whatwhat would that be?
SPEAKER_01 (39:09):
I mean this this is a hard time to give career
advice because I feel likeeverything is is shifting again.
But the thing that has served mewell and and again this is a big
your mileage may vary sort ofthing is I've I've always tried
to take whatever job or whateveropportunity will get me closer
to the thing that I actuallywant to do.
(39:29):
So if you look at my LinkedInyou'll see I started out in as a
software engineer and test atData Stacks because that was the
first position the firstinterview I got at a company
that was doing distributedcomputing that would talk to me,
would even let me in the frontdoor.
And I did not care.
As long as someone would get mea little bit closer to
(39:50):
distributed computing I wouldtake that job and I applied all
over the place and that was whatwhat got me in the door and
that's my my same advice I thinkto to almost everyone I talk to
who's entering computer scienceis you you got to just take
whatever opportunity you canthat gets you one step closer.
Don't be afraid of not going allthe way to what you want to do
in your very first job or yourvery next project.
(40:13):
Just make sure that it's it'sone step closer to the thing you
want to do.
SPEAKER_02 (40:18):
I love it.
I love it.
Russell thank you so much alwaysa pleasure speaking with you I
always you know get out of theseconversations with tons of
insights and of course morequestions uh uh and and learning
so really appreciate you takingthe time to uh jump on this call
of course it's been great thanksso much for time
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Bobby Bones Show
Listen to 'The Bobby Bones Show' by downloading the daily full replay.