Ep37: Vector Database Frustration + Microsoft LazyGraphRAG

In this episode, I discuss my challenges exploring vector databases for an upcoming demo. From what is a vector database to integration issues, hear how I tried a few different approaches with limited success and discover the surprising one with the most promise. I also explore Microsoft's "Lazy Graph RAG" approach, which seems to trade one challenge for another but could be valuable in certain cases.

Episode Transcript

(00:05):
You are listening to the BreaktimeTech Talks podcast, a bite sized tech
podcast for busy developers, where we'llbriefly cover technical topics, new
snippets, and more in short time blocks.
I'm your host, Jennifer Reif, anavid developer and problem solver
with special interest in data,learning, and all things technology.
It's been a bit of a week over here, fromdaylight savings time last weekend to

(00:29):
just some typical developer frustrations,this week has put me through the ringer.
But, I've been stressed all week aboutupcoming projects and tasks that I'm
working on, and after chatting witha colleague, I'm seeing that it's not
just me feeling some of these things.
Which is a good reminder to putthings in perspective and escape a
little bit of the imposter syndromeby reminding myself that others

(00:50):
are experiencing similar things.
First of all this week, I wantto give a huge shout out to
the Devnexus conference.
It was a fabulous event that Iattended last week, and I gave a keynote
presentation there which I loved doing.
I was a little nervous about, butI was super thrilled and excited
to do it and then was superhappy about it afterwards.

(01:11):
I had some really great feedback.
I got to talk with some really amazingpeople and just wonderful people in
the community doing really fun andexciting and inspirational things.
That was a ton of fun.
I made a whole lot of newconnections and got to hear some
ideas and inspiration as well.
But the first topic that I starteddigging in hard on this week was some
vector database exploration and a fewfrustrations I ran into along the way, too.

(01:36):
I'm working on this for anupcoming presentation and demo.
I'm looking at vector databasesand graph databases and RAG and how
all of those kind of play together.
I kind of went back to the basics.
What is a vector database?
I pulled up an article from Pineconebecause initially, when AI hit the
market a couple of years back,Pinecone became the default leader

(01:58):
in the space for vector databases.
And I'd spun up an account a coupleof years back on Pinecone and had
spun up a database and then justnever really did much with it.
So I circled back on that this weekand thought I would explore Pinecone
a little bit more just seeing whatvector databases look like, how they
index, what their data looks like, whatthe similarity searches do, and so on.

(02:18):
But there were a few issuesI ran into along the way.
The first thing was that I got thedatabase up and running with no
problem, and I wanted to connect to itin any way that I felt like I could.
And so I went out to thedocumentation, and I was super excited.
They have a Java SDK.
So I thought, great, Iwill write a Jbang script.

(02:39):
I'll connect using their Java SDKto Pinecone and kind of work that
way and see where things are.
And I spun up the instance, I tried theJava SDK, but the first thing that I
ran across was that their API referencedocumentation is all based around Python,
JavaScript, and curl, which is totallyfine. But I was struggling a little bit

(03:00):
figuring out the syntax that I needed,how the code might look a little bit
differently in Java, and so on, andended up running into some issues where
some of the functions and methods thatwere shown in the Python, JavaScript,
and cURL shell availability APIs arenot available in some of the other
languages, including Java, of course.
Some of the functionality thatwas available in their API

(03:22):
was not available via Java.
So there were just some things like,for instance, to list the indexes.
I could list the index, get a, an array ofresults back of the indexes I had on the
database, but I couldn't actually see if aparticular index was in that list because
I couldn't sift through the indexesand like loop through them, basically.

(03:42):
I would have to parse the resultobject coming back and sift through
it very manually, almost text-based.
So again, the, there were lots morefunctionalities available in some
of the other SDKs, but just the Javaone was a little bit reduced and
not as full fledged, I will say.

(04:04):
I thought, okay, I ran into somewalls there, so I thought, well,
I know Spring AI has reallygreat integration with databases.
I know I can load all kinds offiles and things through that.
Let me just spin up a Spring AI app, withChroma, because I had heard that that
vector database, especially with SpringAI, is super easy to just configure
and auto configure out of the box.
So I spun that up, spun up aSpring Boot project, but was

(04:26):
struggling to get the applicationto connect to the database at all.
Like even just a basic, here's thedependency, it should roll up an
embedded, and then putting in theproperty in the application.properties
file for the initialized schema.
That's all the documentationwas showing was necessary.
But I couldn't even connect.

(04:47):
It was showing that the connectionwould close on the database and so on.
I didn't really dig terribly far intothat because then I also saw that Spring
AI integrates with Pinecone as well.
So I spun up and just switchedout the dependency for Pinecone.
And I was able to connect the databasereally easily there, but then I
was running into some roadblockson how to manipulate some of the

(05:08):
JSON data to get the format thatPinecone is expecting in its database.
For instance, it does an ID,
a vector (key value), and thenmetadata (map), if you will.
And so I needed to convert myJSON into that format, an ID, a
vector, and a map of metadata.
And so I was struggling a little bitto figure out how to parse through

(05:28):
the JSON and figure out how to putinto, put it into an object that
then I could upsert into Pinecone.
I asked around a little bit insidemy company on options for this and
someone mentioned, Hey, we actuallyhave at Neo4j awesome procedures
on Cypher or our APOC library.
We have some procedures and functions thatwill connect directly to vector databases.

(05:49):
And I thought, great, thisfeels a little bit more natural.
I'm much more comfortablemanipulating data with Cypher
and then mapping it to the formatthat Pinecone is going to expect.
And I know how to manipulate andtransform that data in Cypher.
So this felt like a little bit more ofa natural, easy on ramp for me at least.
But I popped into the APOC andPinecone stuff and some of the

(06:09):
documentation's a little bit
behind maybe, or Pinecone haschanged their APIs, one of the two.
So there were some things missing.
There was lacking a little bit of clarityon the config, specifically from the
APOC side, on what I needed to provideand how that was supposed to look.
So I'm still running into some roadblocks.
I did finally get theconnection up and running.
I was able to create indexes and amable to put the data into something

(06:34):
Pinecone is looking for, but I'mstill not able to get it upserted.
But again, I'm making progress.
I will get there.
Working through it, justa little bit more slowly.
Overall, my thoughts and myexperiences on this. I tried multiple
different approaches, ran into a fewdifferent roadblocks along the way.
And it gave me a reminder that Neo4j,yes, from my, my point of view, because

(06:56):
I'm on the inside, I do see some of thegaps and the things that are missing.
But honestly, no product is perfect.
And there are things that Neo4j reallyhandles very well and at least matches
market on several other things.
And it really, actually, because it hasthe awesome procedures on Cypher, I felt,
and it was a more natural translation.
There's multiple options you cango at approaching this problem.

(07:19):
I love that Neo4j has an option that youcan use Cypher in order to port data over
to a vector database or vice versa, right?
You could pull data from a vectordatabase at a Neo4j, and it just gives
you more options for whatever feelsmost natural and comfortable for you.
Again, running into some roadblocks asidethere, hopefully those will get worked out
along the way. But also, AI is changingso, so fast, and I think that's one thing

(07:40):
that is hard for everyone to keep up on.
APIs are changing, libraries are changing,documentation's out of date, features are
getting added, and so much of that justgets the, the gaps and the things that
end up missing fall through the cracks.
And so then you have kind of userissues or frustrations on the other
end because stuff is just changingso fast, it's hard to keep up with.

(08:01):
Also the Java support and a lot ofthe generative AI or AI tools and
APIs is still a little bit laggingbehind things like JavaScript
or Python or even TypeScript.
And it's a little bit limited, whichis a little bit frustrating unless you
have options where you can delve intothe Python end of things and play with
the things that are going on in Python.

(08:22):
If you're dealing with Java and onlyJava, you are kind of limited a little
bit in certain aspects of the toolsand the capabilities that are available
because the Java libraries and toolsjust haven't quite caught up with
everything that's going on in thePython or GenAI front, if you will.
Then, the last piece of which isthat data and databases are the crux

(08:44):
of technology, and you can't escapesome of the pre processing steps, no
matter what database you have chosen.
Neo4j we see some of these frustrationsin a lot of the pre processing and
modeling and things that have togo into before you put data into
the graph, but we're seeing thatwith vector databases, we're seeing
that with other types of databases.
You can't completelyescape a preprocessing.

(09:06):
You have to take some form of rawdata, put it into a usable and
interesting format for your usecase, and then put that into your
database and get value from it.
So there's a little bit of thatwork that has to be done up front,
no matter what database you choose.
The content highlight I want to lookat today is I was trying to catch
up on all of the AI stuff, hence thevector side of things that we just

(09:28):
talked about. But I came across anarticle on Microsoft's LazyGraphRAG,
and this is something that has beenmentioned a couple of different times.
I heard a little bit about it atDevnexus and inside my company
as well, things floating around.
So I pulled out an articlethat was written on Medium,
talking a little bit about what it is.
And just from what I can understand,it's looking at the opportunity costs

(09:52):
of pre processing data for GraphRAGversus post processing data for GraphRAG.
This approach, a lazy GraphRagapproach, provides a way to handle the
processing at runtime or on the fly.
So one of the problems is that when youdo a lot of pre processing, so you have
data that's coming in, maybe a lot of datacoming in very quickly or in real time.

(10:15):
And you end up doing a lot of preprocessing steps to get that data
cleaned up, to put vectors andembeddings on it, to add maybe
some other layers of algorithms orcommunities or something like that.
And so there's a lot of upfrontwork that has to happen.
And then of course, maintenance,every time something changes or
things get added, you have torerun some of those things and do
that pre processing levels again.

(10:37):
LazyGraphRAG gives you an opportunityto only do those things at runtime
and only maybe in certain segmentsof the graph where you need them.
This is going to avoid the heavypreprocessing and the upfront maintenance.
The trade off that I see is that you'regoing to load that latency or your or that
cost into the actual runtime of things.

(10:58):
So when you actually run the queriesthat you need to, you're going to have
to front load those queries and add alittle bit more time ahead of the query
in order to do some of that preprocessing.
It might be worthwhile if you havevery, very large data sets, from what
I can tell, but maybe not for moremanageable or time sensitive results.
So if you have smaller graphs,personally, I would think that pre

(11:22):
processing ahead of time and just routinemaintenance would be more beneficial.
You would get probably faster query timeresults, than loading that processing
and kind of the pre processing and stuffinto the runtime or the query time.
So that was just my take on it, but feelfree to read the article for yourself
and check it out and see what you think.

(11:43):
I'm happy to take input or if someoneknows a little bit more in depth or
has explored this a little bit more,I'd love to hear your take on that.
Even though progress was slow thisweek, I am learning a lot, so that
makes it worthwhile in the long run.
Today, I recounted some frustrations andlearnings and exploring
trying to get data into a vector databasefrom a Java developers perspective.
Then I caught up on the recentMicrosoft LazyGraphRAG and the

(12:06):
opportunities and implicationsthat I see from an initial review.
As always, thanks forlistening and happy coding.