Ep66: Neo4j Data Loading at Scale + Vector Search Filtering

Hear about my hard-won lessons from loading a large-scale book dataset into Neo4j with Ollama embeddings, plus a preview of exciting new vector search features.

Highlights:

Data Loading Battle Stories

• Fixing Ollama OpenAI endpoint issues (drop the /v1 suffix!)
• Choosing embedding models with adequate context windows (nomic-embed-text: 8,192 tokens)
• Optimizing batch sizes and memory configuration
• Using EXPLAIN to identify and eliminate Cypher eager operations
• Error handling with ON ERROR CONTINUE for partial loads (achieved 83% coverage)

Neo4j 2026.01 Preview: Vector Search with Filters Three new approaches that combine vector search with Cypher filtering in a single query:

• Vector Search + Keyword Filters
• Cypher After Vector (post-filtering GraphRAG)
• Cypher Before Vector (pre-filtering on subgraphs)

No more two-step application logic for Graph RAG!

Context Graph demo app:

• Level of detail and perspectives you can view of the context graph and interactions with agents

Event

I will be at Jfokus in Stockholm next week!

Episode Transcript

Jennifer Reif (00:05):
You are listening to the Breaktime Tech Talks podcast, a bite-sized
tech podcast for busy developers wherewe'll briefly cover technical topics, new
snippets, and more in short time blocks.
I'm your host, Jennifer Reif, anavid developer and problem solver
with special interest in data,learning, and all things technology.
We have lots of snow and cold temperatureswhere I am right now, so I am staying

(00:29):
warm and enjoying the view from inside.
Data loading, cleaning andprepping always feels like the
longest pole in the tent for anyapplication or system that I build.
And this week was spent tryingto wrangle a larger data set
than usual into a Neo4j database.
I found some big walls thatI hacked my way through.

(00:50):
And this is a data podcast, after all.
So I'll share some tips and tricksto hopefully help you avoid the
same pitfalls that I ran into.
Then I'm very excited about a previewfeature in the upcoming version of Neo4j.
I will share some info aboutthat as well as some resources
to help you find out more.
I may have mentioned, or you may haveseen me present, with a Goodreads book

(01:15):
data set in the past that has books,reviews, authors, and user information.
However, I had only been working witha smaller version of that data set,
and I have been wanting to test myskills and some Cypher optimization
techniques by loading the whole thing.
I just hadn't reallygotten around to it yet.
So that was my challenge for this week.

(01:36):
I wanted to combine Ollama models,local models, and some new Cypher AI
procedures, which in the recent versionof Neo4j, it now supports Ollama
through configuring the base URL.
The new Cypher AI procedures will takeadvantage of that new vector data type
that I've mentioned in previous episodesin Neo4j, and allow you to generate as

(02:00):
well as store the embeddings with varioustypes of AI models in a Neo4j graph.
To set this up, I needed a few things.
I needed to add the AI procedures to thedbms.security procedures dot allow list
configuration property in the Neo4j.conf.
So I needed to go into theconfiguration file for Neo4j in a

(02:22):
local database instance and set itto allow list these AI procedures.
Then I needed to tweak the base URL forthe Gen AI procedures in order for it
to use the OpenAI like format and justcall the Ollama model under the hood.

(02:44):
So I needed to configurethat in the GenAI.conf.
Was a little bit frustrating actually.
I was kind of expecting and hopingthat it would be a configuration
property in the procedure itself.
But you actually have to setthis in the configuration file
in a separate configurationfile for the Gen AI procedures.
So I set that and then I also, whileI was in these configuration files, I

(03:05):
increased the memory and the page cachefor my local instance because I knew I
was dealing with a larger set of data.
I wanted to make sure I had plentyof processing power for that.
And I tossed in the APOC extendedlibrary and the configuration for APOC
to read local files, just in case.
But I didn't actually endup using this in the end.

(03:27):
After all this configuration, thefirst problem that I encountered was
a 4 0 4 error, and come to find outthe /v1 that is commonly used at the
end of the Ollama OpenAI compatible.
URL actually isn't necessary.
That gets tacked on within theprocedure itself, and so you can

(03:47):
just take off that /v1 at the endof the open AI compatible endpoint.
And so you just have basically, Ithink it's the local host and the
port number there for the base URL.
You don't need the /v1.
It adds it later.
Then after I got through that, I got aninput length error when I was sending

(04:07):
a batch of text properties to a Ollama.
So I checked the various links ofthe book descriptions, the text
that I was trying to send over withthe, mxbai-embed-large model, which
was the Ollama model I had chosen.
And I checked the context model, thecontext window for that model, which
is 512 tokens, not very large actually.

(04:29):
I tried a substring of thatand that still wasn't working.
So I ended up moving to a model witha larger context window, like the NOC
embed text, which has 8,192 tokens.
Much larger, much betterfor longer documents.
I was still getting the error, so I trieda single basic test call just to narrow
down my criteria a bit, and that worked.

(04:51):
So then I tried a tiny batch of threedescriptions and that one worked.
We're making progress.
So I went back to modify my originalquery into smaller batches, and
that failed even with batches ofonly five descriptions at a time.
I went down another path of maybeI need to chunk up this text into
much smaller pieces and try tosend these smaller amounts over.

(05:15):
So I took a slight detour here and neededto calculate the number of chunks that
I was going to need based on how longthe book description was and so on.
And the calculation of the simpledivision didn't work because
it divides into a whole number.
For instance, 1,364 divided by athousand leaves you a result of

(05:36):
one, which leaves off the remainder.
So I tried the round function with azero decimal precision and the ceiling
for the rounding, but that still wasn'tworking because the inner division was
still chopping off that decimal value.
I finally found an ugly hack thatwas then thankfully refined by some
colleagues to multiply one of thenumbers in the division by 1.0, which

(06:01):
basically turns it into a float value,keeping that precision of those floats.
So it retains the decimal.
Then use the seal function toround up and convert it all to a
whole number using two integer.
So yes, it was slightly hacky.
I wish there was a nicer, cleanersolution for that, but that was working.
I ended up going back and abandoningthe chunking though because of the

(06:25):
maintenance and the added overhead.
Every time you chunk up a book descriptioninto, let's say two chunks, five chunks,
30 chunks, whatever it is, you're havingto maintain that, add additional nodes
and relationships in your database,and that's just added overhead.
And this was already arelatively medium sized data set.
And the book descriptionsweren't pages long.

(06:47):
We're talking severalthousand characters max.
So this isn't pages and pagesof of documentation here.
I did query the distribution of thedescriptions that were longer than
10,000, 15,000 and 20,000 characters,and there really weren't that many
in each one of those distributions.
So coming back to the input lengtherror after messing around with the

(07:08):
various batching configurations.
I started to wonder if it wasthe overall context window on the
model, or if somehow the model waspreventing batches that were too large.
The batching had to be small.
After tweaking with that quite abit, it really didn't seem much
value to actually do any batching.
I created a query that didn'tdo the batching, and I got

(07:29):
an out of memory error.
Great new, new, error here.
So even after tweaking the instanceconfiguration to size up my page cache
and my memory and the transactionsize a bit, it was still struggling.
I went down the rabbit hole oflet's do some query optimization,
dig into this a little bit more.
I added on the explain keyword tothe front of my query, which means

(07:53):
that it dives in and pulls up thequery plan without actually running
the query in against the database.
It just gives you the query planof here's what the Cypher planner
will probably do for this query.
And it showed the query plan, and I foundan EAGER calculation in there, which
is not a good thing in a Cypher query.
It's there for a reason.

(08:14):
It's helpful in some cases, but generallyyou don't want eager when you're
dealing with large amounts of data.
I will link a blog postthat I wrote a while ago.
There's also some other documentation outthere about the Cypher eager, but I would
highly encourage you to check it out.
And if you're doing a bunch of Cypher andgraph database queries, I would definitely
recommend to try to narrow that downas much as you can to avoid the eager.

(08:37):
So I needed to try to removethe eager calculation there in
the middle of the query plan.
I worked with a colleague to helpme clean up a couple of other things
about the query, and I ended upbeing able to trim out the eager,
which was a super win for me.
Even after all that though, Iwas still getting the error.
So I took another slightdetour down the contents of the
book descriptions themselves.

(08:58):
They're in various languages, whichcan vary in token counts per character.
This is something I didn't even thinkabout when I was working with this,
is depending on the language thatyou're in, the token counts add up to a
different number of characters dependingon the base type of the language.
I got the distribution of allthe languages that were in the

(09:20):
book descriptions, went down therabbit hole of maybe refiltering
by the language code property, butthat was actually inconsistent.
Only about half of the entities in mydatabase had a language code property,
and it was really hard to filterthat out and, try to trim down and
find the non-Latin based charactersbecause of that code structure.

(09:43):
The language code was not somethinglike a, a list of languages You'd have
to manually go through and put togethera dictionary of those codes and so on.
That would've been really difficult.
So we started down the whole of, well,maybe we ought to try like a regex.
And that was met with limited successbecause of the variety of possible
characters that started the string.

(10:03):
So I went back to the drawing board anddecided to use some error logic that
one of my colleagues had suggested.
And you can use error logic inCypher to skip the ones that fail.
And just see where Iended up on that front.
It turns out it did about 83% ofthe embeddings, which is great.

(10:23):
That's sufficient for my first pass.
That'll get me most of the waythere, and then I can clean up
and hopefully go back and filterout and fix the ones that remain.
But what I ended up doing for this isusing at the end of my Cypher statement
with the call in transactions, you can adderror or retry logic there, and basically
what I added was on error continue.

(10:45):
It will try to do the transaction,but if something fails in that, it
will go ahead and just continue andmove on to the next transaction.
So skips and, and just goesto the next one basically.
With that pass, I was able toget most of the book descriptions
embedded, and that was a great thing.
And then I was able to move on tothe next piece of my data load.
And hopefully I will be able togo back and correct the ones that

(11:09):
didn't make it in the future.
But this does get me closer.
Also, as a heads up, I will beat Jfokus in Stockholm next week.
So if you're planning toattend, I hope to see you there.
The content piece I want to highlight thisweek is the vector search with filters
in Neo4j, version 20206.01 preview.
This is a blog post thatis out on medium now.

(11:32):
And it's all about a newfiltering in Neo4j vector search.
I am really excited about this featurebecause instead of doing a two step
GraphRAG process like I have been doing,basically where you have a query or
a method or something that will calla vector search, get the results back
in your application, and then go outand do a graph retrieval query and

(11:54):
pull the refined results from there.
This can do it all in one shot.
So you could write a single Cypherquery that will do the vector search
and do the graph traversal or othertype of filtering you might wanna do,
all in a single call to the database.
There are three approachesto this new Cypher filtering.
First is the vector search plus filters.

(12:15):
You could do like a keywordfilter and weed out results that
don't necessarily fit specificcriteria you might be looking for.
Then you have the Cypher aftervector, which is the post filtering.
This is the one that Itraditionally think of as GraphRAG.
You do the vector search plus aretrieval query on top of that, and
then you have a Cypher before vectorsearch or a pre-filtering approach

(12:40):
where you can do a Cypher statementand based on this sub graph, then do
the vector search within that space.
The blog post mentions and walksthrough each type of these scenarios,
and when you typically want to useone or another type of search shows
examples of each one of these as well.
This is really super helpful, and thenalso does a performance benchmark test

(13:01):
for all three and show how each oneholds up over the scale of number of
embeddings and, the filtering types.
The last thing I want to highlightis the context graph demo app.
I mentioned the blog post onthe context graphs last week,
and this is the demo app.

(13:22):
I finally had a chance to sit downand at least play a few minutes with
it just to see what it was all about.
What I really like here is the levelof detail when you ask questions
in the chat about the data set.
It comes back with a ton of detail aboutthings, and there's a variety of analyses
that the application will give you,and then of course, you could explore
deeper into certain aspects or others.

(13:44):
It also brings back a part of the graphfor some of the questions so that you
can actually see and poke around onthe actual entities and properties
themselves, and then provide someother information about the schema and
kinda the behind the scenes with somelinks to GitHub and other resources.
I hope to be able to explore thisa little bit more over the next few
weeks, but this at least got me started.

(14:05):
I find that nothing teaches you skillsfaster than exposure and experience.
So even though the data loading waschallenging this week, I emerged
with some wins and learnings thatI can use for the next time around.
Debugging is a really tough andyet satisfying skill to refine.
Neo4j also continues to improve itsfeatures, and I'm really looking
forward to trying out the new vectorsearch with Cypher filters that's

(14:28):
coming up in the next release.
As always, thanks forlistening and happy coding.