EP. 251 Discovering the Power of Atlas Search: Insights from a MongoDB Community Champion

Join us in this engaging conversation with a MongoDB Community Champion as we explore the transformative capabilities of Atlas Search. With nearly a decade of experience using MongoDB at scale, our guest shares insights on managing billions of documents and achieving impressive performance benchmarks. Learn how Atlas Search simplifies data management by eliminating complex pipelines and enhancing efficiency. Whether you're a developer or a data enthusiast, this episode highlights the benefits of leveraging MongoDB's latest features for your projects. Don't miss out on these valuable insights!

Episode Transcript

(00:07):
Hey everyone. So welcome back to the live
stream here from the Expo Hall at dot Local London.
I'm Luz Carter, Developer Advocate here at Mongo DB and on
and off being one of your hosts today.
So I am very, very happy to say that I am joined by one of our
amazing community champions, andyou may have heard from some of
our community champions already today.
We love everything that they do for us, so it's always great to

(00:28):
have them on. So I am joined by Nilesh, one of
our champions, who's travelled quite a long way to be here
today. So, Nilesh, would you like to
introduce yourself? Sure, thanks.
Thanks for having me here myself, Nilesh.
I've worked with more engaged asa technical architect and
recently been former to become really champions.
It's really great to be here in London, like enjoying the trip

(00:49):
as well. Is it your first time in London?
Yes. How have you found it?
It's really nice. The weather is really nice, like
I I like to wear a lot close so.Yeah, but have you?
When did you arrive? Sunday.
Sunday. Nice.
Have you been able to do any tourist things or have you just?
Yeah, yeah, I'm going to Scotland.
Oh wow, I love Scotland. Yeah.

(01:11):
So how, how do you use Mongo DB at the moment?
Obviously normally our communitychampions, you tend to discover
us, you're working with it. So.
Yeah, so mongo DB. I've been using since 2014, but
in the recent organization it's like we just like in terms of
the DB, we have Mongo DB left and right.
So most of the users are on the Mongo DB as like one of the

(01:34):
highest scale in India. Oh wow.
Yeah. So it's like we have a lot of
clusters cross data centers, probably 36C plus S And though
we are using community addition but we get a lot of help from
the Atlas and the Mongo DB and during the Android support as
well. Oh, wow, that's, that's pretty

(01:55):
impressive. Yeah, it's 50 or 60 clusters.
I'm sure we have customers with more than that, but I think it's
probably the largest I've ever spoken to directly, So that's
awesome, yeah. So yeah, I mean like being
experienced with almost 10 yearswith the Mongo DB and like this,
this company gave me a chance touse that at very high scale,

(02:15):
like you know, billions of documents scale at millions RPM,
etcetera. Oh wow, that's amazing.
So like how, what kind of what would you say at the moment is
your like favorite feature of Mongo DB or something that
you've used recently at work that's really helped for Mongo?
DB, yeah, yeah, sure. So I think one of the recent
POCI did, I did it personally and it was on the Atlas search.

(02:39):
So that was really amazing because till now we're using
elastic search and as I said ourmost of the data is in Mongo DB
itself. So the kind of, you know,
pipeline which connects from Mongo DB to syncing the data to
the last six hours, that pipeline is a huge, big and
complex. It is.
It involves a lot of text, text which probably I'll explain in

(03:00):
detail. But that's like the complexity
goes away when I use a glass like OK, just configure the
index and it's done. Yeah, I love that as well
because like I think we've seen that a lot and that's why I get
so excited about these new products we bring out like
search and vector search, because people have been using
another tool to achieve it with Mongo DB and now we're bringing

(03:22):
it out and it's optimised for Mongo DB.
Like you say, just create an index and suddenly your life is
made so much easier. Yeah.
So that's what like I think likeI'll just go through the text
tag itself involved in the current pipeline, right.
That's it's not just Mong. So like after the Mongo, we have
a division connector then like Kafka pipeline and Kafka
consumer groups writing to elastic search, then now if

(03:45):
those whole pipeline has some issues, right, we have Spark
jobs to remediate that issues. So like lot of text tag involved
which goes away with just a conflict.
So that's, that's amazing. Yeah, so, and as you mentioned,
Kafka, so did you do anything like stream processing or the
Kafka Connect or anything like that as well?
Because we have so many, you know, we call it a developer

(04:05):
data suite. And it's so true because we have
so many products that help with so many different problems.
Yes, yes. So like as I said, right, even
the from OP log stream writing to Lassie, that's a part of the
stream itself, right. So likewise, we have a lot of
streams on OP logs based and I think we are also looking to,
you know, go with chain Steam, but I mean that's in the plan,

(04:29):
not that I suppose, but we have our own all the custom
pipelines, which is similar to the Steam use case.
Wow. And so I think you've touched on
it a little bit before about some of the benchmarking you did
see something. You've.
Got from using Mongo DB yes that's right.
I love hearing about bench like our beautiful.
Then if you saw in the keynote the beautiful graph about the
performance of previous 8.0. So I love benchmarks and

(04:51):
performance. So tell me really about it.
Sure. So as I did the POC myself, so I
mean the POC was again scale because at our company most of
the things are around the scale as we talk, right.
The the POC was simply to run 1,000,000 RPM search.
So like running 1,000,000 RPM search over 3 to 400 million

(05:14):
documents. So for those that don't know
what's RPM search like what, what do you mean by RPM per
minute? OK, yeah, just we always people
just make, OK. Yeah, yeah.
So it's like requests per minute.
So if you convert to the seconds, like 16,000 requests
per, Yeah, some of that, yeah. Wow.
Yeah. So and that's too for the
starts, right? So like our earlier pipeline.

(05:37):
So I think I think that was the amazing part like Bing code
developer, I don't trust easily,you know, the tech when they say
that something's really, really good, have to test it.
Yes, very tested. Like as I said, those that
scale, I think what P99 latency,P99 percentile of the latency,

(05:59):
it was around 80 to 90 Ms. Our earlier pipeline had around 120
Ms. milliseconds, right? And the delay between the sync,
right? Because now Atlas search is a
packaged product with the Mongo DB itself, right?
So there's no pipeline. The sync is managed by the Atlas

(06:19):
itself and the sync delay was less than a second.
Why? We had a sync delay of around a
minute. Wow, that must have.
Yeah, at the moment you first saw that, you must have been
like, oh, wow, Yeah, it's been all my life.
Yes. So it removes the pain of like
maintaining all of those tech stack as well.
Your SLA everything is managed by that class.

(06:40):
Not the good part is the cost. So the cost of my current system
in this system was same. And that is like really amazing
because you're getting a managedservice or the same cost, which
you're, I mean, when I say cost like my infra cost equals my
manual service cost. That's that's really.
Oh wow, that's that's incredible.
See, I love here. I have loved hearing stories

(07:01):
like that and I think in in my role, I create a lot of content
and I learn like new products and I'm very lucky about
learning these new products, Butit's I don't necessarily see it
in like enterprise, like I'm notdoing it directly.
So I always love hearing storiesabout these amazing things that
people discover. Like, oh wow, I switched to it
and it like went from all this long delays under a minute like
that's, that's incredible. Have you ever thought about

(07:23):
blogging about your findings? Like actually writing an article
about your like? I will be writing as I said, I
mean the POC was done almost twomonths back.
We are still into the product like production finalization.
Yeah, but it sounds amazing. Yeah.
If you're ever interested in writing, obviously we have a
guest daughter program and we love our champion.
So if you want to write about your experiences, you know, I'd

(07:46):
love to work with you and write that and help you write it
because like I start getting stories out there, like I mean.
I have like one more use case which I did recently like recent
recent two weeks back, it was a coupon distribution and that was
also amazing. Like I think one of the things
that I don't know people would be knowing about the operator
find on an update. Yeah, right.
The so that operator works so well like your your use case of

(08:10):
select and UPDATE and your SQL kind of thing, right.
So coupon distribution was like you have all the coupons data,
you just want to distribute it uniquely, atomically use that
operator done. Yeah, there's the stories of
that, isn't there? You try something, it's like,
oh, well, that was easy. Like I've, I've done some demos

(08:30):
on some things in the Microsoft space around like with, with C#
code. And I'm like, well, how am I
supposed to demo this for half an hour?
Because it's one line of code and then it's all done for you.
And it's like, oh, I love this so.
As I said, right, I mean, I think I'll come back to the
point that pipeline and all the text, right?
I think I mentioned the text this like I mean Taft consumers

(08:54):
elastic search spark job air float to run the spark jobs.
OK, so now all of this gone away.
Now the the other part which helps the company is you don't
need to train your engineer or you don't need to hire the
engineer who knows a lot of things.
Yeah, that. Yeah, that's like one of the

(09:15):
most amazing part of it, right? That you don't really need so
much of technology because tech and everything is.
Now if you talk about even the index right, that's a conflict.
So now Mongo DB itself has become a single I would say
technology where you can manage your content also in the Mongo
DB. Itself, Yeah, it's, it's great.

(09:36):
Yeah. And so obviously we've only got
a few more seconds, but is thereanything as, is there one sort
of learning byte or bit of information that you've either
learned today or you learned from that POC that you wish more
people knew about? I think like as I said from the
POC is very important part as when we talk about the search,

(09:58):
we know open search, elastic search, all those things, but I
think at last search is not thatfamous probably right.
So probably people should explore less search as it's like
very easy, scalable and cost effective.
So that's really amazing part ofit.

(10:18):
And of course, like you get it like completely managed.
So that's the one of the things I would say people should.
Not amazing. Well, thank you so much for
coming to speak to us and hopefully we'll be able to hear
more about your POC in the future and even more amazing
things that you've discovered. Amazing, right?
Thank you. Thank you so much.