September 29, 2025 • 43 mins
Longtime friend of the show Rajiv Shah returns to unpack lessons from a year of building retrieval-augmented generation (RAG) pipelines and reasoning models integrations. We dive into why so many AI pilots stumble, why evaluation and error analysis remain essential data science skills, and why not every enterprise challenge calls for a large language model.
Featuring:
Upcoming Events:
- Join us at the Midwest AI Summit on November 13 in Indianapolis to hear world-class speakers share how they’ve scaled AI solutions. Don’t miss the AI Engineering Lounge, where you can sit down with experts for hands-on guidance. Reserve your spot today!
- Register for upcoming webinars here!
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Jerod (00:04):
Welcome to the Practical AI podcast, where we break down
the real world applications ofartificial intelligence and how
it's shaping the way we live,work, and create. Our goal is to
help make AI technologypractical, productive, and
accessible to everyone. Whetheryou're a developer, business
leader, or just curious aboutthe tech behind the buzz, you're
(00:24):
in the right place. Be sure toconnect with us on LinkedIn, X,
or Blue Sky to stay up to datewith episode drops, behind the
scenes content, and AI insights.You can learn more at
practicalai.fm.
Now, onto the show.
Daniel (00:49):
Welcome to another episode of the Practical AI
Podcast. I'm Daniel Wightnack. Iam CEO at Prediction Guard and
not joined by Chris today, butvery happy to be joined by a
longtime friend of of thepodcast and and friend of all
things AI and data scienceonline, Rajeev Shah from
(01:11):
contextual AI, where he's thechief evangelist, and also, of
course, TikToker extraordinaireall things. So welcome back to
the show, Rajeev. It's great tohave you.
Rajiv (01:23):
It's great to be back. Excited to talk.
Daniel (01:25):
Yeah. And we're we're close neighbors around the the
Midwest as well. I'm lookingforward to seeing you at the the
Midwest AI Summit, which if iflisteners don't know, both Chris
and I will be at, the Midwest AISummit, which is happening
November 13 in Indianapolis. SoI think as you put it on
LinkedIn, Rajeev, if you liveanywhere near Corn and you like
(01:49):
AI, then this is the place tobe. And actually Rajeev will be
there giving a great talk aboutkind of this idea of, I think
you're going to be talking aboutthis stat, the ninety five
percent of pilots that fail, whythat might be.
Yeah, so great to have a fellowMidwest AI Silicon Prairie
(02:11):
friend on the show as well.
Rajiv (02:12):
No, it's great to have these in person events here in
the Midwest. Usually you have tokind of fly out to some large
city to get that. It's nice tobe able to kind of get that
right in Indy.
Daniel (02:21):
Yeah, yeah. So if you're interested in that, go to
midwestaisummit.com and makesure and register for that.
Hopefully we'll see you there inperson. It's going to be a fun
time. But it has been a whilesince we've had you on the show,
Rajeev, and I'm sure there's alot to catch up on.
It almost seems like last timewe talked, there was kind of we
(02:45):
were all getting into Rag andthinking about agents and
reasoning models. We're tryingthings. People were gradually
developing maybe what theythought would be a direction for
best practices or ways toapproach problems. It would be
interesting to hear from then tonow how you've seen that world
(03:07):
advance in these archetypal usecases that people are addressing
with retrieval relatedframeworks?
Rajiv (03:18):
Yeah, no, it's been amazing. About two and a half
years ago, we were really giddyabout the possibility of AI, but
there was a lot of differentthings that we can do. And kind
of since then, we've seen AIdevelop in certain ways where,
for example, in code completion,code development tools, it's
accelerated dramatically, evenover the last year like that.
And we're all kind of usingthose tools on a regular basis.
(03:41):
And then in other areas, right,like chatbots, for example,
we've seen some progress, butwe've also seen that there's
some things that are difficultto do with chatbots as well.
But I think, you know, it's beenincredible to see just the
continued development of AI interms of the capabilities, the
smartness, the ability to kindof use tools to find search
information that there's stillso much for all of us to do in
(04:05):
our lives, kind of for those ofus that work with AI.
Daniel (04:07):
Yeah, yeah. And I guess, certainly there's a lot of
people still I guess when Iinteract with customers or hear
from other people in theindustry, still a lot of people
are wanting, as their firstthings, to kind of whatever is
build a rag chatbot, which maybefor those listeners out there
(04:28):
that are maybe not as familiarwith this, they've maybe heard
rag this and raggedy rag thatand all of that. Maybe just
remind listeners what rag is andhow retrieval may fit into some
of this AI stuff.
Rajiv (04:46):
Yeah. And I'll start simply. Even a couple of years
ago, I was kind of working withthe earliest large language
models and people would be like,I like ChatGPT, but it doesn't
know anything about me or mycompany. Like, how can I train
it to know all that knowledge?And what we figured out is that
it's not about really trainingthat model about the knowledge.
It's instead finding thatknowledge, searching that
(05:08):
knowledge, passing that on tothe model, and then be able to
use that. And that's where kindof this idea of retrieval,
finding stuff, and thenaugmenting the generation,
augmenting the written responsefor that came about. And I think
RAG today is kind of one of themost important or it's one of
the most widely used use cases,I'd say,
Daniel (05:29):
the general AI space. Yeah.
Rajiv (05:31):
Every company is probably running some type of rag at this
point for searching its internalknowledge, helping folks figure
out through HR documentation,using it for customer support.
There's a lot of use cases wherewe have huge amounts of
information. We want to be ableto find things from that. But we
also want to use the smarts ofAI for that. We don't want a
(05:54):
Google search results of 20things.
We want a nice AI summary of itor we want to extract out all
the information. I just want thenames and dates out of all this.
I don't need to read every one.And so that's where the AI
really comes into it and why Ragis so powerful and we see it so
widely used.
Daniel (06:11):
Yeah, and I love what you were saying, just sort of
contrasting this with trainingor even fine tuning. I find
this, of course, to be a verywidespread misconception about
how these tools work. EvenChatGPT, the application, it
does seem like it kind of quotetrains on your information that
(06:33):
you put in, your previous chathistory, etcetera. And then it's
almost like you have a model ofyour own, right? It almost seems
like OpenAI has a separate modelfor every person on the planet,
which is not feasible from thedata science training
perspective.
So could you highlight some ofthose things and highlight, I
(06:56):
know you educate a lot as well,so I'm sure you've seen this as
well. There's kind of thisjargon of training thrown around
a lot, which is confusing maybein how it's used. Yeah.
Rajiv (07:08):
Yeah. And I think partly is a lot of these companies want
to make you think that they haveone thing that does everything
in the world. But if youactually take a look at
something like ChatGPT, it's asystem. It actually composes of
multiple parts. So if you weretrying to build your own
ChatGPT, these are some of thethings that you would have to
think about.
(07:28):
So one is I need to be able toretrieve from lots of different
sources. I have all my knowledgeinside my companies, inside of
Confluence. I need to be able toaccess and retrieve that. That
could be one thing. So one thingis using Rag, being able to
access tools.
Another thing is memory. Like Iwanted to remember that last
conversation I had, whether itwas two things I said ago or
(07:52):
like one of the nice thingsabout ChatGPT is you can come
back a conversation or a daylater and it remembers a lot
about you. It could maybe alittle bit scary sometimes how
much it remembers about you andhow well it can profile you. And
so this is kind of what we seeinside of AI engineering as
context engineering, wherethere's a number of different
(08:12):
parts of managing interactionswith these models, whether it's
rag, whether it's rememberingthe memory, when it's knowing
how to summarize conversationsto do things like multi turn so
we can have those repeatedconversations, but keep track of
what was said earlier in ourconversations as well.
Daniel (08:31):
Yeah, so I love that idea of context engineering. How
is that idea of contextengineering kind of different
from what we would maybe fromthe traditional data science
side consider model training, Iguess? So there's almost like a
pipelining mindset versus anactual GPU model training piece.
Rajiv (08:56):
Yeah. I think what's remarkable nowadays is the
amount of information that'sstored inside of these large
language models. I do a thingwhen I kind of talk about large
language models where imagine ifyou were sitting outside and
kind of reading a book and youcould read all day, all night.
If you read for ten years, Ithink that's on the order of
something like about a billiontokens. And these models are
(09:20):
trained on the order of like15,000,000,000,000 tokens.
So it's just an inconceivableamount of information that these
models hold inside there. And Ithink one of the biggest
improvements we've seen in thesemodels is not only kind of
stuffing all that information inthere, it's effectively using
all of that information inthere, that these models really
(09:40):
today are untapped in terms ofeverything they can do. And it's
one of these things on thetechnical side is why you see
that the capabilities of themodels have continued to grow
without them necessarily havingto grow immensely in size simply
because we're better tappinginto all the abilities that they
have inside them.
Daniel (09:59):
Yeah. And I know you're up for this because we know each
other, but I'll present ascenario to you that sometimes
comes up for us, whether I'mteaching a workshop or
something, and you can help usunderstand how this retrieval
type of thing fits in. So Ioften Maybe I'm talking to folks
(10:21):
in healthcare, and what'sinteresting is there's all of
this huge amount of data thatthese models have been trained
on. And there is good, let'ssay, medical information in that
data, right? Or there's careguidelines for nurses or medics
or whatever it is.
Let's take that as an example.The interesting thing is there's
(10:43):
even if you think aboutfactuality in a situation, it is
kind of relative in the sensethat if I'm a medic that works
for this provider, let's saytheir care guidelines are
different from another provider.Or if I'm treating maybe
children, those guidelines areprobably different than if I'm
(11:05):
treating or senior adults orsomething like that. So there
can be these facts out therethat actually conflict with one
another. It's really You talkedabout the context.
It's really about the contextthat you're working in, what it
should pull. So these models,assuming that all that
information is public, right,like you said, and it's been
scraped off the internet, all ofthat's kind of You could imagine
(11:29):
that maybe somehow that isembedded somewhere in the model,
the base model, so whether we'retalking about LAMA or GPT or
whatever. Let's say I'm ahealthcare company, but I'm
interested context. Now, how doI make that connection? Still
use maybe, I'm not going toretrain that model, but how do I
(11:52):
use the model then in thiscontext of kind of conflicting
facts?
Rajiv (11:56):
Yeah, absolutely. As much as the models know and been
trained so widely, they don'tknow everything. And so
sometimes that's you have yourown information. You want to
pass that into the model. And sothat's where that retrieval
augmented generation comes in.
We want to grab that informationand and then we want to pass it
into the model. And this iswhere for folks who are in the
space for a while promptengineering comes into play,
(12:20):
where we think about the inputsthat go into the model, where we
take the facts that we knowabout our own given situation,
along with maybe someinstructions for what we want
along with it. And that all ofthat is manipulating the
context, right? The larger worldthat this model is sitting
within there. And bymanipulating and giving it
information, instructions andother facts, we can get outputs
(12:42):
that better match kind of whatwe're looking for like that.
Daniel (12:46):
Yeah, yeah, that's great. And I guess there's
another term that's been thrownaround the last couple of years.
One would be this kind ofretrieval piece, which you've
talked about, and this idea thatin the context of a certain
question or something, I'm ableto connect to one of my internal
data sources, get that rightThere's this other thing that
(13:08):
would be related to reasoning.Now, some people might just
consider that all generative AImodels reason in one way or
another, but it is now almostlike a term of art. It means a
certain thing talking aboutthese models.
Could you help highlight thatalso to put that in context? And
(13:29):
then maybe we'll circle aroundand combine some of these
things.
Rajiv (13:32):
Yeah. A lot of times when we talk about models, we want to
make it convenient to helppeople explain things until we
kind of anthropomorphize them,right? We give them human like
qualities when they're notactually human like qualities.
And so, for example, reasoningis something we say for these
models. Now they're notreasoning like a human would
through a problem.
But what we typically mean kindof when we think about reasoning
(13:54):
with these models is they'redoing lots of extra steps and
they're doing these steps in alogical way to better solve a
problem. So if you take kind ofa math word problem, a great way
to see this is if you take aword problem like the train is
moving east, you know, at 50miles an hour and another train
is moving west another 40 milesan hour and you have to figure
(14:17):
out like what point they'regoing to cross, right? There's
no kind of quick answer to that.You need to kind of calculate
the first train, calculate thesecond train, and then you can
figure out the solution. Andwhat we've done, and that's what
these reasoning models havedone, is we've trained them that
don't come up with the answerright away.
Think through it. Think aboutthe first thing. Think about the
second thing. Connect all thedots and then put an answer in.
(14:41):
And the way we've done this iswe've literally given the model.
Well, we've trained it inmultiple ways, but in some ways
we've literally give the modelan example of, hey, this is how
I solve this word problem. I didthis all these steps here. I
want you to learn how to gothrough these problems step by
step. And this is where wediffer from two and a half years
ago when the first models didn'tknow how to do that. Those first
(15:02):
models, all we had trained it upwas like, hey, tell me if a
movie is a good movie or a badmovie or tell me if this
sentiment is happy or sad.
But since then, we've had timeto develop more training data.
We've given them more complextraining. And the cool thing is
the models have picked up.They've been able to learn this
capability for doing this. Andso now we're able to do this
much more complex kind of I'mdoing the air quotes of
(15:25):
reasoning to solve theseproblems.
Daniel (15:28):
Yeah, I guess one thing that people, and this is part of
why I really enjoy watching yourvideos online, is you often
break down a lot of this jargonand kind of help. It's almost
like people feel the shock ofall of this terminology and a
new model coming out every 30minutes or whatever it is, and
(15:51):
just not really knowing how todeal with that. And there's kind
of all of these things that havehappened. So there's reasoning
models, there's small languagemodels, there's tool calling,
there's retrieval, there's allof these different mix of
things. Based on yourexperience, both what you've
implemented personally, alsowhat you've seen and interacted
(16:13):
with folks on, if people couldbring in some focus and maybe
imagine a company's getting intoAI, however they view AI
transformation within theirorganization, and they're
thinking about some of the usecases that are the initial ones
(16:35):
on their roadmap at the timebeing, what would you encourage
them to pick the signal out ofthe noise?
So what would you encourage themto maybe focus on to see some of
that time to value upfront? Notthat they don't want to explore
some of the other topics thatare happening or read about them
(16:55):
or whatever that is, but howwould you at least recommend to
get that best time to value, ormaybe just the things that are
producing the most value? Andyou could maybe flip that as
well and say, what are some ofthose things that are cool
things, but maybe let's wait andsee what happens and maybe just
(17:16):
sort of don't get distracted atthe moment.
Rajiv (17:18):
Yeah. Mean, you're taking me away from all the fun, cool
technologies that are thelatest, like automation, things
that I can kind of fire up likethat. I think when you're
thinking about this from theperspective of kind of a
company, if you're kind of amanager in these situations, you
have to figure out what usecases you want to put on a very
different hat than just thinkingabout the technology itself. And
(17:42):
I can tell you this, like I wasburned from personal experience
when I was just starting out indata science. I was entranced by
the latest technology.
So I remember, right, we weretalking about code development.
Like ten years ago, I wasworking at State Farm and I
think Andrij Karpathy haswritten his paper about the
unreasonable effectiveness ofLSTMs. It was something like
(18:06):
that. But part of that paper hadthe idea of you could translate
code, you could complete codefrom one language to another.
And I was like, hey, come on,guys.
Like, we've got a lot of thiscobalt code sitting around.
Like, give me some give me a fewGPUs and some data and I can
Daniel (18:22):
do it.
Rajiv (18:22):
I'll solve this. Naive. They didn't fund that project or
anything, but I think, yes, itcan be very easy to be kind of
seduced by the technologies,kind of what a shiny demo is
versus when you're in anorganization, you really have to
think about kind of the problemsthat you have. Part of it will
be, you know, how complicatedthis is from a technical point
(18:44):
of view to get it up andrunning. That's one factor, But
that's not the only factor.
We also need to consider what'sthe value to this organization.
I talk to lots of enterprises ona regular basis and I see often
what I call science experimentswhere teams like the latest
technology, they go out and kindof run this stuff, but there's
(19:05):
no way for them to actually getthat implemented inside the
company in a useful way. Andthey're literally just kind of
interesting experiments thatpeople are running like that.
Daniel (19:14):
Does that get partially to like the ninety five percent
of AI pilot failing type ofreport from MIT?
Rajiv (19:23):
Absolutely. Now, the 95%, of course, is like a little bit
of a hype number that they liketo put out in this. And for
those of us who've been in thespace for a while, we remember
the fact of 80% of data scienceprojects fail, I think was
something that we had. And tosome extent, that's okay. You
can't expect every initiative,every experiment, everything
(19:44):
that you start to succeed.
You want things to fail becausepartly is if something works,
that means you have to maintainit. You have to monitor it. You
have to put up a lot of guardsaround it. There's a cost for
something that actuallysucceeds. But when we talk about
AI, it's very easy to build acool demo, but it's not only the
value to the company.
(20:05):
You have to figure out how tointegrate this into people's
everyday work life. And so youcan build a very shiny widget
that sits and can do somethingawesome. But if that's not
inside somebody's regularworkflow of how they work, the
tools that they work in, ifthey're not properly trained on
how to use it, if theirleadership isn't supporting you
(20:26):
to use it, there's lots offactors like that that go into
why people might not actuallyadopt and use a technology. And
it's really nothing about AI.It's really about organizational
change and introducingtechnology into companies.
Daniel (20:40):
Yeah, I know from just a founder perspective, can be,
know, just from a different sideof this, it can be frustrating
when you see like, Oh, there'sthis company over here that the
technology side is fairlysimple. Like, oh, it's just a
simple model that does a simplething, or a browser extension
(21:02):
that does this. And you're like,wow, I could have vibe coded
that in a weekend. How are theyscaling to the moon? We have all
this cool technology.
And I think part of it is thatside of it that you talked
about, part of the hard problemis cracking what actually does
(21:24):
provide value to yourorganization, what can be
adopted, how you communicatethat, how you tell that story,
and how you deliver on yourpromises, how you provide
customer support. A lot of thatis not really related to the
technology and that component.So maybe, I don't know if this
(21:45):
would be accurate to say, butmaybe the first step people
could take is just gettingsomething off the ground that's
fairly simple and interacts withan AI model. Maybe it's just to
do a very simple task, butreally pushing that through,
like you say, to be embedded ina product or be embedded in a
process. That may be the bestway for people to kind of start
(22:09):
that journey is to really startfrom that simple side and deal
with some of, in all honesty,the harder problems around the
periphery of the technology.
Rajiv (22:19):
And I think a big part of that, like what you're saying,
is just to get closer to thoseend users, the stakeholders,
because I think once you oftencut through that, sometimes you
figure out that really theydon't necessarily need a fancy
kind of GPT-five model to solvetheir problems. Maybe you can
solve it with almost a simple ifthen rule that you can just
(22:41):
implement in And some so this iswhere kind of looking at the
data, spending time talking tothose end users that often gives
you a much better result. It'sgoing to give you the biggest
bang for your buck than goingout and reading some archive
paper.
Daniel (22:57):
Yeah, yeah, that's true. And I guess now you should just
have the AI model read the paperfor you and give you some
summary points. A big fan
Rajiv (23:08):
of that. On my walks, often just sit and I'll talk to
Chad GPT and we'll talk throughpapers and what are the main
technical points and stuff.
Daniel (23:14):
Yeah, I'm glad I'm not the only one. Thanks for
validating that. I guess gettingback more into the development,
retrieval kind of reasoningstuff, what are some of the Now
that we've been working withthis technology for a while,
we've got more cycles. Someonecan spin up a rag pipeline in
(23:35):
whatever, ten minutes I can spinit up and I have something
going. But it's another thing tokind of, of course, scale that,
maintain it over time, deal withsome of the issues.
I guess my question is whatpitfalls are people falling into
that maybe we didn't know aboutwhatever it was a year ago when
we were kind of just gettinginto these initial kind of naive
(23:57):
rag sorts of things? Whatchallenges or consistent
challenges and pitfalls do yousee people kind of falling into?
Rajiv (24:05):
Yeah, think the consistent thing I see with
something like RAG is it'sfairly easy to build a quick
demo. You can grab an off theshelf embedding model to do
that. You can combine that witha generation model like an
OpenAI model and you can buildyourself a quick kind of proof
of concepts. I think the troublethat people get into with it is
(24:25):
scaling it up. It's great on 100documents, but now all of a
sudden I have to go to 100,000or 1,000,000 documents.
How am I going to do that? Orwhen I first did my demo, I did
a couple of very simple queries,text extraction queries. But now
when I put it out in front of myusers, I find out all of a
(24:45):
sudden they're not giving niceone sentence queries. They're
just asking two words and then Ineed to add a query
reformulation step or somethingto do that. Or the accuracy is
not kind of what I was lookingfor.
And so I've added a bunch ofpieces in there. I've added a re
ranker. I've added other steps.But now my latency is suffering,
(25:08):
right? There's all these kind oftrade offs as you kind of get to
production.
And then you're like, oh, youknow, do I go back and you go
and you look online and you seethat, oh, wait a minute, there's
like 15 I think there's like 25,30 flavors of rag. You're like,
oh, did I set up myinfrastructure wrong? Oh, do I
go back and I change my chunkingstrategies? I can see there's 10
(25:29):
different chunking strategiespeople are doing that. And so I
think this is the cycle of whereit's very easy to get started,
but getting to that final kindof production quality rag can
kind of be a little worrisome.
Daniel (25:40):
Yeah. And do you think that that's where the real human
engineering piece of thisdevelopment still will be with
us for some time? Because tosome degree, you can describe
let's say describe that sort ofproblem to my AI coding
assistant. Is it reasonable forme to think that that kind of
(26:02):
debugging and process could helpme, or that kind of assistant
type of thing could help me getto the bottom of this or update
my retrieval pipeline and thatsort of thing?
Rajiv (26:13):
So I'm kind of optimistic that the reasoning models that
we have now are going to get usmuch farther towards helping you
solve that problem. Now, ofcourse, that reasoning model's
got to understand how you'rethinking about how to solve that
problem. But already today, ifyou take the traditional Rag
approach, for example, but youpair it with one of the
reasoning models that can maketool calls, can kind of look at
(26:37):
the results that come back,think about it, decide, hey, I
want to re query it in adifferent way. You can improve
the quality of your results inthat way by using that reasoning
to do that. So I'm prettyoptimistic that we're going to
keep finding new ways as long asthere are workflows that we can
train these models on that arefairly, let's say, logical.
Daniel (26:57):
Or typo. Yeah, exactly. Gotcha.
Rajiv (27:00):
A way we can connect the dots and teach the models to do
that, that I think a lot ofthese things that if we give it
the budget for spending timethinking, doing those tokens
there, it's going to cost usmore latency, but we're going to
see better results. And I thinksome of us, we already see that
in using some of these toolslike deep research, where we can
(27:20):
see that by spending more timeon the task, it's able to give
us a better result.
Daniel (27:25):
Yeah. Yeah. And one area that we haven't talked about yet
is this sort of world of agents,which I know is loaded term,
it's probably related to whatyou were just talking about in
terms of time of compute andsteps in the process and the
reasoning models and all ofthose sorts of things. Could you
(27:47):
help us parse apart from yourperspective? It almost seems to
me like it's one of those sureyou remember when it seemed like
every conversation I got on, thefirst part of the presentation
was, what is the differencebetween data science and AI and
machine learning, or thedifference between machine
(28:09):
learning and AI or whatever.
And at a certain point, I waslike, well, these terms all mean
nothing essentially becausepeople use them so
interchangeably. I feel likethat's sort of where we're
getting with agents andassistants and all of these
sorts of terms. But from yourperspective, I guess if that
word agent has a meaningfuldifference to you, what stands
(28:32):
out in your mind there?
Rajiv (28:33):
Yeah, and I think we all like the idea of this agent,
right? Like something I can givea problem to and it solves the
problem. And now I think wherethe definitions break down to is
how much autonomy is this agent,how structured is what we do
like that. But if we just thinkback about the bigger picture of
like, I have a problem, it's nota straightforward problem that I
(28:56):
want to give to an agent. Now,there's at least two different
ways that we can kind of tacklethis.
One is I can give them a step bystep list of instructions. And
this is what we call a workflowoften, do this, do this, do
this. And then I can check theirwork at every step and make sure
that they're on the track tosolve it. Or I can just be like,
this is the difference maybebetween my kind of five year old
(29:17):
and a 13 year old, my 13 yearold, like I'll give them the
list and I'll cross my fingersand hope that they'll finish it.
And, you know, usually they do,but not always.
But I'm not involved in everystep of the way. And I think one
of the things is we're watchingthe agents evolve and this is
one of the big trade offs thatdevelopers have today is how
much structure, how muchbabysitting am I doing for this
(29:39):
agent? How do I do versus kindof the hands off? Now, I think
the trend is we're going to beable to do much more hands off.
Just like we've seen thesemodels be able to gain the
reasoning ability over the lasttwo years or so, I have no doubt
that we're going to be able totrain them to do more complex
tasks, to be able to followthose steps.
(30:01):
It's just a matter of kind ofgiving them the training data,
having the experience to dothat. So my bet is in the long
term for many of these tasks,we'll be able to be much more
hands off and the modelsthemselves will strive to be
able to solve them themselves.
Daniel (30:17):
I just thought it would be good to get your input on a
theory I've been having, whichis maybe related. I mean, maybe
it's an offshoot from what we'retalking about, which is really
maybe the ability to use theseVibe coding tools or others to
or assistance or agents toupdate retrieval processes or
(30:39):
kind of architect our AIpipelines, if you will. I've had
this sort of thought, and I'mcurious about your opinion
because you also have abackground kind of pre
generative AI in the datascience world that previously I
had in my mind this mental modelof, on the one side, you have
traditional softwareengineering, DevOps,
(31:03):
infrastructure. On the otherside, you have business and the
product and marketing, all thosethings. And in the middle is
data scientists because youtranslate the business problems
and understand how to connect itto the data and the tech and
produce your predictive model.
You're living between those twoworlds. It's almost like I see
(31:26):
that middle zone shrinking andshrinking and shrinking because
those domain experts on thebusiness side are actually able
to use very sophisticated toolsnow to kind of self serve
themselves a lot of the kind ofmaybe stuff that would normally
fit on the plate of a datascientist. So part of me is
(31:48):
wondering, me personally, likeDaniel Witenack as a data
scientist, what is the future ofthat data science world when
this middle is shrinking? I'mcurious if you also see it that
way or see it slightlydifferently, and what your
thoughts are in terms of thatview. Yeah.
Rajiv (32:07):
So I don't think the data science world is changing at
all. First of all, I'm excitedthat the bar is kind of dropping
in terms of people being able touse code to build solutions.
Like my nine year old canliterally like vibe code of a
game that he can play as well asmy 23 year old who has a who has
a degree in computer science.And you couldn't tell the
(32:28):
difference between like thegames that they built like that.
So there's a great ability injust allowing everybody to be
able to kind of more participateand work kind of with code that
we're being able to see now.
Now, how does that changesomething like data science?
Like data science, the originaltriangle was, part of it was
coding. I think data scientistswere never thought of as really
(32:51):
great coders, which is why were80 trying to put in
Daniel (32:56):
percent of those projects didn't make it past
pilot too.
Rajiv (32:59):
Exactly. Right. Like they would not write production code
and right. There was MLEengineers kind of became the
offshoot to kind of do theproduction piece like that. Now,
for me, it's a similar thing tolike if you think of like
journalists in media, right?
Everybody says, oh, right,everything's going online. We're
not going to have anyjournalists. Well, if you think
(33:19):
at the end of the day, ajournalist is a storyteller
telling you about kind of thefacts of what's going on. For
me, the data science is asimilar piece where it's still
kind of a mission, a work thatwe're doing in terms of we're
helping a business solveproblems by looking at their
data. The tools have changed,but the same problem still
(33:41):
exists.
And I think this is the mostimportant thing is you still
need a flexible mind as a datascientist to be able to look at
data, to be able to talk to astakeholder, to be able to go
out and figure out what is thecoding, what is the math, the
algorithms to bring to solvethat problem where you need a
lot of this kind of left brain,right brain stuff. And so it's
(34:03):
still a fairly unique role. Andyou can see this where you start
talking to like AI engineers,where you have developers that
are trying to kind of bridgethis and solve the business
problems. And we see one of thebiggest problems they have is
with evaluations. And for datascientists, they're trained on
(34:23):
how to do evaluations coming up.
Like you look at the data, youtalk to people like error
analysis, something that's builtinto kind of data scientists.
But I always look over and seelike how we have to kind of
teach software developers thatskill if they really want to be
able to kind of do the same kindof work like that.
Daniel (34:41):
Yeah, that's super interesting. I kind of posed the
question in maybe a little bitof a controversial way. I think
I echo what you're saying. Imean, there's elements of this
in certain cases, in certainindustries where, hey, if you're
(35:03):
using computer vision to analyzeparts coming off of medicine
coming off of a pharmamanufacturing line and needing
to do that 10,000 times aminute. This is not a problem
that is like, hey, just promptan LLM.
On one side, there's very hardmodeling problems that need to
(35:27):
be solved there. I think on theother side, to what you're
saying, I also see this gaparound AI engineering where
it's, Okay, we can architect thepipeline. And where I see a lot
of people spinning their wheelsis saying, Well, it seems like
this is working. That's kind ofwhere they end. And like, Well,
we could also measure if it'sworking and construct a test set
(35:50):
and maybe automate that.
And as we update our pipeline,we could test the retrieval and
those sorts of things. Love thatperspective around the
evaluation especially. I guessyou see that side being stressed
more and more as maybe softwareengineers see that the future is
(36:11):
AI related and really want topush into that?
Rajiv (36:14):
So I see that there is kind of a growing emphasis for
developers and engineers tounderstand evaluations and do
that. My thing is that it'salways going to be a little bit
of attention for those folksbecause often the folks that are
really good at softwaredevelopment have a very black
and white way of looking at theworld, that they focus on
(36:37):
optimization, that there is abest solution, which necessarily
isn't the same type of mindsetthat you need. And of course,
this is a graduated spectrum.Everybody's a little different.
So this is where I think there'salways going to be that gap
between just having kind ofsoftware developers fully step
into it.
But I want to take up one otherthing that you say is a lot of
(36:59):
times, like the hype we seearound generative AI and NVIDIA
and stuff kind of draws out andmakes kind of the problems that
we can solve with generative AIkind of much bigger than I think
the actual usefulness of them.And what I mean is that there's
a lot of problems inside anenterprise that can be solved
without large language models.And my worry is the folks inside
(37:23):
them that have been doing datascience for ten years know that.
They know that I could useoperations or optimization to
solve this problem, or this is atime series problem to do that.
My biggest fear is the peoplecoming into kind of AI and data
science nowadays aren't seeingthose types of problems and
understanding that there's awhole set of tools to be able to
(37:44):
solve those problems where oftenkind of everything comes.
We're using generative AI as thehammer for solving every type of
problem like that.
Daniel (37:52):
Yeah, maybe this exists out there. And so I'm already
building something, so someonecan totally steal my idea if
they want. But I wonder if thereexists out there this kind of
idea of some sort of assistantthat would live at the
orchestration level above thesekind of traditional data science
(38:15):
tools and help you towards thatanalysis. So just by way of
example, I'm assuming you couldput Facebook's profit for time
series forecasting behind an MCPserver and be able to have that
discovered by this orchestrationlevel and maybe guide people to
that. Now, that might not be theinterface that you want to have
(38:36):
for your time series modeling inproduction, but it could
potentially guide folks to someof traditional data science
tools and kind of help teachthem maybe what they need to put
in place that's not on the largelanguage model side and actually
have the large language modeltell you that, Hey, I'm not the
(38:59):
best for this and you should useFacebook profit or whatever.
Rajiv (39:04):
Yeah, I'm hoping that we'll be able to get to that. I
think there's some element ofThese models are great for
brainstorming, thinking throughthings, solutions through, too.
But if the space is too large ofpossibilities of different ways
to slice the problem, differentways to think about how you
could set the predictions orwhat data to use, then even in
(39:26):
LM, you're not going to be ableto feed it all the relevant
information to be able toactually kind of make that
decision. This is where ashumans, we have to kind of often
be the piece that takes in a lotof that disparate information
and figure out like, okay, thisis what the business really
focuses on. Let's zoom it downto this piece.
And now I'm going use my LM tohelp me think about, hey,
there's three different toolshere and strategies, like tell
(39:48):
me the trade offs, let's figureout which I was doing this
earlier today, like whichpackage should I spend my time
learning how to use to solvethis problem.
Daniel (39:55):
Yeah, yeah, that makes sense. Well, Rajeev, I'm sure
we'll have many other greatconversations at the Midwest AI
Summit and at upcoming or futurepodcast episodes. But as we kind
of get closer to the end hereand you look out towards the
future, what is it that kind ofexcites you about the next steps
(40:18):
of our journey in this space?
Rajiv (40:21):
Yeah, no, I mean, it's just been a great time of
innovation inside of datascience, which is why I love it.
I mean, everything from kind ofgoing from XGBoost to CNNs to
kind of where we are now. And soI'm looking forward to more
innovation, especially in kindof the area of large language
models. But I also want toremind people, like we were
talking about, there's a greatwake of tools that are out there
(40:44):
that I still like to pointpeople to that might not get the
most attention, but there's alot of times a more efficient
way of solving your problem aswell.
Daniel (40:52):
Yeah. And where would you Obviously, you produce a lot
of content and that sort ofthing, but as just a person
that's more intimately familiarwith that kind of ecosystem, if
folks are like, Hey, you know, Iheard, for example, I'm at a
Rally Innovation Conference herein Indianapolis today, off in a
(41:15):
corner, and I heard KevinO'Leary from Shark Tank this
morning, he was saying, Everyday you should spend 30% of your
mental capacity trying somethingnew, like keep those juices
flowing. So maybe it's ourlisteners today, they're taking
away, hey, I should try one ofthese non Gen AI things. Like,
(41:37):
where would I even go to tostart that? Any any suggestions?
Rajiv (41:40):
Yeah, no, I love that idea of like spending a thirty
minutes or an hour a day, likecontinual learning is the is the
future like that. So I have myown content that I put out at
logistics that tries to kind ofinspire you to push you in
different ways that kind of AIis doing, give people simple
kind of nuggets like that. So Iwould of course kind of point to
(42:01):
myself as well. I think theother area that I really like
are newsletters. I thinknewsletters are a nice way to be
able to take in all theinformation that's coming in,
but in a little bit of a slowerkind of meditative way rather
than just kind of reacting tothe latest trending post.
Daniel (42:18):
Yeah, that's awesome. And I'm sure we'll include a few
links in our show notes tothings that will be useful for
people. But really appreciateyou joining us again, Rajeev.
Looking forward to seeing you inperson. Yeah, keep up the great
work.
(42:38):
It's always good to hear yourperspective and looking forward
to having you on the show again.
Rajiv (42:43):
Thanks so much. I think this is one of the longest
running data science podcastsout there, so it's been great to
be part of it. Thanks so
Daniel (42:49):
much. Thanks.
Rajiv (42:57):
All
Jerod (42:57):
right, that's our show for this week. If you haven't
checked out our website, head topracticalai.fm and be sure to
connect with us on LinkedIn, X,or Blue Sky. You'll see us
posting insights related to thelatest AI developments, and we
would love for you to join theconversation. Thanks to our
partner Prediction Guard forproviding operational support
for the show. Check them out atpredictionguard.com.
(43:19):
Also, thanks to BreakmasterCylinder for the beats and to
you for listening. That's allfor now, but you'll hear from us
again next week.
Welcome to the Practical AI podcast, where we break down
the real world applications ofartificial intelligence and how
it's shaping the way we live,work, and create. Our goal is to
help make AI technologypractical, productive, and
accessible to everyone. Whetheryou're a developer, business
leader, or just curious aboutthe tech behind the buzz, you're
(00:24):
in the right place. Be sure toconnect with us on LinkedIn, X,
or Blue Sky to stay up to datewith episode drops, behind the
scenes content, and AI insights.You can learn more at
practicalai.fm.
Now, onto the show.
Daniel (00:49):
Welcome to another episode of the Practical AI
Podcast. I'm Daniel Wightnack. Iam CEO at Prediction Guard and
not joined by Chris today, butvery happy to be joined by a
longtime friend of of thepodcast and and friend of all
things AI and data scienceonline, Rajeev Shah from
(01:11):
contextual AI, where he's thechief evangelist, and also, of
course, TikToker extraordinaireall things. So welcome back to
the show, Rajeev. It's great tohave you.
Rajiv (01:23):
It's great to be back. Excited to talk.
Daniel (01:25):
Yeah. And we're we're close neighbors around the the
Midwest as well. I'm lookingforward to seeing you at the the
Midwest AI Summit, which if iflisteners don't know, both Chris
and I will be at, the Midwest AISummit, which is happening
November 13 in Indianapolis. SoI think as you put it on
LinkedIn, Rajeev, if you liveanywhere near Corn and you like
(01:49):
AI, then this is the place tobe. And actually Rajeev will be
there giving a great talk aboutkind of this idea of, I think
you're going to be talking aboutthis stat, the ninety five
percent of pilots that fail, whythat might be.
Yeah, so great to have a fellowMidwest AI Silicon Prairie
(02:11):
friend on the show as well.
Rajiv (02:12):
No, it's great to have these in person events here in
the Midwest. Usually you have tokind of fly out to some large
city to get that. It's nice tobe able to kind of get that
right in Indy.
Daniel (02:21):
Yeah, yeah. So if you're interested in that, go to
midwestaisummit.com and makesure and register for that.
Hopefully we'll see you there inperson. It's going to be a fun
time. But it has been a whilesince we've had you on the show,
Rajeev, and I'm sure there's alot to catch up on.
It almost seems like last timewe talked, there was kind of we
(02:45):
were all getting into Rag andthinking about agents and
reasoning models. We're tryingthings. People were gradually
developing maybe what theythought would be a direction for
best practices or ways toapproach problems. It would be
interesting to hear from then tonow how you've seen that world
(03:07):
advance in these archetypal usecases that people are addressing
with retrieval relatedframeworks?
Rajiv (03:18):
Yeah, no, it's been amazing. About two and a half
years ago, we were really giddyabout the possibility of AI, but
there was a lot of differentthings that we can do. And kind
of since then, we've seen AIdevelop in certain ways where,
for example, in code completion,code development tools, it's
accelerated dramatically, evenover the last year like that.
And we're all kind of usingthose tools on a regular basis.
(03:41):
And then in other areas, right,like chatbots, for example,
we've seen some progress, butwe've also seen that there's
some things that are difficultto do with chatbots as well.
But I think, you know, it's beenincredible to see just the
continued development of AI interms of the capabilities, the
smartness, the ability to kindof use tools to find search
information that there's stillso much for all of us to do in
(04:05):
our lives, kind of for those ofus that work with AI.
Daniel (04:07):
Yeah, yeah. And I guess, certainly there's a lot of
people still I guess when Iinteract with customers or hear
from other people in theindustry, still a lot of people
are wanting, as their firstthings, to kind of whatever is
build a rag chatbot, which maybefor those listeners out there
(04:28):
that are maybe not as familiarwith this, they've maybe heard
rag this and raggedy rag thatand all of that. Maybe just
remind listeners what rag is andhow retrieval may fit into some
of this AI stuff.
Rajiv (04:46):
Yeah. And I'll start simply. Even a couple of years
ago, I was kind of working withthe earliest large language
models and people would be like,I like ChatGPT, but it doesn't
know anything about me or mycompany. Like, how can I train
it to know all that knowledge?And what we figured out is that
it's not about really trainingthat model about the knowledge.
It's instead finding thatknowledge, searching that
(05:08):
knowledge, passing that on tothe model, and then be able to
use that. And that's where kindof this idea of retrieval,
finding stuff, and thenaugmenting the generation,
augmenting the written responsefor that came about. And I think
RAG today is kind of one of themost important or it's one of
the most widely used use cases,I'd say,
Daniel (05:29):
the general AI space. Yeah.
Rajiv (05:31):
Every company is probably running some type of rag at this
point for searching its internalknowledge, helping folks figure
out through HR documentation,using it for customer support.
There's a lot of use cases wherewe have huge amounts of
information. We want to be ableto find things from that. But we
also want to use the smarts ofAI for that. We don't want a
(05:54):
Google search results of 20things.
We want a nice AI summary of itor we want to extract out all
the information. I just want thenames and dates out of all this.
I don't need to read every one.And so that's where the AI
really comes into it and why Ragis so powerful and we see it so
widely used.
Daniel (06:11):
Yeah, and I love what you were saying, just sort of
contrasting this with trainingor even fine tuning. I find
this, of course, to be a verywidespread misconception about
how these tools work. EvenChatGPT, the application, it
does seem like it kind of quotetrains on your information that
(06:33):
you put in, your previous chathistory, etcetera. And then it's
almost like you have a model ofyour own, right? It almost seems
like OpenAI has a separate modelfor every person on the planet,
which is not feasible from thedata science training
perspective.
So could you highlight some ofthose things and highlight, I
(06:56):
know you educate a lot as well,so I'm sure you've seen this as
well. There's kind of thisjargon of training thrown around
a lot, which is confusing maybein how it's used. Yeah.
Rajiv (07:08):
Yeah. And I think partly is a lot of these companies want
to make you think that they haveone thing that does everything
in the world. But if youactually take a look at
something like ChatGPT, it's asystem. It actually composes of
multiple parts. So if you weretrying to build your own
ChatGPT, these are some of thethings that you would have to
think about.
(07:28):
So one is I need to be able toretrieve from lots of different
sources. I have all my knowledgeinside my companies, inside of
Confluence. I need to be able toaccess and retrieve that. That
could be one thing. So one thingis using Rag, being able to
access tools.
Another thing is memory. Like Iwanted to remember that last
conversation I had, whether itwas two things I said ago or
(07:52):
like one of the nice thingsabout ChatGPT is you can come
back a conversation or a daylater and it remembers a lot
about you. It could maybe alittle bit scary sometimes how
much it remembers about you andhow well it can profile you. And
so this is kind of what we seeinside of AI engineering as
context engineering, wherethere's a number of different
(08:12):
parts of managing interactionswith these models, whether it's
rag, whether it's rememberingthe memory, when it's knowing
how to summarize conversationsto do things like multi turn so
we can have those repeatedconversations, but keep track of
what was said earlier in ourconversations as well.
Daniel (08:31):
Yeah, so I love that idea of context engineering. How
is that idea of contextengineering kind of different
from what we would maybe fromthe traditional data science
side consider model training, Iguess? So there's almost like a
pipelining mindset versus anactual GPU model training piece.
Rajiv (08:56):
Yeah. I think what's remarkable nowadays is the
amount of information that'sstored inside of these large
language models. I do a thingwhen I kind of talk about large
language models where imagine ifyou were sitting outside and
kind of reading a book and youcould read all day, all night.
If you read for ten years, Ithink that's on the order of
something like about a billiontokens. And these models are
(09:20):
trained on the order of like15,000,000,000,000 tokens.
So it's just an inconceivableamount of information that these
models hold inside there. And Ithink one of the biggest
improvements we've seen in thesemodels is not only kind of
stuffing all that information inthere, it's effectively using
all of that information inthere, that these models really
(09:40):
today are untapped in terms ofeverything they can do. And it's
one of these things on thetechnical side is why you see
that the capabilities of themodels have continued to grow
without them necessarily havingto grow immensely in size simply
because we're better tappinginto all the abilities that they
have inside them.
Daniel (09:59):
Yeah. And I know you're up for this because we know each
other, but I'll present ascenario to you that sometimes
comes up for us, whether I'mteaching a workshop or
something, and you can help usunderstand how this retrieval
type of thing fits in. So Ioften Maybe I'm talking to folks
(10:21):
in healthcare, and what'sinteresting is there's all of
this huge amount of data thatthese models have been trained
on. And there is good, let'ssay, medical information in that
data, right? Or there's careguidelines for nurses or medics
or whatever it is.
Let's take that as an example.The interesting thing is there's
(10:43):
even if you think aboutfactuality in a situation, it is
kind of relative in the sensethat if I'm a medic that works
for this provider, let's saytheir care guidelines are
different from another provider.Or if I'm treating maybe
children, those guidelines areprobably different than if I'm
(11:05):
treating or senior adults orsomething like that. So there
can be these facts out therethat actually conflict with one
another. It's really You talkedabout the context.
It's really about the contextthat you're working in, what it
should pull. So these models,assuming that all that
information is public, right,like you said, and it's been
scraped off the internet, all ofthat's kind of You could imagine
(11:29):
that maybe somehow that isembedded somewhere in the model,
the base model, so whether we'retalking about LAMA or GPT or
whatever. Let's say I'm ahealthcare company, but I'm
interested context. Now, how doI make that connection? Still
use maybe, I'm not going toretrain that model, but how do I
(11:52):
use the model then in thiscontext of kind of conflicting
facts?
Rajiv (11:56):
Yeah, absolutely. As much as the models know and been
trained so widely, they don'tknow everything. And so
sometimes that's you have yourown information. You want to
pass that into the model. And sothat's where that retrieval
augmented generation comes in.
We want to grab that informationand and then we want to pass it
into the model. And this iswhere for folks who are in the
space for a while promptengineering comes into play,
(12:20):
where we think about the inputsthat go into the model, where we
take the facts that we knowabout our own given situation,
along with maybe someinstructions for what we want
along with it. And that all ofthat is manipulating the
context, right? The larger worldthat this model is sitting
within there. And bymanipulating and giving it
information, instructions andother facts, we can get outputs
(12:42):
that better match kind of whatwe're looking for like that.
Daniel (12:46):
Yeah, yeah, that's great. And I guess there's
another term that's been thrownaround the last couple of years.
One would be this kind ofretrieval piece, which you've
talked about, and this idea thatin the context of a certain
question or something, I'm ableto connect to one of my internal
data sources, get that rightThere's this other thing that
(13:08):
would be related to reasoning.Now, some people might just
consider that all generative AImodels reason in one way or
another, but it is now almostlike a term of art. It means a
certain thing talking aboutthese models.
Could you help highlight thatalso to put that in context? And
(13:29):
then maybe we'll circle aroundand combine some of these
things.
Rajiv (13:32):
Yeah. A lot of times when we talk about models, we want to
make it convenient to helppeople explain things until we
kind of anthropomorphize them,right? We give them human like
qualities when they're notactually human like qualities.
And so, for example, reasoningis something we say for these
models. Now they're notreasoning like a human would
through a problem.
But what we typically mean kindof when we think about reasoning
(13:54):
with these models is they'redoing lots of extra steps and
they're doing these steps in alogical way to better solve a
problem. So if you take kind ofa math word problem, a great way
to see this is if you take aword problem like the train is
moving east, you know, at 50miles an hour and another train
is moving west another 40 milesan hour and you have to figure
(14:17):
out like what point they'regoing to cross, right? There's
no kind of quick answer to that.You need to kind of calculate
the first train, calculate thesecond train, and then you can
figure out the solution. Andwhat we've done, and that's what
these reasoning models havedone, is we've trained them that
don't come up with the answerright away.
Think through it. Think aboutthe first thing. Think about the
second thing. Connect all thedots and then put an answer in.
(14:41):
And the way we've done this iswe've literally given the model.
Well, we've trained it inmultiple ways, but in some ways
we've literally give the modelan example of, hey, this is how
I solve this word problem. I didthis all these steps here. I
want you to learn how to gothrough these problems step by
step. And this is where wediffer from two and a half years
ago when the first models didn'tknow how to do that. Those first
(15:02):
models, all we had trained it upwas like, hey, tell me if a
movie is a good movie or a badmovie or tell me if this
sentiment is happy or sad.
But since then, we've had timeto develop more training data.
We've given them more complextraining. And the cool thing is
the models have picked up.They've been able to learn this
capability for doing this. Andso now we're able to do this
much more complex kind of I'mdoing the air quotes of
(15:25):
reasoning to solve theseproblems.
Daniel (15:28):
Yeah, I guess one thing that people, and this is part of
why I really enjoy watching yourvideos online, is you often
break down a lot of this jargonand kind of help. It's almost
like people feel the shock ofall of this terminology and a
new model coming out every 30minutes or whatever it is, and
(15:51):
just not really knowing how todeal with that. And there's kind
of all of these things that havehappened. So there's reasoning
models, there's small languagemodels, there's tool calling,
there's retrieval, there's allof these different mix of
things. Based on yourexperience, both what you've
implemented personally, alsowhat you've seen and interacted
(16:13):
with folks on, if people couldbring in some focus and maybe
imagine a company's getting intoAI, however they view AI
transformation within theirorganization, and they're
thinking about some of the usecases that are the initial ones
(16:35):
on their roadmap at the timebeing, what would you encourage
them to pick the signal out ofthe noise?
So what would you encourage themto maybe focus on to see some of
that time to value upfront? Notthat they don't want to explore
some of the other topics thatare happening or read about them
(16:55):
or whatever that is, but howwould you at least recommend to
get that best time to value, ormaybe just the things that are
producing the most value? Andyou could maybe flip that as
well and say, what are some ofthose things that are cool
things, but maybe let's wait andsee what happens and maybe just
(17:16):
sort of don't get distracted atthe moment.
Rajiv (17:18):
Yeah. Mean, you're taking me away from all the fun, cool
technologies that are thelatest, like automation, things
that I can kind of fire up likethat. I think when you're
thinking about this from theperspective of kind of a
company, if you're kind of amanager in these situations, you
have to figure out what usecases you want to put on a very
different hat than just thinkingabout the technology itself. And
(17:42):
I can tell you this, like I wasburned from personal experience
when I was just starting out indata science. I was entranced by
the latest technology.
So I remember, right, we weretalking about code development.
Like ten years ago, I wasworking at State Farm and I
think Andrij Karpathy haswritten his paper about the
unreasonable effectiveness ofLSTMs. It was something like
(18:06):
that. But part of that paper hadthe idea of you could translate
code, you could complete codefrom one language to another.
And I was like, hey, come on,guys.
Like, we've got a lot of thiscobalt code sitting around.
Like, give me some give me a fewGPUs and some data and I can
Daniel (18:22):
do it.
Rajiv (18:22):
I'll solve this. Naive. They didn't fund that project or
anything, but I think, yes, itcan be very easy to be kind of
seduced by the technologies,kind of what a shiny demo is
versus when you're in anorganization, you really have to
think about kind of the problemsthat you have. Part of it will
be, you know, how complicatedthis is from a technical point
(18:44):
of view to get it up andrunning. That's one factor, But
that's not the only factor.
We also need to consider what'sthe value to this organization.
I talk to lots of enterprises ona regular basis and I see often
what I call science experimentswhere teams like the latest
technology, they go out and kindof run this stuff, but there's
(19:05):
no way for them to actually getthat implemented inside the
company in a useful way. Andthey're literally just kind of
interesting experiments thatpeople are running like that.
Daniel (19:14):
Does that get partially to like the ninety five percent
of AI pilot failing type ofreport from MIT?
Rajiv (19:23):
Absolutely. Now, the 95%, of course, is like a little bit
of a hype number that they liketo put out in this. And for
those of us who've been in thespace for a while, we remember
the fact of 80% of data scienceprojects fail, I think was
something that we had. And tosome extent, that's okay. You
can't expect every initiative,every experiment, everything
(19:44):
that you start to succeed.
You want things to fail becausepartly is if something works,
that means you have to maintainit. You have to monitor it. You
have to put up a lot of guardsaround it. There's a cost for
something that actuallysucceeds. But when we talk about
AI, it's very easy to build acool demo, but it's not only the
value to the company.
(20:05):
You have to figure out how tointegrate this into people's
everyday work life. And so youcan build a very shiny widget
that sits and can do somethingawesome. But if that's not
inside somebody's regularworkflow of how they work, the
tools that they work in, ifthey're not properly trained on
how to use it, if theirleadership isn't supporting you
(20:26):
to use it, there's lots offactors like that that go into
why people might not actuallyadopt and use a technology. And
it's really nothing about AI.It's really about organizational
change and introducingtechnology into companies.
Daniel (20:40):
Yeah, I know from just a founder perspective, can be,
know, just from a different sideof this, it can be frustrating
when you see like, Oh, there'sthis company over here that the
technology side is fairlysimple. Like, oh, it's just a
simple model that does a simplething, or a browser extension
(21:02):
that does this. And you're like,wow, I could have vibe coded
that in a weekend. How are theyscaling to the moon? We have all
this cool technology.
And I think part of it is thatside of it that you talked
about, part of the hard problemis cracking what actually does
(21:24):
provide value to yourorganization, what can be
adopted, how you communicatethat, how you tell that story,
and how you deliver on yourpromises, how you provide
customer support. A lot of thatis not really related to the
technology and that component.So maybe, I don't know if this
(21:45):
would be accurate to say, butmaybe the first step people
could take is just gettingsomething off the ground that's
fairly simple and interacts withan AI model. Maybe it's just to
do a very simple task, butreally pushing that through,
like you say, to be embedded ina product or be embedded in a
process. That may be the bestway for people to kind of start
(22:09):
that journey is to really startfrom that simple side and deal
with some of, in all honesty,the harder problems around the
periphery of the technology.
Rajiv (22:19):
And I think a big part of that, like what you're saying,
is just to get closer to thoseend users, the stakeholders,
because I think once you oftencut through that, sometimes you
figure out that really theydon't necessarily need a fancy
kind of GPT-five model to solvetheir problems. Maybe you can
solve it with almost a simple ifthen rule that you can just
(22:41):
implement in And some so this iswhere kind of looking at the
data, spending time talking tothose end users that often gives
you a much better result. It'sgoing to give you the biggest
bang for your buck than goingout and reading some archive
paper.
Daniel (22:57):
Yeah, yeah, that's true. And I guess now you should just
have the AI model read the paperfor you and give you some
summary points. A big fan
Rajiv (23:08):
of that. On my walks, often just sit and I'll talk to
Chad GPT and we'll talk throughpapers and what are the main
technical points and stuff.
Daniel (23:14):
Yeah, I'm glad I'm not the only one. Thanks for
validating that. I guess gettingback more into the development,
retrieval kind of reasoningstuff, what are some of the Now
that we've been working withthis technology for a while,
we've got more cycles. Someonecan spin up a rag pipeline in
(23:35):
whatever, ten minutes I can spinit up and I have something
going. But it's another thing tokind of, of course, scale that,
maintain it over time, deal withsome of the issues.
I guess my question is whatpitfalls are people falling into
that maybe we didn't know aboutwhatever it was a year ago when
we were kind of just gettinginto these initial kind of naive
(23:57):
rag sorts of things? Whatchallenges or consistent
challenges and pitfalls do yousee people kind of falling into?
Rajiv (24:05):
Yeah, think the consistent thing I see with
something like RAG is it'sfairly easy to build a quick
demo. You can grab an off theshelf embedding model to do
that. You can combine that witha generation model like an
OpenAI model and you can buildyourself a quick kind of proof
of concepts. I think the troublethat people get into with it is
(24:25):
scaling it up. It's great on 100documents, but now all of a
sudden I have to go to 100,000or 1,000,000 documents.
How am I going to do that? Orwhen I first did my demo, I did
a couple of very simple queries,text extraction queries. But now
when I put it out in front of myusers, I find out all of a
(24:45):
sudden they're not giving niceone sentence queries. They're
just asking two words and then Ineed to add a query
reformulation step or somethingto do that. Or the accuracy is
not kind of what I was lookingfor.
And so I've added a bunch ofpieces in there. I've added a re
ranker. I've added other steps.But now my latency is suffering,
(25:08):
right? There's all these kind oftrade offs as you kind of get to
production.
And then you're like, oh, youknow, do I go back and you go
and you look online and you seethat, oh, wait a minute, there's
like 15 I think there's like 25,30 flavors of rag. You're like,
oh, did I set up myinfrastructure wrong? Oh, do I
go back and I change my chunkingstrategies? I can see there's 10
(25:29):
different chunking strategiespeople are doing that. And so I
think this is the cycle of whereit's very easy to get started,
but getting to that final kindof production quality rag can
kind of be a little worrisome.
Daniel (25:40):
Yeah. And do you think that that's where the real human
engineering piece of thisdevelopment still will be with
us for some time? Because tosome degree, you can describe
let's say describe that sort ofproblem to my AI coding
assistant. Is it reasonable forme to think that that kind of
(26:02):
debugging and process could helpme, or that kind of assistant
type of thing could help me getto the bottom of this or update
my retrieval pipeline and thatsort of thing?
Rajiv (26:13):
So I'm kind of optimistic that the reasoning models that
we have now are going to get usmuch farther towards helping you
solve that problem. Now, ofcourse, that reasoning model's
got to understand how you'rethinking about how to solve that
problem. But already today, ifyou take the traditional Rag
approach, for example, but youpair it with one of the
reasoning models that can maketool calls, can kind of look at
(26:37):
the results that come back,think about it, decide, hey, I
want to re query it in adifferent way. You can improve
the quality of your results inthat way by using that reasoning
to do that. So I'm prettyoptimistic that we're going to
keep finding new ways as long asthere are workflows that we can
train these models on that arefairly, let's say, logical.
Daniel (26:57):
Or typo. Yeah, exactly. Gotcha.
Rajiv (27:00):
A way we can connect the dots and teach the models to do
that, that I think a lot ofthese things that if we give it
the budget for spending timethinking, doing those tokens
there, it's going to cost usmore latency, but we're going to
see better results. And I thinksome of us, we already see that
in using some of these toolslike deep research, where we can
(27:20):
see that by spending more timeon the task, it's able to give
us a better result.
Daniel (27:25):
Yeah. Yeah. And one area that we haven't talked about yet
is this sort of world of agents,which I know is loaded term,
it's probably related to whatyou were just talking about in
terms of time of compute andsteps in the process and the
reasoning models and all ofthose sorts of things. Could you
(27:47):
help us parse apart from yourperspective? It almost seems to
me like it's one of those sureyou remember when it seemed like
every conversation I got on, thefirst part of the presentation
was, what is the differencebetween data science and AI and
machine learning, or thedifference between machine
(28:09):
learning and AI or whatever.
And at a certain point, I waslike, well, these terms all mean
nothing essentially becausepeople use them so
interchangeably. I feel likethat's sort of where we're
getting with agents andassistants and all of these
sorts of terms. But from yourperspective, I guess if that
word agent has a meaningfuldifference to you, what stands
(28:32):
out in your mind there?
Rajiv (28:33):
Yeah, and I think we all like the idea of this agent,
right? Like something I can givea problem to and it solves the
problem. And now I think wherethe definitions break down to is
how much autonomy is this agent,how structured is what we do
like that. But if we just thinkback about the bigger picture of
like, I have a problem, it's nota straightforward problem that I
(28:56):
want to give to an agent. Now,there's at least two different
ways that we can kind of tacklethis.
One is I can give them a step bystep list of instructions. And
this is what we call a workflowoften, do this, do this, do
this. And then I can check theirwork at every step and make sure
that they're on the track tosolve it. Or I can just be like,
this is the difference maybebetween my kind of five year old
(29:17):
and a 13 year old, my 13 yearold, like I'll give them the
list and I'll cross my fingersand hope that they'll finish it.
And, you know, usually they do,but not always.
But I'm not involved in everystep of the way. And I think one
of the things is we're watchingthe agents evolve and this is
one of the big trade offs thatdevelopers have today is how
much structure, how muchbabysitting am I doing for this
(29:39):
agent? How do I do versus kindof the hands off? Now, I think
the trend is we're going to beable to do much more hands off.
Just like we've seen thesemodels be able to gain the
reasoning ability over the lasttwo years or so, I have no doubt
that we're going to be able totrain them to do more complex
tasks, to be able to followthose steps.
(30:01):
It's just a matter of kind ofgiving them the training data,
having the experience to dothat. So my bet is in the long
term for many of these tasks,we'll be able to be much more
hands off and the modelsthemselves will strive to be
able to solve them themselves.
Daniel (30:17):
I just thought it would be good to get your input on a
theory I've been having, whichis maybe related. I mean, maybe
it's an offshoot from what we'retalking about, which is really
maybe the ability to use theseVibe coding tools or others to
or assistance or agents toupdate retrieval processes or
(30:39):
kind of architect our AIpipelines, if you will. I've had
this sort of thought, and I'mcurious about your opinion
because you also have abackground kind of pre
generative AI in the datascience world that previously I
had in my mind this mental modelof, on the one side, you have
traditional softwareengineering, DevOps,
(31:03):
infrastructure. On the otherside, you have business and the
product and marketing, all thosethings. And in the middle is
data scientists because youtranslate the business problems
and understand how to connect itto the data and the tech and
produce your predictive model.
You're living between those twoworlds. It's almost like I see
(31:26):
that middle zone shrinking andshrinking and shrinking because
those domain experts on thebusiness side are actually able
to use very sophisticated toolsnow to kind of self serve
themselves a lot of the kind ofmaybe stuff that would normally
fit on the plate of a datascientist. So part of me is
(31:48):
wondering, me personally, likeDaniel Witenack as a data
scientist, what is the future ofthat data science world when
this middle is shrinking? I'mcurious if you also see it that
way or see it slightlydifferently, and what your
thoughts are in terms of thatview. Yeah.
Rajiv (32:07):
So I don't think the data science world is changing at
all. First of all, I'm excitedthat the bar is kind of dropping
in terms of people being able touse code to build solutions.
Like my nine year old canliterally like vibe code of a
game that he can play as well asmy 23 year old who has a who has
a degree in computer science.And you couldn't tell the
(32:28):
difference between like thegames that they built like that.
So there's a great ability injust allowing everybody to be
able to kind of more participateand work kind of with code that
we're being able to see now.
Now, how does that changesomething like data science?
Like data science, the originaltriangle was, part of it was
coding. I think data scientistswere never thought of as really
(32:51):
great coders, which is why were80 trying to put in
Daniel (32:56):
percent of those projects didn't make it past
pilot too.
Rajiv (32:59):
Exactly. Right. Like they would not write production code
and right. There was MLEengineers kind of became the
offshoot to kind of do theproduction piece like that. Now,
for me, it's a similar thing tolike if you think of like
journalists in media, right?
Everybody says, oh, right,everything's going online. We're
not going to have anyjournalists. Well, if you think
(33:19):
at the end of the day, ajournalist is a storyteller
telling you about kind of thefacts of what's going on. For
me, the data science is asimilar piece where it's still
kind of a mission, a work thatwe're doing in terms of we're
helping a business solveproblems by looking at their
data. The tools have changed,but the same problem still
(33:41):
exists.
And I think this is the mostimportant thing is you still
need a flexible mind as a datascientist to be able to look at
data, to be able to talk to astakeholder, to be able to go
out and figure out what is thecoding, what is the math, the
algorithms to bring to solvethat problem where you need a
lot of this kind of left brain,right brain stuff. And so it's
(34:03):
still a fairly unique role. Andyou can see this where you start
talking to like AI engineers,where you have developers that
are trying to kind of bridgethis and solve the business
problems. And we see one of thebiggest problems they have is
with evaluations. And for datascientists, they're trained on
(34:23):
how to do evaluations coming up.
Like you look at the data, youtalk to people like error
analysis, something that's builtinto kind of data scientists.
But I always look over and seelike how we have to kind of
teach software developers thatskill if they really want to be
able to kind of do the same kindof work like that.
Daniel (34:41):
Yeah, that's super interesting. I kind of posed the
question in maybe a little bitof a controversial way. I think
I echo what you're saying. Imean, there's elements of this
in certain cases, in certainindustries where, hey, if you're
(35:03):
using computer vision to analyzeparts coming off of medicine
coming off of a pharmamanufacturing line and needing
to do that 10,000 times aminute. This is not a problem
that is like, hey, just promptan LLM.
On one side, there's very hardmodeling problems that need to
(35:27):
be solved there. I think on theother side, to what you're
saying, I also see this gaparound AI engineering where
it's, Okay, we can architect thepipeline. And where I see a lot
of people spinning their wheelsis saying, Well, it seems like
this is working. That's kind ofwhere they end. And like, Well,
we could also measure if it'sworking and construct a test set
(35:50):
and maybe automate that.
And as we update our pipeline,we could test the retrieval and
those sorts of things. Love thatperspective around the
evaluation especially. I guessyou see that side being stressed
more and more as maybe softwareengineers see that the future is
(36:11):
AI related and really want topush into that?
Rajiv (36:14):
So I see that there is kind of a growing emphasis for
developers and engineers tounderstand evaluations and do
that. My thing is that it'salways going to be a little bit
of attention for those folksbecause often the folks that are
really good at softwaredevelopment have a very black
and white way of looking at theworld, that they focus on
(36:37):
optimization, that there is abest solution, which necessarily
isn't the same type of mindsetthat you need. And of course,
this is a graduated spectrum.Everybody's a little different.
So this is where I think there'salways going to be that gap
between just having kind ofsoftware developers fully step
into it.
But I want to take up one otherthing that you say is a lot of
(36:59):
times, like the hype we seearound generative AI and NVIDIA
and stuff kind of draws out andmakes kind of the problems that
we can solve with generative AIkind of much bigger than I think
the actual usefulness of them.And what I mean is that there's
a lot of problems inside anenterprise that can be solved
without large language models.And my worry is the folks inside
(37:23):
them that have been doing datascience for ten years know that.
They know that I could useoperations or optimization to
solve this problem, or this is atime series problem to do that.
My biggest fear is the peoplecoming into kind of AI and data
science nowadays aren't seeingthose types of problems and
understanding that there's awhole set of tools to be able to
(37:44):
solve those problems where oftenkind of everything comes.
We're using generative AI as thehammer for solving every type of
problem like that.
Daniel (37:52):
Yeah, maybe this exists out there. And so I'm already
building something, so someonecan totally steal my idea if
they want. But I wonder if thereexists out there this kind of
idea of some sort of assistantthat would live at the
orchestration level above thesekind of traditional data science
(38:15):
tools and help you towards thatanalysis. So just by way of
example, I'm assuming you couldput Facebook's profit for time
series forecasting behind an MCPserver and be able to have that
discovered by this orchestrationlevel and maybe guide people to
that. Now, that might not be theinterface that you want to have
(38:36):
for your time series modeling inproduction, but it could
potentially guide folks to someof traditional data science
tools and kind of help teachthem maybe what they need to put
in place that's not on the largelanguage model side and actually
have the large language modeltell you that, Hey, I'm not the
(38:59):
best for this and you should useFacebook profit or whatever.
Rajiv (39:04):
Yeah, I'm hoping that we'll be able to get to that. I
think there's some element ofThese models are great for
brainstorming, thinking throughthings, solutions through, too.
But if the space is too large ofpossibilities of different ways
to slice the problem, differentways to think about how you
could set the predictions orwhat data to use, then even in
(39:26):
LM, you're not going to be ableto feed it all the relevant
information to be able toactually kind of make that
decision. This is where ashumans, we have to kind of often
be the piece that takes in a lotof that disparate information
and figure out like, okay, thisis what the business really
focuses on. Let's zoom it downto this piece.
And now I'm going use my LM tohelp me think about, hey,
there's three different toolshere and strategies, like tell
(39:48):
me the trade offs, let's figureout which I was doing this
earlier today, like whichpackage should I spend my time
learning how to use to solvethis problem.
Daniel (39:55):
Yeah, yeah, that makes sense. Well, Rajeev, I'm sure
we'll have many other greatconversations at the Midwest AI
Summit and at upcoming or futurepodcast episodes. But as we kind
of get closer to the end hereand you look out towards the
future, what is it that kind ofexcites you about the next steps
(40:18):
of our journey in this space?
Rajiv (40:21):
Yeah, no, I mean, it's just been a great time of
innovation inside of datascience, which is why I love it.
I mean, everything from kind ofgoing from XGBoost to CNNs to
kind of where we are now. And soI'm looking forward to more
innovation, especially in kindof the area of large language
models. But I also want toremind people, like we were
talking about, there's a greatwake of tools that are out there
(40:44):
that I still like to pointpeople to that might not get the
most attention, but there's alot of times a more efficient
way of solving your problem aswell.
Daniel (40:52):
Yeah. And where would you Obviously, you produce a lot
of content and that sort ofthing, but as just a person
that's more intimately familiarwith that kind of ecosystem, if
folks are like, Hey, you know, Iheard, for example, I'm at a
Rally Innovation Conference herein Indianapolis today, off in a
(41:15):
corner, and I heard KevinO'Leary from Shark Tank this
morning, he was saying, Everyday you should spend 30% of your
mental capacity trying somethingnew, like keep those juices
flowing. So maybe it's ourlisteners today, they're taking
away, hey, I should try one ofthese non Gen AI things. Like,
(41:37):
where would I even go to tostart that? Any any suggestions?
Rajiv (41:40):
Yeah, no, I love that idea of like spending a thirty
minutes or an hour a day, likecontinual learning is the is the
future like that. So I have myown content that I put out at
logistics that tries to kind ofinspire you to push you in
different ways that kind of AIis doing, give people simple
kind of nuggets like that. So Iwould of course kind of point to
(42:01):
myself as well. I think theother area that I really like
are newsletters. I thinknewsletters are a nice way to be
able to take in all theinformation that's coming in,
but in a little bit of a slowerkind of meditative way rather
than just kind of reacting tothe latest trending post.
Daniel (42:18):
Yeah, that's awesome. And I'm sure we'll include a few
links in our show notes tothings that will be useful for
people. But really appreciateyou joining us again, Rajeev.
Looking forward to seeing you inperson. Yeah, keep up the great
work.
(42:38):
It's always good to hear yourperspective and looking forward
to having you on the show again.
Rajiv (42:43):
Thanks so much. I think this is one of the longest
running data science podcastsout there, so it's been great to
be part of it. Thanks so
Daniel (42:49):
much. Thanks.
Rajiv (42:57):
All
Jerod (42:57):
right, that's our show for this week. If you haven't
checked out our website, head topracticalai.fm and be sure to
connect with us on LinkedIn, X,or Blue Sky. You'll see us
posting insights related to thelatest AI developments, and we
would love for you to join theconversation. Thanks to our
partner Prediction Guard forproviding operational support
for the show. Check them out atpredictionguard.com.
(43:19):
Also, thanks to BreakmasterCylinder for the beats and to
you for listening. That's allfor now, but you'll hear from us
again next week.
Popular Podcasts
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.