April 14, 2025 • 42 mins
In this episode, Daniel sits down with Pavel Veller, EPAM’s Chief Technologist, to explore the practical challenges of orchestrating many AI agents and managing connections to disparate systems/tools. Pavel shares insights from his hands-on work with agentic architectures and internal tools like "DIAL". Pavel also helps us understand things like MCP servers and why connecting assistants via APIs is easy—but making them useful is hard.
Featuring:
Links:
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Jerod (00:04):
Welcome to Practical AI, the podcast that makes
artificial intelligencepractical, productive, and
accessible to all. If you likethis show, you will love the
changelog. It's news on Mondays,deep technical interviews on
Wednesdays, and on Fridays, anawesome talk show for your
weekend enjoyment. Find us bysearching for the changelog
(00:24):
wherever you get your podcasts.Thanks to our partners at
fly.io.
Launch your AI apps in fiveminutes or less. Learn how
fly.io.
Daniel (00:44):
Welcome to another episode of the Practical AI
Podcast. This is DanielWitenack. I'm CEO of
PredictionGuard, and I'm reallyexcited today to dig a little
bit more into GenAIorchestration agents, coding
assistants, all of those thingswith my guest, Pavel Veller, who
(01:06):
is Chief Technologist at EPAMSystems. Welcome, Pavel. Great
to have you here.
Pavel (01:10):
Thank you. Hello. Hello.
Daniel (01:12):
Yeah, yeah. Well, I mean, there's a lot of topics
even before we kicked off theshow. We were chatting in the
background about some reallyinteresting things. I'm
wondering if you could just kindof level set us of people may or
may not have heard of EPAM. Ithink one of the things that I
saw that you all were working onwas this GenAI orchestration
(01:34):
platform dial.
Maybe before we get into some ofthe specifics about that and
other things that you'reinterested in, maybe just give
us a background of what EPAM is.I know you mentioned even in our
discussions that some of whatyou're doing right now maybe
wouldn't have even been possiblea couple of years ago, and so
(01:54):
things are developing rapidly.Just level set the kind of
background of this area whereyou're working.
Pavel (02:01):
Sure. Yeah, yeah. EPAM is a professional services
organization. We're global.We're in 50 something countries.
50,000 people globally work withclients. We have been for, I
think, thirty two years to date.We do a lot of different things,
as you can imagine. What I wasmentioning about doing things
(02:22):
that would not be possible isdoing things with GenAI today.
We do a lot of work for our ownclients.
We also do work for ourselves,applying the same technology,
because EPAM historically as acompany has been running on
software that we ourselvesbuilt. The philosophy has always
been that things that do notdifferentiate you, like an
(02:44):
accounting software or like CRM,you would go and buy off the
shelf. Things that differentiateyou, how we actually work, how
we operate, how we executeprojects, how we hire people,
how we create teams, how wedeploy teams. All of that
software has always been our ownsince as early as late '90s, and
(03:06):
we keep iterating on thatsoftware for ourselves. That
software today is very much AIfirst, and a lot of things we
do, we do with AI and really dobecause AI in its current form
exists.
Daniel (03:17):
Interesting. Yeah. And how, I guess, does I think when
we initially kind of wereprompted to reach out to you,
part of it was around thisorchestration platform. So talk
a little bit maybe generally,not necessarily about the
platform per se, although we'llget into that, but just Gen AI
(03:39):
orchestration generally. Youtalked about some of these
things that are becomingpossible.
Where does orchestration fit inthat, and what do you mean by
orchestration?
Pavel (03:49):
You probably think of Dial. You can Google it. We do a
lot of applied innovation ingeneral as a company, and this
is one of the good examples ofapplied innovation to AI. The
best way to think of Dial wouldbe you guys all know ChatGPT.
Right?
ChatGPT isn't an LLM. It's anapplication that connects to an
(04:13):
LLM and gives you certainfunctionalities. It can be as
simple as just chatting andasking questions. It can be a
little more complex, uploadingdocuments and speaking to them,
like Talk to my documents. Itcan be even more complex when
you start connecting your owntools to it.
We see our clients not only dothis, but also want something
(04:33):
like this for their own businessprocesses. This orchestration
engine becomes, How do I make itso that I don't have 20
different teams doing the samesimilar things over and over
again in their own silos? How doI connect my teams and their AIs
and their thoughts and resultsinto a consolidated ecosystem?
(04:56):
It's likely because of GenAI andbecause of what we can do with
conversation and text becomessort of conversation first. You
can think of conversation firstapplication mashups almost.
Right? Like you talk, express aproblem, and what comes back is
not just the answer. Maybe whatcomes back is UI elements,
(05:17):
buttons you can click, forms youcan fill out, things you can do,
as well as things that are donefor you by agents automatically.
Dial, in that sense, is Well, bythe way, it is open source. You
guys can also go look, download,and play with it.
But it is a ChatGPT likeconversational application that
(05:38):
has many capabilities that gobeyond. We have dial apps. They
predate MCP, but the idea isthat you so dial itself has a
contract, an API that youimplement. You basically come
back with a streaming, API thatcan receive a user prompt.
Whatever you do, you do, and youcome back to Dial with not just
(06:01):
text.
It's a much more powerfulpayload with UI elements,
interactive elements, and thingsthat Dial will display for me,
the user, to continue myinteraction. And Dial becomes
this sort of center mass of howyour company can build,
implement, integrate AI intothis single point of entry. And
(06:22):
then Dial goes Well, from dayone, Dial was a load balancing
model agnostic proxy. Right? Soevery model, every deployment
has limits, you know, tokens perminute, tokens per day,
whatever, requests per minute.
You'll likely if you're a largeorganization with large
different workflows, your AIappetite will go well beyond a
(06:44):
single model deployment. You'dlike to load balance across
multiple, and then you'd like totry different models, ideally
with the same API for you, theconsumer. So Dial started like
that. It started like loadbalancing, model agnostic proxy,
single point of entry. We canlog everything that is prompted
in the organization.
We can do analysis on thatseparately because that's very
(07:07):
helpful to know what kind ofproblems your teams are trying
to solve. Then it evolved intothis application hosting
ecosystem. Now it's evolvingclearly towards what MCP can
bring in because now you canconnect a lot more things to it
through MCP. So I think it'srunning at 20 something clients
by now.
Daniel (07:27):
So just a couple of follow-up questions. It's been
in the news a lot, but just sopeople understand if maybe they
haven't seen it, what are youreferring to with MCP and how
that relates to some of this APIinterface that you're enabling?
Pavel (07:44):
Well, the easiest is to Google it. You can Google it,
you're going to find it, it's onClaude. Let me tell you how I
think about this because notwhat it actually That's helpful.
Yeah. Yeah.
I think about it in a verysimple term. So MCP allows to
connect the existing softwareworld to LLMs. In a way, think
(08:05):
like I don't want to hype it toomuch because it's not yet the
global standard or anything.It's very early, early, early
days. It's been months, right?
But let's say HTML and browsersand HTTP, they're enabled to
connect us. People are too softfor all over the world. MCP does
(08:26):
that but for LLMs. Today, if Itoday want to be able to prompt
my application that is in frontof an LLM to do things with
additional tools Let's say Iwanted to be able to search file
system based on what I promptedand find a file and something in
that file, right? So myapplication needs to be able to
(08:48):
do that.
My option is what? I can writethat function. I can then tell
my LLM, Hey, here's thisfunction you can call if you
want to. Call it. I'm going tocall it for you.
Great, that's one function. Whatif I need to do something else?
I want to go talk to my CRMsystem and get something out of
there. I'm going to write thatfunction. If I'm going to write
all the functions I can thinkof, it's going to take me years,
(09:11):
probably hundreds of years.
Instead, what I can do today,can say, Hey, my LLM
application, can you talk aprotocol? Because there's a
protocol called MCP. I'm goingto bring you MCP servers that
other people have built. For myCRM system, for my file system,
for my CLI, there are MCPservers for everything. IntelliJ
exposes itself as an MCP serverto do things that IDE can do.
(09:35):
Now you can orchestrate thosethings through LLM. So you
connect all those MCP serversthrough an MCP client, this
application in front of LLM, toLLM, expose the tools to LLM.
LLM can now ask the client tocall a tool, and through this
MCP protocol, the client callsthe server, the server does the
function that has been writtenin that server, and boom, LLM
(09:58):
gets results. It's thisconnective tissue that did not
exist three months ago. Threemonths ago, everybody was
writing their own.
And right now, everybody, as faras I can tell, writing MCP
servers, and those who talk toLLMs, they consume MCP servers.
Daniel (10:13):
Yeah. And maybe just give so I like the example that
you gave of searching filesystems. Just to kind of expand
people's understanding of someof possibilities, what are some
of the things that you've seenmaybe implemented in Dial as
things that are beingorchestrated? In general terms,
(10:34):
what are kind of some of thesethings?
Pavel (10:36):
Let me give you a higher level and much more sort of
fruitful example, okay? Yeah. Wehave our own agentic developer.
It's called AIRun Codeme,because AIRun has multiple
different agentic systems.Codeme is specifically coding
(10:58):
oriented.
We have others oriented at otherparts of SDLC workflow. By the
way, you guys can go to SWEbench look at verified list. I
believe Codeme as of now takesfifth, it's number five on the
list of all the agents whocompete for solving open source
defects and stuff. Codeme as anagentic system has many
(11:21):
different assistants in it. Dialas a generic front door, as a
ChatGPT, would like to be ableto run those assistants for you
as you talk to Dial.
Until MCP, it really couldn'tother than, Hey, Codeme.
Implement an API for all of yourassistants. Let me learn to call
(11:43):
all of your APIs. Now the storyis, Hey, Codeme. Give me an MCP
server for you, which is whatthey have done.
Dial as an MCP client can nowconnect to all Codeme features,
all the assistants, expose themas tools to an LLM, and
orchestrate them for me. So Icome into the chat, I ask for
(12:07):
something, that somethingincludes reading a code base and
making architecture sketches orproposals or evaluation, right?
LLM will ask Codeme assistanceto go and read that code base
because there is a feature inCodeme that does it, and Dial
needs to only orchestrate butdoesn't need to rebuild or built
(12:31):
from scratch. That's the idea,so this is the example.
Daniel (12:34):
Yeah. Could you talk a little bit I'm asking selfish
questions because sometimes Iget these asked of me and I'm
always curious how people answerthis. One of the questions that
I get asked a lot in respect tothis topic is, okay, I have tool
or function or assistant one,and then I have assistant two,
(12:57):
and then I kind of have a few.Right? And it's fairly easy to
route between them becausethey're very distinct, right?
But now if you imagine, okay,well now I could call one of a
thousand assistants or functionsor something, or later on
10,000, right? How does the sortof scaling and routing kind of
(13:17):
actually how has that affectedas you kind of expand the space
of things that you can do?
Pavel (13:25):
So that I think and again, I can't know, and I don't
know, but I think that is stillthe secret sauce. In a way, that
is still why there is all of allof these coding agents in SWE
bench. All of them work with,let's say, Cloud Sonnet 3.5 or
Cloud Sonnet 3.7 or GPT fourpoint zero. LLM is the same, and
(13:49):
yet results are clearlydifferent. Some score 10 points
higher than the other.
You go to cursor, IDE cursor,you ask it something, it does
something. You switch the modeto max. They've introduced very
recently. Cursor on Sonnet 3.7and now on Gemini two point
zero, I think, they have a maxmode, which is pay per use
(14:12):
versus their normal monthlyplans, because max will do more
iterations, will spend moretokens, will be more expensive,
will likely run through morecomplex orchestrations of
prompts and tools and whatnot togive you better results. How you
build the pyramid of choices foryour LLM, how you Yeah, you will
(14:35):
not ask LLM, you will not giveit a thousand tools.
If you as a human look athousand options and you lose
yourself, a hundred options init, again, I don't know. I
expect LLM to have the same sortof oops, overwhelmed effect. You
don't want to give it a thousandtools. Want to give it groups.
You want to say, Hey, pick agroup.
(14:56):
So you want to do this basicallylike a pyramid, like a tree. But
how you build it and how youprompt it and how you do this,
now that's still on you. This isthe application that connects
the MCPs, the tools that ititself has, the prompt that the
user has given, the systeminstructions, and building some
(15:18):
of the chain of thought LLM canbuild. And this is going to be a
very interesting balance. Whatdo you ask LLM to build?
How much of this sequencing ofsteps will be on you in your
hands versus how much you'regoing to delegate to LLM and ask
LLM to come up with a sequenceof steps. And from what I've
seen over the last year, you'rebetter off delegating more to
(15:42):
LLMs because they get better atit. So the more you control the
sequence yourself, the more sortof inflexible it becomes. You're
better off delegating to a LM,but you don't expect it to just
figure out from one prompt.Daniel, I can give you that
example that I gave in thebeginning, if you want, about
the failure.
(16:02):
Yeah, go for it. I use AI. Ibuilt with AI, but I also use AI
as a developer. I'm on cursor asmy primary ID these days. I use
the AIRunCodemey that Imentioned.
I play around with other thingsas they come up, like Cloud Code
and things, but I also recordwhat I do. Little snippets,
(16:24):
five, ten minutes videos for myengineering audience at EPAM for
the guys to just look what it isthat I'm doing, learn from how I
do it, try to think the sameway, try to replicate, get on
board with using AI. So I setout to do a task. I wanted to,
on record, get a productivityincrease with a timer. My plan
(16:47):
was I'm going to estimate howlong it's going take me,
announce, let's say two hours,do it with an agent, and I
always pause my video when theagent is thinking because that's
a boring step.
But the timer's going to getticking. And at the end, I'm
going to arrive at, let's say,an hour, maybe forty minutes out
of two, boom, that's theproductivity gains. And thirty
minutes in, I completely failed.I had to scrap everything that
(17:10):
LLM and agents wrote for me andstart from scratch. My problem
was I overprompted.
I thought I knew what I wantedan agent to do. There were three
steps (17:19):
copy this, write this, refactor this, and you're done.
It did it. It iterated for tenminutes. It was the Codeme
agentic developer that we have.
When I scrapped it and starteddoing it myself, I did half of
it, stopped, and realized thatthe other half was not needed.
(17:40):
It was stupid of me to ask. Thecorrect approach would have been
to iterate, do the first half,stop, rethink, and then decide
what to do next. But the agentwas given the instruction to go
all the way, so it went all theway. This is the other thing
with thousand instructions,right?
(18:01):
You don't want an agent to beasked to do something that you
think you know but you onlyreally will know as you iterate
through.
Daniel (18:12):
In these cases as well, so like I find your experience
with the balancing how youprompt it, how far the agent
goes. All of this is intuitionthat you're kind of learning.
One of the things that wasinteresting, we just had Kyle,
the CEO of GitHub on, and wewere talking about agents and
(18:34):
coding assistants. One of histhoughts was also around the
orchestration after you havegenerated some code. It's one
thing to create a project,create something new, but most
of software development happenspast that point.
And I'm curious as someone whois really trialing these tools
(18:55):
day in and day out as your dailydriver and utilizing these
things, I think that's onpeople's mind is, Oh, cool. I
can go into this tool, generatea new project that maybe
whatever it is, you always seethe demo of creating a new video
game or whatever the thing is.But ultimately, I have a code
(19:19):
base that is very massive,right? I'm maintaining it over
time. Most of the work is moreon that operational side.
So in your experience with thisset of tooling, what has been
your learning? Any insightsthere? Any thoughts on kind of
where that side of things isheading? Especially for you're
(19:40):
dealing with, I'm sure, realworld use cases with your
customers who have large codebases, right?
Pavel (19:47):
Well, that's great. I'm so glad that you asked because
what I do is actually thatlatter aspect. I have a mono
repo of 20 different things init that could have been separate
repos of their own. I have alarge code base that I work
with. And I actually saw our owndeveloper agent occasionally
(20:08):
choke because it attempts toread too much, and it just
chokes on tokens and limits andthings that it can do per minute
or per hour or something.
So that's one thing. But what Ifind myself doing with Cursor,
for example, I actually pinpointit very actively, very often,
cause I wanted to work withthese files when it's something
specific. I'll just point thefiles at it, and I'm gonna ask
(20:31):
I'm gonna prompt it in contextof these three or four files,
and that limits how much it'sgonna go out. But really, back
to your question, to me, it'snot about code bases that much.
I don't think it's going to bewell, maybe if I do something
greenfield and funny, I'm goingto write it, I'm going to run
it, and if it works, it's all Ineed.
It's correct, it works, great.Today, and it's still a mental
(20:54):
shift, it's still early, I'mstill looking and thinking of
the code base that I write withmy agents as code base that will
be supported by other people,likely with agents, but people
still. So correct by itself isnot good enough. I want it to be
aesthetically the same, I wantit to follow the same patterns,
(21:15):
I want it to make sense for myother developers who will come
in after me. I want it to be asif it's the code that I have
written, or at least more orless that I have written.
That slows me down a little bit,clearly, I'm sure. But the other
thing is I am the bottleneck. Anagent will take minutes, small
digit, like single digitminutes, if not less, to spit
(21:39):
out whatever it spits out.Oftentimes in code bases, it's
not a single file. It's edits inmultiple places.
Then I have to come in and readit. Here's the difference. When
I write myself, my brain has atimeline. I was thinking as I
was typing, as I was thinking, Iwas I know how I arrived at what
I have arrived at. I may decidethat it's bull, you know, scrap,
(22:03):
let's start over.
That happens, we're alldevelopers. But I know how I
arrived at where I am. When Ilook at what Agent produced for
me, I have no idea how itarrived at where I am. I need to
reverse engineer why, what didit do? It takes time.
I tried recording it and I can'tbecause I can't speak as I think
(22:24):
at the same time. This is thebottleneck, literally. Is the
bottleneck. The other thing is,when I was doing that video with
a timer, I expected certainoutcomes, but I also knew that
if it works, I'm going to saythis at the end. I'm going to
say, Guys, look, it took metwenty minutes, let's say thirty
minutes out of an hour, so it's2x, right?
(22:46):
Literally 2x productivityimprovement. Amazing, isn't it?
But here's the thing. Within thethirty minutes that I've spent,
the percentage of time I spentcritically thinking was much
higher than normal. Thepercentage of time I spent doing
boilerplate is much lowerbecause the agents did I really
critically thought about what toask, how to prompt, and then
(23:08):
analyzing what it did, thinkingwhat to do next.
Do I edit? Do I reprompt? Can Isustain the same higher
percentage of critical thinkingfor the full day to get to X in
a day? Probably I can't. Sowhat's probably going to happen,
I'm going to get to X, but I'mgoing to use the time in between
as agent work to do somethingelse.
(23:30):
My day will likely get brokendown into more smaller sections.
My overall daily productivity islikely to increase. I'm likely
to do more things in parallel.Maybe I'll do some research.
Maybe I'll answer more emails,right?
But it's going to be morechaotic, also likely more
taxing. I don't think we'velearned don't think we've had
(23:52):
enough experience yet. I don'tthink many people talk about
this yet. People talk aboutthis, Oh my god, look what I've
built with agents. I wonder howthey're going to talk about how
they've worked for six monthswith agents and how six months
that they've done with agents isbetter than six months without
and how they feel at the end ofthe day.
(24:15):
And think about in the zone.Will, I hope, as engineers, like
to be like, you know, disconnectall emails, whatever, get the
music on, IDE in front of you,you're in it for, like, two
hours. With agents, you justcan't. You prompt an agent, it
goes off doing something. Whatdo you do?
Do you pull up your phone andthen your productivity increases
(24:37):
one way, your screen timeincreases the other way? It's
not a good idea. What can youdo? What do you do in this
minute and a half or three? Andyou don't know how long.
Well, you can see the outcomescoming up, but the agents are
still spinning, it's stillspinning. I'm sorry, it's a long
answer to your question, butthat's what I'm thinking about
constantly, and that's what Idon't yet have answers for. But
(25:02):
I really hope to eventually,through experiments and
recording and thinking, arriveat least what it means for me,
because I cannot even tell youwhat it means for me yet.
Daniel (25:10):
Yeah. I mean, I experienced this yesterday too
because I'm preparing variousthings for investors, updating
some competitive analysis andthat sort of thing. And I just
when you have whatever it is, Ithink it was 116 companies and
(25:32):
I'm like, oh, I'm going toupdate all of these things for
all of these companies.Obviously, I'm going to use an
AI agent to do this. This is notsomething I want to do manually,
is put in all of these thingsand search websites.
I did that, but to your point, Icould figure out how to do a
piece of that and get itrunning. And then I see it
(25:55):
running and I realize that thiswill take however long it is,
right? Ten minutes or whateverthe timeframe is. And then you
context switch out of that tosomething else, which for me I
think was email or whatever. I'mlike, Oh, this is going to run.
I'm going to go answer someemails or something like that,
(26:15):
which in one way was productive,but then I had to context switch
back. They're like, Oh, why didI output all these things? Or it
happened to be that I wasn'twatching the output. In one case
when I ran it, I was like, Oh,well, I really should have had
it output this column or thisfield, but I didn't think of
(26:36):
that before. And I wasn'tlooking because I turned away
from the agent back to my email.
Yeah, I think this is a reallyinteresting set of problems that
is more of like a new yeah. It'sa new way of working that hasn't
been parsed out yet. Right?
Pavel (26:52):
And I tried not to do it. Like, tried, but then you sit
idle. Like, you'll literally sitidle. It's like and it it it
doesn't feel good. It feels itfeels like, oh my god.
Why am I not doing anything?
Daniel (27:02):
Yeah. It's it's an interesting dynamic. That's for
sure. And I've definitely seenpeople that show having multiple
agents working on differentprojects at the same time. That,
when I see someone with twoscreens and things popping up
all the place, there's no way Icould, in my brain, monitor all
(27:24):
of that that's going on.
Pavel (27:25):
It must be very taxing first, and second, half of those
merge requests, pull requestsfrom the agents will be, let's
say, subpar. Frustration, andyou will rise to it. You will
think, Man, I would have done italready myself much better. What
is this? Emotionally, it is avery different way of working
(27:46):
emotionally.
Daniel (27:47):
Yes. I
Pavel (27:52):
keep thinking. I can forget. I advise people also to
think, not just think aboutproductivity gains. Not just
think about delegating to agentsand enjoying the results. Think
about how it changes the dynamicof your day and how you think
about it afterwards, right?
Daniel (28:09):
Yeah. Yeah. That's interesting. So I know we're
circling kind of way back. Wasan interesting discussion, but I
do wanna make sure people cankind of find some of what you're
doing with Dial.
You mentioned kind of the opensource piece of this. What sort
of needed from the userperspective to kind of spin this
up and start testing it? And anyfor those that are out there
(28:33):
that are interested in tryingsome things with the project,
what would you kind of tell themas starting point and like, what
the process is like to kinda geta system like this up and
running?
Pavel (28:45):
I actually, I'm not sure I can tell, for dial specific.
Have nobody is running localdials. It's not something you
run locally. Gotcha. It'ssomething that you run sort of
centrally in an organization ofsize can be different, you
expose it to your people throughlike a URL that they all can go
(29:09):
to and use, use AI through dialand do things through dial.
Daniel (29:15):
Interesting.
Pavel (29:15):
One of the apps we built, as an example, earlier, it was
last year, Talk to Your Data. Ifyou look at analytics like
Snowflakes over the world, theyall have something like this
today, like semantic layer,which you work on. Then through
semantic layer, throughprompting and through some query
conversions and connectors todata warehouses and data lakes,
(29:39):
you get yourself a chat withyour data, like analytical
reports, graphs, tables. So webuilt that. That was built into
Dial.
So you go to Dial, and thenagain, imagine Imagine ChatGPT
that allows you to choose whatmodel you talk to, right? Not
just OpenAI models, but all ofthe other models that exist, as
well as applications. So go tothis ChargeDepT, which is in our
(30:03):
case, Dial, you select this dataheart AI, we call it, which is
our Talk To Data, and you starttalking to it. This is still
your Dial experience, but you'rereally talking to an app that
then talks to semantic layer,then builds queries based on
your questions, runs them, getsdata back, visualizes it in
(30:24):
because Dial has all thesevisualizations, capabilities
that explain how it's not justtext coming back, builds your
charts and you can interact withit. But again, you don't run
Dial locally.
If you want to explore what itis, I hope, I expect that if you
go to, I think it'srailepam.com.
Daniel (30:42):
EPM rail. Yeah.
Pavel (30:44):
Oh, EPMrail.com. Thank you. And you're going to read
about what it is and you'regoing to find all the links to
hopefully documentation, how toBut also, most companies who we
work with, they want more thanjust, Hey, how do we install it?
And now we want to build withit. And that's where we come in
(31:07):
with professional services.
We can build them things fortheir dial so that they can do
the AI that matters to them intheir context, with their data,
with their workflows, with theirrestrictions on things they can
and cannot do and yada yadayada.
Daniel (31:25):
Yeah. And I'm wondering for this kind of if you think
about this zoo of underlyingapplications or assistance, I'm
wondering, because you'veobviously been working in this
area for some time, do you haveany insights or learning around
(31:46):
easy wins for underlyingfunctions or agents that can be
tied into this orchestrationlayer, or maybe more challenging
ones, things that you've learnedover time developing and working
with these things in terms ofthings that you could highlight
as easy types of wins and thingsthat I mean, you mentioned the
(32:10):
workflow stuff around some ofwhat isn't yet kind of figured
out, but more on theorchestration layer and the
function calling. What are someareas of challenge or things
that might not be figured outyet that you think are
interesting to explore in thefuture?
Pavel (32:27):
Let me think. My first thought was to So you're asking
about connecting tools andfunctions to an LLM and which of
the functions or what type ofconnectivity is easier?
Daniel (32:40):
Yeah, yeah. Is there anything that's out of scope or
more of a challenge currently,or is it fair game for kind of I
guess it's whatever you canbuild in that function in the
assistant. What limitations arethere or challenges in that mode
of development of developingthese underlying functions or
tools?
Pavel (32:59):
I see. It's kind of a twofold answer. If you take the
technicality aspect, like how doI build a tool that does x, the
complexity is really in x. Ifyou want to go and query a
database, how hard is that?Well, not hard.
(33:20):
Right? I mean, connectivity tothe database, if you have a
query, you run it, you getresults back. So it's not hard
to do the technicality ofquerying a database. Making it
useful and making the resultuseful in context of users'
prompt and conversation is a lotmore challenging. I had this I'm
(33:40):
running a service.
It actually has a public webpagecalled api.epim.com. It's our
own You will not really go pastthe front page, but you'll
understand what it is. It's acollection of APIs that we
built, my team has built, thatexposes a lot of data. Remember
I said EPM runs on internalsoftware? So all of those
applications, they stream theirdata and their events out into a
(34:03):
global data hub.
Think big, big, big Kafkacluster. But that's Kafka, so
you can read data out of it as aKafka consumer, but if you want
to have more modern API search,lookup, this, that, so we have
an API service, all of the data.Somebody came to me today and
said, Hey, have you heard ofMCP? I'm like, Yes, of course I
(34:24):
have. Why don't you guys buildMCP for api..com?
My answer is it is easy tobuild. Api. Epim Com speaks
RSQL. I can build a server thatwill take your query, create
RSQL, LLM will be able to dothat easily, run it, give back
the data. But I said it's notgoing to be useful because this
(34:46):
is single dataset APIs.
Your questions are likelyanalytical. You likely want to
ask something that expects me todo summary by month, this, this,
this, give you a Which, that's avery different question. You ask
me about MCP to an API, easy todo. Make it useful for your
(35:06):
actual use case, much harder todo. I likely need to do a lot
more than just connectivity oftool to an LLM.
I need to understand what you'reasking, figure out the
orchestration that is required,maybe custom apps, maybe
something else. Then you starthitting authentication, legacy
(35:27):
apps, all the other roadblocks.And in a way, the Talk to Your
Data is an amazing prototypethat we built, and I have a
video about this. But we sort ofstopped because we clearly
sensed how steep the curve is toget it to actual Because what we
wanted to do, what we envisionedwe could do, was analytics
(35:49):
democratized. So you don't haveto go to an analytical team, ask
them to build you a new Power BIreport, and them spending a week
doing so.
You can just come into Dial andsay, Hey, show me this, this,
this, and this. And yes, wetechnically can do it. But to be
able to do this for all kinds ofquestions you can ask about our
data, that's a much harder thingto do.
Daniel (36:12):
Yeah. To your point, underlying systems might have
limitations. Think in analyticsrelated use cases that we've
encountered with our customers,Often I'll just ask the question
around, Hey, if you gave thisdatabase schema or whatever it
(36:34):
is to a reasonably educatedcollege intern or whatever that
is, and you ask what columnswould be relevant to query based
on this natural language query,you can pretty easily tease out,
Well, I look at all thesecolumns, I have field 157 and
(36:58):
customnewfield. There's no wayfor just someone off yet to know
anything about that. So it's notreally a limitation of what's
possible in terms of thetechnicality, like you said.
It's more of you're not alwaysset up for success in terms of
utility, like you mentioned.
Pavel (37:19):
And for data, that's where semantic layer comes in.
So if you have descriptions ofyour columns, of your tables
with business meaning Mhmm. Thenconnecting that semantic layer
with some data samples to LLMwill allow it to write the query
(37:39):
that you thought was impossibleto write because it is
impossible without the semanticlayer sort of can explain the
data that you have in businessterms in the language that the
questions will be asked of yourassistant. That's what allows us
to do this talk to a dataanalytics.
Daniel (37:59):
Yeah. Well, I know that we've talked about a lot of
things. I think you are probablyseeing a good number of use
cases across your clients atEPAM and also your own
experiments with Dial and otherthings. I'm wondering as you
kind of lay lay in bed at nightor whenever you're thinking
(38:20):
about the future of of AI or ormaybe it's all the time or maybe
it's maybe it's not, at night.But, yeah, as as you kind of see
what is to your point, justbringing it all the way back to
the beginning, you see what ispossible to do now, which even
six months, a year ago, whateverit was, you know, it not
(38:40):
possible.
What kind of is most excitingfor you or most interesting for
you to see how it plays out inthe next six to twelve months?
What is constantly on your mindof where things are going?
Sounds like you know, how wework with these tools is one of
those things. We already talkedabout that a little bit, but
(39:01):
what else is exciting for you orencouraging in terms of how you
see these things developing?
Pavel (39:09):
My answer may surprise you. When I think about it,
don't, you know, think oranticipate any new greatness to
come. I actually mostly worry.And I worry because I know that
my thinking is linear. Like mostof us, even though looking back,
we know that technology has beenevolving rather exponentially,
(39:34):
our ability to project into thefuture and think what's coming
next is linear.
So I am unlikely to properlyanticipate and get ready for and
then expect, right, and wait forwhat's to come. I am sure to be
surprised, and I guess aseverybody else, I'll be doing my
best to hold on, to not falloff. So I worry seeing how the
(40:00):
entry barriers rise. It's harderfor more junior people to get in
today. When I'm asked aboutskills I recommend that people
focus on as far as trying to bebetter prepared for the future,
I always answer it with the samethings.
I always say fundamentals andthen critical system thinking.
(40:23):
And fundamentals, you can readabout a lot, but you really
master them when you work withthem yourself, not when someone
else works with them for you.And not having them is likely
gonna constrain you from beingable to properly curate and
orchestrate all these powerfulAI agents. And when they get so
(40:44):
powerful that they don't needyou to curate and orchestrate
them, then what does it do toyou as an engineer? And maybe
that's not the right thinking,but this is what I think about
at night, like you asked, when Ithink about AI and what's
coming.
I am excited as an engineer. Ilike using all of this. I just
(41:05):
don't know how it's gonnareshape the industry and how
it's gonna change my work, youknow, in years to come.
Daniel (41:13):
Yeah. Well, I I think it's something, even in talking
through with you kind of some ofthe work that you and I have
been doing with agents and howthat really has triggered a lot
of questions in our own mind ofwhat is the proper way of of
working around this. And I thinkthere is going to be a you know,
that is a widespread, issue thatpeople are gonna have to
(41:36):
navigate. So, yeah, I think it'sI think it's very valid. And
we'll we will be interested tosee how it develops, we'd love
to have you back on the show toto have your learnings again in,
in six or twelve months of ofhow it's shaking out, shaking
out for you.
Really appreciate you joining.It's been a great conversation.
Pavel (41:56):
Thank you very much. It's been a pleasure.
Jerod (42:04):
Alright. That is our show for this week. If you haven't
checked out our Changelognewsletter, head to
changelog.com/news. There you'llfind 29 reasons. Yes.
29 reasons why you shouldsubscribe. I'll tell you reason
number 17. You might actuallystart looking forward to
Mondays.
Pavel (42:25):
Sounds like somebody's got a case of the Mondays.
Jerod (42:28):
28 more reasons are waiting for you at
changelog.com/news. Thanks againto our partners at fly.io, to
Brakemaster Cylinder for theBeats, and to you for listening.
That is all for now, but we'lltalk to you again next time.
Welcome to Practical AI, the podcast that makes
artificial intelligencepractical, productive, and
accessible to all. If you likethis show, you will love the
changelog. It's news on Mondays,deep technical interviews on
Wednesdays, and on Fridays, anawesome talk show for your
weekend enjoyment. Find us bysearching for the changelog
(00:24):
wherever you get your podcasts.Thanks to our partners at
fly.io.
Launch your AI apps in fiveminutes or less. Learn how
fly.io.
Daniel (00:44):
Welcome to another episode of the Practical AI
Podcast. This is DanielWitenack. I'm CEO of
PredictionGuard, and I'm reallyexcited today to dig a little
bit more into GenAIorchestration agents, coding
assistants, all of those thingswith my guest, Pavel Veller, who
(01:06):
is Chief Technologist at EPAMSystems. Welcome, Pavel. Great
to have you here.
Pavel (01:10):
Thank you. Hello. Hello.
Daniel (01:12):
Yeah, yeah. Well, I mean, there's a lot of topics
even before we kicked off theshow. We were chatting in the
background about some reallyinteresting things. I'm
wondering if you could just kindof level set us of people may or
may not have heard of EPAM. Ithink one of the things that I
saw that you all were working onwas this GenAI orchestration
(01:34):
platform dial.
Maybe before we get into some ofthe specifics about that and
other things that you'reinterested in, maybe just give
us a background of what EPAM is.I know you mentioned even in our
discussions that some of whatyou're doing right now maybe
wouldn't have even been possiblea couple of years ago, and so
(01:54):
things are developing rapidly.Just level set the kind of
background of this area whereyou're working.
Pavel (02:01):
Sure. Yeah, yeah. EPAM is a professional services
organization. We're global.We're in 50 something countries.
50,000 people globally work withclients. We have been for, I
think, thirty two years to date.We do a lot of different things,
as you can imagine. What I wasmentioning about doing things
(02:22):
that would not be possible isdoing things with GenAI today.
We do a lot of work for our ownclients.
We also do work for ourselves,applying the same technology,
because EPAM historically as acompany has been running on
software that we ourselvesbuilt. The philosophy has always
been that things that do notdifferentiate you, like an
(02:44):
accounting software or like CRM,you would go and buy off the
shelf. Things that differentiateyou, how we actually work, how
we operate, how we executeprojects, how we hire people,
how we create teams, how wedeploy teams. All of that
software has always been our ownsince as early as late '90s, and
(03:06):
we keep iterating on thatsoftware for ourselves. That
software today is very much AIfirst, and a lot of things we
do, we do with AI and really dobecause AI in its current form
exists.
Daniel (03:17):
Interesting. Yeah. And how, I guess, does I think when
we initially kind of wereprompted to reach out to you,
part of it was around thisorchestration platform. So talk
a little bit maybe generally,not necessarily about the
platform per se, although we'llget into that, but just Gen AI
(03:39):
orchestration generally. Youtalked about some of these
things that are becomingpossible.
Where does orchestration fit inthat, and what do you mean by
orchestration?
Pavel (03:49):
You probably think of Dial. You can Google it. We do a
lot of applied innovation ingeneral as a company, and this
is one of the good examples ofapplied innovation to AI. The
best way to think of Dial wouldbe you guys all know ChatGPT.
Right?
ChatGPT isn't an LLM. It's anapplication that connects to an
(04:13):
LLM and gives you certainfunctionalities. It can be as
simple as just chatting andasking questions. It can be a
little more complex, uploadingdocuments and speaking to them,
like Talk to my documents. Itcan be even more complex when
you start connecting your owntools to it.
We see our clients not only dothis, but also want something
(04:33):
like this for their own businessprocesses. This orchestration
engine becomes, How do I make itso that I don't have 20
different teams doing the samesimilar things over and over
again in their own silos? How doI connect my teams and their AIs
and their thoughts and resultsinto a consolidated ecosystem?
(04:56):
It's likely because of GenAI andbecause of what we can do with
conversation and text becomessort of conversation first. You
can think of conversation firstapplication mashups almost.
Right? Like you talk, express aproblem, and what comes back is
not just the answer. Maybe whatcomes back is UI elements,
(05:17):
buttons you can click, forms youcan fill out, things you can do,
as well as things that are donefor you by agents automatically.
Dial, in that sense, is Well, bythe way, it is open source. You
guys can also go look, download,and play with it.
But it is a ChatGPT likeconversational application that
(05:38):
has many capabilities that gobeyond. We have dial apps. They
predate MCP, but the idea isthat you so dial itself has a
contract, an API that youimplement. You basically come
back with a streaming, API thatcan receive a user prompt.
Whatever you do, you do, and youcome back to Dial with not just
(06:01):
text.
It's a much more powerfulpayload with UI elements,
interactive elements, and thingsthat Dial will display for me,
the user, to continue myinteraction. And Dial becomes
this sort of center mass of howyour company can build,
implement, integrate AI intothis single point of entry. And
(06:22):
then Dial goes Well, from dayone, Dial was a load balancing
model agnostic proxy. Right? Soevery model, every deployment
has limits, you know, tokens perminute, tokens per day,
whatever, requests per minute.
You'll likely if you're a largeorganization with large
different workflows, your AIappetite will go well beyond a
(06:44):
single model deployment. You'dlike to load balance across
multiple, and then you'd like totry different models, ideally
with the same API for you, theconsumer. So Dial started like
that. It started like loadbalancing, model agnostic proxy,
single point of entry. We canlog everything that is prompted
in the organization.
We can do analysis on thatseparately because that's very
(07:07):
helpful to know what kind ofproblems your teams are trying
to solve. Then it evolved intothis application hosting
ecosystem. Now it's evolvingclearly towards what MCP can
bring in because now you canconnect a lot more things to it
through MCP. So I think it'srunning at 20 something clients
by now.
Daniel (07:27):
So just a couple of follow-up questions. It's been
in the news a lot, but just sopeople understand if maybe they
haven't seen it, what are youreferring to with MCP and how
that relates to some of this APIinterface that you're enabling?
Pavel (07:44):
Well, the easiest is to Google it. You can Google it,
you're going to find it, it's onClaude. Let me tell you how I
think about this because notwhat it actually That's helpful.
Yeah. Yeah.
I think about it in a verysimple term. So MCP allows to
connect the existing softwareworld to LLMs. In a way, think
(08:05):
like I don't want to hype it toomuch because it's not yet the
global standard or anything.It's very early, early, early
days. It's been months, right?
But let's say HTML and browsersand HTTP, they're enabled to
connect us. People are too softfor all over the world. MCP does
(08:26):
that but for LLMs. Today, if Itoday want to be able to prompt
my application that is in frontof an LLM to do things with
additional tools Let's say Iwanted to be able to search file
system based on what I promptedand find a file and something in
that file, right? So myapplication needs to be able to
(08:48):
do that.
My option is what? I can writethat function. I can then tell
my LLM, Hey, here's thisfunction you can call if you
want to. Call it. I'm going tocall it for you.
Great, that's one function. Whatif I need to do something else?
I want to go talk to my CRMsystem and get something out of
there. I'm going to write thatfunction. If I'm going to write
all the functions I can thinkof, it's going to take me years,
(09:11):
probably hundreds of years.
Instead, what I can do today,can say, Hey, my LLM
application, can you talk aprotocol? Because there's a
protocol called MCP. I'm goingto bring you MCP servers that
other people have built. For myCRM system, for my file system,
for my CLI, there are MCPservers for everything. IntelliJ
exposes itself as an MCP serverto do things that IDE can do.
(09:35):
Now you can orchestrate thosethings through LLM. So you
connect all those MCP serversthrough an MCP client, this
application in front of LLM, toLLM, expose the tools to LLM.
LLM can now ask the client tocall a tool, and through this
MCP protocol, the client callsthe server, the server does the
function that has been writtenin that server, and boom, LLM
(09:58):
gets results. It's thisconnective tissue that did not
exist three months ago. Threemonths ago, everybody was
writing their own.
And right now, everybody, as faras I can tell, writing MCP
servers, and those who talk toLLMs, they consume MCP servers.
Daniel (10:13):
Yeah. And maybe just give so I like the example that
you gave of searching filesystems. Just to kind of expand
people's understanding of someof possibilities, what are some
of the things that you've seenmaybe implemented in Dial as
things that are beingorchestrated? In general terms,
(10:34):
what are kind of some of thesethings?
Pavel (10:36):
Let me give you a higher level and much more sort of
fruitful example, okay? Yeah. Wehave our own agentic developer.
It's called AIRun Codeme,because AIRun has multiple
different agentic systems.Codeme is specifically coding
(10:58):
oriented.
We have others oriented at otherparts of SDLC workflow. By the
way, you guys can go to SWEbench look at verified list. I
believe Codeme as of now takesfifth, it's number five on the
list of all the agents whocompete for solving open source
defects and stuff. Codeme as anagentic system has many
(11:21):
different assistants in it. Dialas a generic front door, as a
ChatGPT, would like to be ableto run those assistants for you
as you talk to Dial.
Until MCP, it really couldn'tother than, Hey, Codeme.
Implement an API for all of yourassistants. Let me learn to call
(11:43):
all of your APIs. Now the storyis, Hey, Codeme. Give me an MCP
server for you, which is whatthey have done.
Dial as an MCP client can nowconnect to all Codeme features,
all the assistants, expose themas tools to an LLM, and
orchestrate them for me. So Icome into the chat, I ask for
(12:07):
something, that somethingincludes reading a code base and
making architecture sketches orproposals or evaluation, right?
LLM will ask Codeme assistanceto go and read that code base
because there is a feature inCodeme that does it, and Dial
needs to only orchestrate butdoesn't need to rebuild or built
(12:31):
from scratch. That's the idea,so this is the example.
Daniel (12:34):
Yeah. Could you talk a little bit I'm asking selfish
questions because sometimes Iget these asked of me and I'm
always curious how people answerthis. One of the questions that
I get asked a lot in respect tothis topic is, okay, I have tool
or function or assistant one,and then I have assistant two,
(12:57):
and then I kind of have a few.Right? And it's fairly easy to
route between them becausethey're very distinct, right?
But now if you imagine, okay,well now I could call one of a
thousand assistants or functionsor something, or later on
10,000, right? How does the sortof scaling and routing kind of
(13:17):
actually how has that affectedas you kind of expand the space
of things that you can do?
Pavel (13:25):
So that I think and again, I can't know, and I don't
know, but I think that is stillthe secret sauce. In a way, that
is still why there is all of allof these coding agents in SWE
bench. All of them work with,let's say, Cloud Sonnet 3.5 or
Cloud Sonnet 3.7 or GPT fourpoint zero. LLM is the same, and
(13:49):
yet results are clearlydifferent. Some score 10 points
higher than the other.
You go to cursor, IDE cursor,you ask it something, it does
something. You switch the modeto max. They've introduced very
recently. Cursor on Sonnet 3.7and now on Gemini two point
zero, I think, they have a maxmode, which is pay per use
(14:12):
versus their normal monthlyplans, because max will do more
iterations, will spend moretokens, will be more expensive,
will likely run through morecomplex orchestrations of
prompts and tools and whatnot togive you better results. How you
build the pyramid of choices foryour LLM, how you Yeah, you will
(14:35):
not ask LLM, you will not giveit a thousand tools.
If you as a human look athousand options and you lose
yourself, a hundred options init, again, I don't know. I
expect LLM to have the same sortof oops, overwhelmed effect. You
don't want to give it a thousandtools. Want to give it groups.
You want to say, Hey, pick agroup.
(14:56):
So you want to do this basicallylike a pyramid, like a tree. But
how you build it and how youprompt it and how you do this,
now that's still on you. This isthe application that connects
the MCPs, the tools that ititself has, the prompt that the
user has given, the systeminstructions, and building some
(15:18):
of the chain of thought LLM canbuild. And this is going to be a
very interesting balance. Whatdo you ask LLM to build?
How much of this sequencing ofsteps will be on you in your
hands versus how much you'regoing to delegate to LLM and ask
LLM to come up with a sequenceof steps. And from what I've
seen over the last year, you'rebetter off delegating more to
(15:42):
LLMs because they get better atit. So the more you control the
sequence yourself, the more sortof inflexible it becomes. You're
better off delegating to a LM,but you don't expect it to just
figure out from one prompt.Daniel, I can give you that
example that I gave in thebeginning, if you want, about
the failure.
(16:02):
Yeah, go for it. I use AI. Ibuilt with AI, but I also use AI
as a developer. I'm on cursor asmy primary ID these days. I use
the AIRunCodemey that Imentioned.
I play around with other thingsas they come up, like Cloud Code
and things, but I also recordwhat I do. Little snippets,
(16:24):
five, ten minutes videos for myengineering audience at EPAM for
the guys to just look what it isthat I'm doing, learn from how I
do it, try to think the sameway, try to replicate, get on
board with using AI. So I setout to do a task. I wanted to,
on record, get a productivityincrease with a timer. My plan
(16:47):
was I'm going to estimate howlong it's going take me,
announce, let's say two hours,do it with an agent, and I
always pause my video when theagent is thinking because that's
a boring step.
But the timer's going to getticking. And at the end, I'm
going to arrive at, let's say,an hour, maybe forty minutes out
of two, boom, that's theproductivity gains. And thirty
minutes in, I completely failed.I had to scrap everything that
(17:10):
LLM and agents wrote for me andstart from scratch. My problem
was I overprompted.
I thought I knew what I wantedan agent to do. There were three
steps (17:19):
copy this, write this, refactor this, and you're done.
It did it. It iterated for tenminutes. It was the Codeme
agentic developer that we have.
When I scrapped it and starteddoing it myself, I did half of
it, stopped, and realized thatthe other half was not needed.
(17:40):
It was stupid of me to ask. Thecorrect approach would have been
to iterate, do the first half,stop, rethink, and then decide
what to do next. But the agentwas given the instruction to go
all the way, so it went all theway. This is the other thing
with thousand instructions,right?
(18:01):
You don't want an agent to beasked to do something that you
think you know but you onlyreally will know as you iterate
through.
Daniel (18:12):
In these cases as well, so like I find your experience
with the balancing how youprompt it, how far the agent
goes. All of this is intuitionthat you're kind of learning.
One of the things that wasinteresting, we just had Kyle,
the CEO of GitHub on, and wewere talking about agents and
(18:34):
coding assistants. One of histhoughts was also around the
orchestration after you havegenerated some code. It's one
thing to create a project,create something new, but most
of software development happenspast that point.
And I'm curious as someone whois really trialing these tools
(18:55):
day in and day out as your dailydriver and utilizing these
things, I think that's onpeople's mind is, Oh, cool. I
can go into this tool, generatea new project that maybe
whatever it is, you always seethe demo of creating a new video
game or whatever the thing is.But ultimately, I have a code
(19:19):
base that is very massive,right? I'm maintaining it over
time. Most of the work is moreon that operational side.
So in your experience with thisset of tooling, what has been
your learning? Any insightsthere? Any thoughts on kind of
where that side of things isheading? Especially for you're
(19:40):
dealing with, I'm sure, realworld use cases with your
customers who have large codebases, right?
Pavel (19:47):
Well, that's great. I'm so glad that you asked because
what I do is actually thatlatter aspect. I have a mono
repo of 20 different things init that could have been separate
repos of their own. I have alarge code base that I work
with. And I actually saw our owndeveloper agent occasionally
(20:08):
choke because it attempts toread too much, and it just
chokes on tokens and limits andthings that it can do per minute
or per hour or something.
So that's one thing. But what Ifind myself doing with Cursor,
for example, I actually pinpointit very actively, very often,
cause I wanted to work withthese files when it's something
specific. I'll just point thefiles at it, and I'm gonna ask
(20:31):
I'm gonna prompt it in contextof these three or four files,
and that limits how much it'sgonna go out. But really, back
to your question, to me, it'snot about code bases that much.
I don't think it's going to bewell, maybe if I do something
greenfield and funny, I'm goingto write it, I'm going to run
it, and if it works, it's all Ineed.
It's correct, it works, great.Today, and it's still a mental
(20:54):
shift, it's still early, I'mstill looking and thinking of
the code base that I write withmy agents as code base that will
be supported by other people,likely with agents, but people
still. So correct by itself isnot good enough. I want it to be
aesthetically the same, I wantit to follow the same patterns,
(21:15):
I want it to make sense for myother developers who will come
in after me. I want it to be asif it's the code that I have
written, or at least more orless that I have written.
That slows me down a little bit,clearly, I'm sure. But the other
thing is I am the bottleneck. Anagent will take minutes, small
digit, like single digitminutes, if not less, to spit
(21:39):
out whatever it spits out.Oftentimes in code bases, it's
not a single file. It's edits inmultiple places.
Then I have to come in and readit. Here's the difference. When
I write myself, my brain has atimeline. I was thinking as I
was typing, as I was thinking, Iwas I know how I arrived at what
I have arrived at. I may decidethat it's bull, you know, scrap,
(22:03):
let's start over.
That happens, we're alldevelopers. But I know how I
arrived at where I am. When Ilook at what Agent produced for
me, I have no idea how itarrived at where I am. I need to
reverse engineer why, what didit do? It takes time.
I tried recording it and I can'tbecause I can't speak as I think
(22:24):
at the same time. This is thebottleneck, literally. Is the
bottleneck. The other thing is,when I was doing that video with
a timer, I expected certainoutcomes, but I also knew that
if it works, I'm going to saythis at the end. I'm going to
say, Guys, look, it took metwenty minutes, let's say thirty
minutes out of an hour, so it's2x, right?
(22:46):
Literally 2x productivityimprovement. Amazing, isn't it?
But here's the thing. Within thethirty minutes that I've spent,
the percentage of time I spentcritically thinking was much
higher than normal. Thepercentage of time I spent doing
boilerplate is much lowerbecause the agents did I really
critically thought about what toask, how to prompt, and then
(23:08):
analyzing what it did, thinkingwhat to do next.
Do I edit? Do I reprompt? Can Isustain the same higher
percentage of critical thinkingfor the full day to get to X in
a day? Probably I can't. Sowhat's probably going to happen,
I'm going to get to X, but I'mgoing to use the time in between
as agent work to do somethingelse.
(23:30):
My day will likely get brokendown into more smaller sections.
My overall daily productivity islikely to increase. I'm likely
to do more things in parallel.Maybe I'll do some research.
Maybe I'll answer more emails,right?
But it's going to be morechaotic, also likely more
taxing. I don't think we'velearned don't think we've had
(23:52):
enough experience yet. I don'tthink many people talk about
this yet. People talk aboutthis, Oh my god, look what I've
built with agents. I wonder howthey're going to talk about how
they've worked for six monthswith agents and how six months
that they've done with agents isbetter than six months without
and how they feel at the end ofthe day.
(24:15):
And think about in the zone.Will, I hope, as engineers, like
to be like, you know, disconnectall emails, whatever, get the
music on, IDE in front of you,you're in it for, like, two
hours. With agents, you justcan't. You prompt an agent, it
goes off doing something. Whatdo you do?
Do you pull up your phone andthen your productivity increases
(24:37):
one way, your screen timeincreases the other way? It's
not a good idea. What can youdo? What do you do in this
minute and a half or three? Andyou don't know how long.
Well, you can see the outcomescoming up, but the agents are
still spinning, it's stillspinning. I'm sorry, it's a long
answer to your question, butthat's what I'm thinking about
constantly, and that's what Idon't yet have answers for. But
(25:02):
I really hope to eventually,through experiments and
recording and thinking, arriveat least what it means for me,
because I cannot even tell youwhat it means for me yet.
Daniel (25:10):
Yeah. I mean, I experienced this yesterday too
because I'm preparing variousthings for investors, updating
some competitive analysis andthat sort of thing. And I just
when you have whatever it is, Ithink it was 116 companies and
(25:32):
I'm like, oh, I'm going toupdate all of these things for
all of these companies.Obviously, I'm going to use an
AI agent to do this. This is notsomething I want to do manually,
is put in all of these thingsand search websites.
I did that, but to your point, Icould figure out how to do a
piece of that and get itrunning. And then I see it
(25:55):
running and I realize that thiswill take however long it is,
right? Ten minutes or whateverthe timeframe is. And then you
context switch out of that tosomething else, which for me I
think was email or whatever. I'mlike, Oh, this is going to run.
I'm going to go answer someemails or something like that,
(26:15):
which in one way was productive,but then I had to context switch
back. They're like, Oh, why didI output all these things? Or it
happened to be that I wasn'twatching the output. In one case
when I ran it, I was like, Oh,well, I really should have had
it output this column or thisfield, but I didn't think of
(26:36):
that before. And I wasn'tlooking because I turned away
from the agent back to my email.
Yeah, I think this is a reallyinteresting set of problems that
is more of like a new yeah. It'sa new way of working that hasn't
been parsed out yet. Right?
Pavel (26:52):
And I tried not to do it. Like, tried, but then you sit
idle. Like, you'll literally sitidle. It's like and it it it
doesn't feel good. It feels itfeels like, oh my god.
Why am I not doing anything?
Daniel (27:02):
Yeah. It's it's an interesting dynamic. That's for
sure. And I've definitely seenpeople that show having multiple
agents working on differentprojects at the same time. That,
when I see someone with twoscreens and things popping up
all the place, there's no way Icould, in my brain, monitor all
(27:24):
of that that's going on.
Pavel (27:25):
It must be very taxing first, and second, half of those
merge requests, pull requestsfrom the agents will be, let's
say, subpar. Frustration, andyou will rise to it. You will
think, Man, I would have done italready myself much better. What
is this? Emotionally, it is avery different way of working
(27:46):
emotionally.
Daniel (27:47):
Yes. I
Pavel (27:52):
keep thinking. I can forget. I advise people also to
think, not just think aboutproductivity gains. Not just
think about delegating to agentsand enjoying the results. Think
about how it changes the dynamicof your day and how you think
about it afterwards, right?
Daniel (28:09):
Yeah. Yeah. That's interesting. So I know we're
circling kind of way back. Wasan interesting discussion, but I
do wanna make sure people cankind of find some of what you're
doing with Dial.
You mentioned kind of the opensource piece of this. What sort
of needed from the userperspective to kind of spin this
up and start testing it? And anyfor those that are out there
(28:33):
that are interested in tryingsome things with the project,
what would you kind of tell themas starting point and like, what
the process is like to kinda geta system like this up and
running?
Pavel (28:45):
I actually, I'm not sure I can tell, for dial specific.
Have nobody is running localdials. It's not something you
run locally. Gotcha. It'ssomething that you run sort of
centrally in an organization ofsize can be different, you
expose it to your people throughlike a URL that they all can go
(29:09):
to and use, use AI through dialand do things through dial.
Daniel (29:15):
Interesting.
Pavel (29:15):
One of the apps we built, as an example, earlier, it was
last year, Talk to Your Data. Ifyou look at analytics like
Snowflakes over the world, theyall have something like this
today, like semantic layer,which you work on. Then through
semantic layer, throughprompting and through some query
conversions and connectors todata warehouses and data lakes,
(29:39):
you get yourself a chat withyour data, like analytical
reports, graphs, tables. So webuilt that. That was built into
Dial.
So you go to Dial, and thenagain, imagine Imagine ChatGPT
that allows you to choose whatmodel you talk to, right? Not
just OpenAI models, but all ofthe other models that exist, as
well as applications. So go tothis ChargeDepT, which is in our
(30:03):
case, Dial, you select this dataheart AI, we call it, which is
our Talk To Data, and you starttalking to it. This is still
your Dial experience, but you'rereally talking to an app that
then talks to semantic layer,then builds queries based on
your questions, runs them, getsdata back, visualizes it in
(30:24):
because Dial has all thesevisualizations, capabilities
that explain how it's not justtext coming back, builds your
charts and you can interact withit. But again, you don't run
Dial locally.
If you want to explore what itis, I hope, I expect that if you
go to, I think it'srailepam.com.
Daniel (30:42):
EPM rail. Yeah.
Pavel (30:44):
Oh, EPMrail.com. Thank you. And you're going to read
about what it is and you'regoing to find all the links to
hopefully documentation, how toBut also, most companies who we
work with, they want more thanjust, Hey, how do we install it?
And now we want to build withit. And that's where we come in
(31:07):
with professional services.
We can build them things fortheir dial so that they can do
the AI that matters to them intheir context, with their data,
with their workflows, with theirrestrictions on things they can
and cannot do and yada yadayada.
Daniel (31:25):
Yeah. And I'm wondering for this kind of if you think
about this zoo of underlyingapplications or assistance, I'm
wondering, because you'veobviously been working in this
area for some time, do you haveany insights or learning around
(31:46):
easy wins for underlyingfunctions or agents that can be
tied into this orchestrationlayer, or maybe more challenging
ones, things that you've learnedover time developing and working
with these things in terms ofthings that you could highlight
as easy types of wins and thingsthat I mean, you mentioned the
(32:10):
workflow stuff around some ofwhat isn't yet kind of figured
out, but more on theorchestration layer and the
function calling. What are someareas of challenge or things
that might not be figured outyet that you think are
interesting to explore in thefuture?
Pavel (32:27):
Let me think. My first thought was to So you're asking
about connecting tools andfunctions to an LLM and which of
the functions or what type ofconnectivity is easier?
Daniel (32:40):
Yeah, yeah. Is there anything that's out of scope or
more of a challenge currently,or is it fair game for kind of I
guess it's whatever you canbuild in that function in the
assistant. What limitations arethere or challenges in that mode
of development of developingthese underlying functions or
tools?
Pavel (32:59):
I see. It's kind of a twofold answer. If you take the
technicality aspect, like how doI build a tool that does x, the
complexity is really in x. Ifyou want to go and query a
database, how hard is that?Well, not hard.
(33:20):
Right? I mean, connectivity tothe database, if you have a
query, you run it, you getresults back. So it's not hard
to do the technicality ofquerying a database. Making it
useful and making the resultuseful in context of users'
prompt and conversation is a lotmore challenging. I had this I'm
(33:40):
running a service.
It actually has a public webpagecalled api.epim.com. It's our
own You will not really go pastthe front page, but you'll
understand what it is. It's acollection of APIs that we
built, my team has built, thatexposes a lot of data. Remember
I said EPM runs on internalsoftware? So all of those
applications, they stream theirdata and their events out into a
(34:03):
global data hub.
Think big, big, big Kafkacluster. But that's Kafka, so
you can read data out of it as aKafka consumer, but if you want
to have more modern API search,lookup, this, that, so we have
an API service, all of the data.Somebody came to me today and
said, Hey, have you heard ofMCP? I'm like, Yes, of course I
(34:24):
have. Why don't you guys buildMCP for api..com?
My answer is it is easy tobuild. Api. Epim Com speaks
RSQL. I can build a server thatwill take your query, create
RSQL, LLM will be able to dothat easily, run it, give back
the data. But I said it's notgoing to be useful because this
(34:46):
is single dataset APIs.
Your questions are likelyanalytical. You likely want to
ask something that expects me todo summary by month, this, this,
this, give you a Which, that's avery different question. You ask
me about MCP to an API, easy todo. Make it useful for your
(35:06):
actual use case, much harder todo. I likely need to do a lot
more than just connectivity oftool to an LLM.
I need to understand what you'reasking, figure out the
orchestration that is required,maybe custom apps, maybe
something else. Then you starthitting authentication, legacy
(35:27):
apps, all the other roadblocks.And in a way, the Talk to Your
Data is an amazing prototypethat we built, and I have a
video about this. But we sort ofstopped because we clearly
sensed how steep the curve is toget it to actual Because what we
wanted to do, what we envisionedwe could do, was analytics
(35:49):
democratized. So you don't haveto go to an analytical team, ask
them to build you a new Power BIreport, and them spending a week
doing so.
You can just come into Dial andsay, Hey, show me this, this,
this, and this. And yes, wetechnically can do it. But to be
able to do this for all kinds ofquestions you can ask about our
data, that's a much harder thingto do.
Daniel (36:12):
Yeah. To your point, underlying systems might have
limitations. Think in analyticsrelated use cases that we've
encountered with our customers,Often I'll just ask the question
around, Hey, if you gave thisdatabase schema or whatever it
(36:34):
is to a reasonably educatedcollege intern or whatever that
is, and you ask what columnswould be relevant to query based
on this natural language query,you can pretty easily tease out,
Well, I look at all thesecolumns, I have field 157 and
(36:58):
customnewfield. There's no wayfor just someone off yet to know
anything about that. So it's notreally a limitation of what's
possible in terms of thetechnicality, like you said.
It's more of you're not alwaysset up for success in terms of
utility, like you mentioned.
Pavel (37:19):
And for data, that's where semantic layer comes in.
So if you have descriptions ofyour columns, of your tables
with business meaning Mhmm. Thenconnecting that semantic layer
with some data samples to LLMwill allow it to write the query
(37:39):
that you thought was impossibleto write because it is
impossible without the semanticlayer sort of can explain the
data that you have in businessterms in the language that the
questions will be asked of yourassistant. That's what allows us
to do this talk to a dataanalytics.
Daniel (37:59):
Yeah. Well, I know that we've talked about a lot of
things. I think you are probablyseeing a good number of use
cases across your clients atEPAM and also your own
experiments with Dial and otherthings. I'm wondering as you
kind of lay lay in bed at nightor whenever you're thinking
(38:20):
about the future of of AI or ormaybe it's all the time or maybe
it's maybe it's not, at night.But, yeah, as as you kind of see
what is to your point, justbringing it all the way back to
the beginning, you see what ispossible to do now, which even
six months, a year ago, whateverit was, you know, it not
(38:40):
possible.
What kind of is most excitingfor you or most interesting for
you to see how it plays out inthe next six to twelve months?
What is constantly on your mindof where things are going?
Sounds like you know, how wework with these tools is one of
those things. We already talkedabout that a little bit, but
(39:01):
what else is exciting for you orencouraging in terms of how you
see these things developing?
Pavel (39:09):
My answer may surprise you. When I think about it,
don't, you know, think oranticipate any new greatness to
come. I actually mostly worry.And I worry because I know that
my thinking is linear. Like mostof us, even though looking back,
we know that technology has beenevolving rather exponentially,
(39:34):
our ability to project into thefuture and think what's coming
next is linear.
So I am unlikely to properlyanticipate and get ready for and
then expect, right, and wait forwhat's to come. I am sure to be
surprised, and I guess aseverybody else, I'll be doing my
best to hold on, to not falloff. So I worry seeing how the
(40:00):
entry barriers rise. It's harderfor more junior people to get in
today. When I'm asked aboutskills I recommend that people
focus on as far as trying to bebetter prepared for the future,
I always answer it with the samethings.
I always say fundamentals andthen critical system thinking.
(40:23):
And fundamentals, you can readabout a lot, but you really
master them when you work withthem yourself, not when someone
else works with them for you.And not having them is likely
gonna constrain you from beingable to properly curate and
orchestrate all these powerfulAI agents. And when they get so
(40:44):
powerful that they don't needyou to curate and orchestrate
them, then what does it do toyou as an engineer? And maybe
that's not the right thinking,but this is what I think about
at night, like you asked, when Ithink about AI and what's
coming.
I am excited as an engineer. Ilike using all of this. I just
(41:05):
don't know how it's gonnareshape the industry and how
it's gonna change my work, youknow, in years to come.
Daniel (41:13):
Yeah. Well, I I think it's something, even in talking
through with you kind of some ofthe work that you and I have
been doing with agents and howthat really has triggered a lot
of questions in our own mind ofwhat is the proper way of of
working around this. And I thinkthere is going to be a you know,
that is a widespread, issue thatpeople are gonna have to
(41:36):
navigate. So, yeah, I think it'sI think it's very valid. And
we'll we will be interested tosee how it develops, we'd love
to have you back on the show toto have your learnings again in,
in six or twelve months of ofhow it's shaking out, shaking
out for you.
Really appreciate you joining.It's been a great conversation.
Pavel (41:56):
Thank you very much. It's been a pleasure.
Jerod (42:04):
Alright. That is our show for this week. If you haven't
checked out our Changelognewsletter, head to
changelog.com/news. There you'llfind 29 reasons. Yes.
29 reasons why you shouldsubscribe. I'll tell you reason
number 17. You might actuallystart looking forward to
Mondays.
Pavel (42:25):
Sounds like somebody's got a case of the Mondays.
Jerod (42:28):
28 more reasons are waiting for you at
changelog.com/news. Thanks againto our partners at fly.io, to
Brakemaster Cylinder for theBeats, and to you for listening.
That is all for now, but we'lltalk to you again next time.
Popular Podcasts
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Therapy Gecko
An unlicensed lizard psychologist travels the universe talking to strangers about absolutely nothing. TO CALL THE GECKO: follow me on https://www.twitch.tv/lyleforever to get a notification for when I am taking calls. I am usually live Mondays, Wednesdays, and Fridays but lately a lot of other times too. I am a gecko.