October 3, 2024 • 39 mins
Unlock the future of finance with Bin Ren, Founder & CEO of SigTech, as he reveals the transformative potential of AI in capital markets. Discover how AI is revolutionizing financial decision-making processes by enhancing productivity tools for professionals in investment management, trading, and risk management. Learn about the critical role of a robust data foundation in building AI-driven systems and the intricate stages of pre-training and post-training large language models. Bin shares practical examples to illustrate how AI can swiftly process and summarize complex information, potentially altering how financial decisions are made.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:06):
Welcome to Trading Tomorrow Navigating Trends in
Capital Markets the podcastwhere we deep dive into
technologies reshaping the worldof capital markets.
I'm your host, jim Jockle, aveteran of the finance industry
with a passion for thecomplexities of financial
technologies and market trends.
In each episode, we'll explorethe cutting-edge trends, tools
and strategies driving today'sfinancial landscapes and paving
(00:29):
the way for the future.
With the finance industry at apivotal point, influenced by
groundbreaking innovations, it'smore crucial than ever to
understand how thesetechnological advancements
interact with market dynamics.
In today's episode, we'reexploring one of the most
(00:57):
transformative forces in financeartificial intelligence.
Ai is reshaping capital markets, with buy-side and sell-side
firms adopting innovativetechnologies to gain an edge.
But with all this innovationcomes the need for a robust data
foundation.
Today, we're going to explorehow AI is not only boosting
productivity, but also changingthe roles of financial
(01:19):
professionals and what thefuture holds as AI becomes more
deeply integrated into thefinancial landscape.
To help us navigate this topic,we're joined by Bin Ren, founder
and CEO of SigTech, a companyat the forefront of AI
innovation in finance.
Before starting SigTech in 2019, bin was the chief investment
officer of the systemicinvestment group at Brevin
(01:39):
Howard, where he led thequantitative investment funds.
He began his career at Barclaysas an equity exotics trader.
Welcome, ben.
Thank you for having me today,jim, absolutely Pleasure.
So you know what.
Just to start off, why don't wetalk about SigTech?
Speaker 2 (01:53):
So SigTech we have been around since 2019, so over
five years.
We spun up from a hedge fundcalled Brevin Howard and we
really focus on building andshipping the best productivity
tools for people who work in thefront office of the capital
markets.
(02:13):
So our users tend to make veryimportant financial decisions,
such as investment management ortrading or risk management.
So in their daily life, thespeed and quality of the
decisions matter.
So that's why we want to buildthe best tools to help them.
Speaker 1 (02:34):
And what is the key role of AI in the platform?
Speaker 2 (02:39):
So we started five years ago.
So a big part of our product isto help people to do data-driven
analysis.
So in finance, it's very muchabout numbers, their time series
, but also textual data you'regetting from different sources
on a daily basis.
So AI has really helped us inthe last two years to lower the
(03:02):
hurdle in terms of how muchknowledge a user has to have in
terms of programming and dataanalysis and to actually do what
used to be the specialist jobof a data analyst.
So I think AI has made a hugedifference in the last two years
to be able to write veryhigh-quality code on behalf of a
(03:23):
user and be able to actuallyprocess a lot of the fairly
complicated so-called naturallanguage processing problems.
For example, Jay Powell justgave a speech at the Jackson
Hole and talking about making itactually very important and
saying central banks are notready to cut rates, and what
happened is that actually thespeech was published while he
(03:46):
started speaking and our usersare able to immediately, through
large language models, trainand fine-tune by us to be able
to ask questions like summarizea speech for me right now, focus
on potential for rate cut infive seconds, even way before
the speech is finished.
I think this is a sort of oneof the many interesting use
(04:11):
cases that AI has been able todeliver.
Speaker 1 (04:14):
I guess gone are the days of everybody huddling
around CNBC and the floor goingquiet.
That's amazing.
So you know the AI product isbacked by robust data and you
mentioned that you know.
Perhaps you can explain theimportance of having strong data
foundation, especially whenbuilding AI-driven systems.
Speaker 2 (04:35):
I think if we think about how the launch language
models work, I think there aremultiple stages of training.
Okay, there's the first stage.
Actually, actually it's calledthe pre-training.
Pre-training is where we have ahuge corpus of data we're
talking about trillions ofso-called tokens covering as
(04:57):
many different domains aspossible to train a very large
neural network to essentiallybecome a very competent and
knowledgeable generalist.
And then there's the secondstage, which is called
post-training is to align thislarge language model with human
objectives.
So when we interact with themodels, we want the model to
(05:24):
respond to us in a certain waythat aligns with our style of
conversation, of our way ofthinking, rather than just
predicting the next token, whichis what the pre-training is
about.
So in those two stages, clearlythe input, which is the data,
is absolutely essential.
If anything, I would say thatthe architecture of the neural
(05:44):
networks are very well known.
Everybody is using roughly thesame architecture the bigger one
, there are smaller ones, butroughly the same same family.
So really the difference interms of performance really
comes down to three things whichis the size of the model and
then the amount of compute thatyou have to use to train the
(06:07):
models.
Actually, also, the larger themodel requires more compute.
And then there's, finally,which is the data and how much
data you have.
It's like how much knowledgeyou can actually gather and
clean to train the AI, andthat's how we just get to the
large language models.
(06:28):
But to build applications ontop of it, we have to again
provide domain-specific data,such as financial markets, like
news or documents or research,but then we're talking about
time series.
So everything, frankly, isabout data.
Ai model is the engine, buteverything else is it's really
(06:48):
about data.
Data is the field, data is thefoundation, everything really
you know it's funny, you make methink about.
Speaker 1 (06:55):
You know ongoing training, right.
So you know, back back in theold days and I'm an old guy um,
you know when and I started outin municipal finance and when we
had issues in the market.
You know we would go back tothe New York City bankruptcy of
1978 or the Texas mortgagedefaults in history to get a
(07:18):
better understanding of howmarkets are going to perform
today in a particular crisis.
Given the dynamic of themarkets, the volatility, the
speed in which markets aretraining, how are these models
getting updated?
Are they just constantlylearning through data input and
(07:41):
data flow on a regular basis andthe interaction, or how do you
continuously?
Speaker 2 (07:46):
train?
That's a great question.
So, again, if we go back tokind of the stages of training,
so the pre-training, there's acut-off time.
So if anyone had used ChaiGBT,you would remember the cut-off
time originally was likeNovember 2022 and then updated
to like June 2023.
So the pre-training corpus ofknowledge has a cutoff time.
(08:09):
But what's going on is thesedays the AI companies, they do
not just do one kind of one-timepre-training anymore, they are
actually continuously doingpre-training, just with
different checkpoints.
Every day For example, today'straining, or every hour or
something they will have acheckpoint of the entire model.
(08:29):
So they are generatingversion-controlled large
language models continuously andthey will be updating this
corpus of data for training on a, say, maybe monthly basis.
Because, to be honest, maybethere are a lot of news, but the
fundamental knowledge doesn'tgrow on a daily basis.
So they probably can update iton a weekly or monthly basis.
(08:51):
But even given, for example,when we use a large language
model with a cutoff time of sixmonths ago, what we can do is
because the large language modelalso has a context length.
So these days a model has it'slike the memory, it's like the
size of the memory.
So today open-air models havesomething like 128K tokens,
(09:15):
which is quite big.
It's not that big but it'sdecent.
It can put quite a few researchpapers into it, like the
short-term memory, so you canactually just get the latest
data you're interested in, putit into the context window,
(09:36):
which is equivalent to theshort-term memory, and ask the
large language model to doinference, combining the
pre-trained knowledge and thein-context knowledge.
So that's how people do it.
And then there's another waywhich is actually very important
for finance, which is the toolcalling, which is a lot of
knowledge can be only accessiblebehind the API services.
(10:00):
It could be some calculation oftime series, could be like some
more complicated like datacrunching, say generating a risk
report.
So what happened is the systemkeep running.
So we're not making all the ITsystem obsolete.
But you can say, hey, when weask about certain specific
(10:21):
problems, instead of trying tofigure out yourself, which is
impossible by yourself I meanthe large language models try
calling this API on the fly andfetch the result returned by
that service and use it as partof the response.
So in that case, response isalso always up to and in terms
(10:41):
of prompts.
Speaker 1 (10:42):
Right, I always think about, you know, the layperson.
You know, myself included inthat regard.
Everyone I meet has differentlevels of prompting skills, some
very good, some absolutelyhorrible.
You know, how do you manageprompting.
(11:05):
Is it an education issue?
Are you doing some secretprompting in the behind so your
front office users can ask somereally dumb questions and get
meaningful answers?
How are you dealing with that?
Speaker 2 (11:19):
Yeah, that's such a great question, jim.
There are two parts to it.
The first part is we do a lotof prompting in terms of the
system instructions, becausewhen we build the AI agents to
specialize in different parts ofthe capital markets, we have to
make sure that specialistbehaves like a specialist.
So we use very elaborate systeminstructions to make sure that
(11:43):
they do that.
And actually, you know, peoplemay think that prompting is just
a few sentences, but actuallyin some of the more elaborate AI
agents we built, the promptscan be several pages.
There are a structure to theprompts, there's a description,
there's an objective, there arestyles and then we have to give
examples, both positive andnegative examples.
(12:04):
It's pretty comprehensive.
It's less like a prompt.
There are styles and then wehave to give examples, both
positive and negative examples.
It's pretty comprehensive.
It's less like a prompt, it'smore like a little mini course
to behave this way.
It's more like an agenthandbook, follow this handbook
and then the user has to prompt.
And I think that's actuallyvery underappreciated, because
the one thing we have seen inthat the AI system is actually
(12:28):
very ironic.
The AI system is supposed to bequite user-friendly because the
user interface is just having achat.
It cannot be simpler thanhaving a chat.
But what happens is peoplesometimes because before large
language models, the way wethink about software is entirely
deterministic.
You know, we click a button, weknow exactly what that button
(12:50):
is supposed to do.
We click it.
It does the same thing over andover again.
If we click it once it doesn'twork, we think it's broken.
So that's our intuition aboutsoftware.
But with large language modelsit's non-deterministic, it's
statistical.
You ask the same question, youcan get slightly different
answers, even if the gist of theresponse are the same.
(13:11):
But when people speak to thelarge language model, they sort
of get confused.
They're like okay, should I tryto be clever in my question?
Should I be more specific or bemore open-ended?
Or sometimes they justintuitively try to test the
intelligence of the system.
It's like, okay, let me ask themost difficult question I can
(13:35):
think of, just to see whathappens.
And then so we tell them that,look, I think you know this is a
tool.
So think about the question youwould ask a human analyst or
human junior analyst, so thatyou can get something useful and
productive out of thatconversation.
Don't try to make it difficult.
(13:57):
You know you're not here tryingto embarrass the system.
The system will to make itdifficult.
You're not here to try toembarrass the system.
The system will get clever overtime, but the objective here is
to try to get somethingproductive and useful out of
this conversation.
So, for example, in general,just use common sense.
The more specific you ask, themore specific the questions are.
If you assign a task to yourcolleague, you're not going to
(14:25):
ask some very open-endedphilosophical question because
most likely the colleague willask what do you mean exactly
right?
So I think those are just kindof the good practice to follow
when we use large languagemodels-powered application.
Just use common sense.
A better, more specificquestion will get us a better,
more specific answer.
Speaker 1 (14:42):
That's how my staff responds to my emails when I
give them a task.
What do you mean, jim?
Come on, you do make me thinkof one question.
Right, and you said many of thearchitectures are well known.
They're the same.
You know, data is the fuel,right, but what is the
(15:02):
differentiator?
What makes one AI systemproprietary to another?
Is it the training?
Is it the data?
You know, because a lot of thetechnology is open source.
So where's the magic.
Speaker 2 (15:18):
The magic, I think.
If the audience wants to knowwhere is the magic, I recommend
reading the latest technicalreport by Meta, when they
released Lama 3.1 open sourcemodel.
So there's this 200 or 100, 200page paper describing exactly
how they did it.
The magic is there's no magic,you just have to do it at a huge
(15:42):
, huge scale.
We're talking about, you know,like 30 trillion tokens.
We're talking about like 33,000GPUs in a data center and the
data center gets overheating andevery day the GPU falls over.
It's more like an engineeringproblem now than in terms of the
(16:08):
neural network architecture,than a research problem.
So it's literally just likebuilding it bigger and bigger.
So people are not talking about.
You know, up until now, thebiggest data center is about 100
or 200 megawatts.
And people are thinking aboutOK, how do we build a gigawatt
on data center?
Where do we get the power?
(16:29):
How do we do the cooling?
It's like, you know, 5x thecapacity does not mean 5x the
complexity.
It's probably much higher thanthat.
So I think now we're justthinking about it's more of an
engineering problem.
Speaker 1 (16:39):
Well, I would also argue it's three words small
fusion reactors, Absolutely.
It's certainly a very hardtopic.
So let's stay on magic for asecond.
So in August, SigTechintroduced a new AI-backed
product called Magic.
Can you tell us a little bitmore about what your?
Speaker 2 (16:58):
magic does Our magic coincidentally stands for
multi-agent generativeinvestment co-pilot.
So it's our application, whichis made up of a team of AI
agents.
Each agent is a specialist in aspecific domain of the
financial market.
So we have an agent analyzing,say, central bank's statements,
(17:20):
press conferences and speeches.
We have an agent analyzing theequity market, while analyzing
the microeconomic indicators.
One is like a quant strategistthat can turn your trading or
investment ideas into Pythoncode.
Do the backtesting, give youthe results on the fly.
So we build all these differentagents each doing specific
(17:41):
things, and try to make surethat they do it well.
And then what happens is theuser is like having a group chat
with this team of AI expertsand you ask a question, and then
they will first come up with aplan.
They will say, oh, to answeryour question, Jim, we need a
plan which is made up of 12steps.
Step one this agent is assignedto do it.
(18:05):
And step two, maybe a differentagent is assigned to do it.
So, step by step, it breaksdown your question, your problem
, and then each agent or theexpert is assigned to the right
one and they work incollaboration and then
synthesize the entire teamoutput and then give it to you.
So this is the product.
Can the agents talk to eachother?
Yeah, agents talk to each other.
(18:27):
I think what we managed tofigure out are two things.
One is how to build thesespecialist agents in finance,
and the second thing we figuredout is how to orchestrate, aka
how to manage your AI team.
How to manage your team?
How can they pass informationback and forth, what kind of a
(18:49):
context they have to share?
How do you assign the rightagent to the right task?
So that's what SIG, Tech Magic,is about.
Speaker 1 (18:59):
Wow, that's amazing.
And have the agents startedtheir own language yet, or taken
it over the world, or buildingtheir own portfolios, or
anything weird happen.
Speaker 2 (19:16):
I think today, given they are powered by the current
large language models probablynot AGI yet.
So we can't just ask a questionlike oh, make me a billion
dollars, do it now.
But it's certainly proving tobe very, first, very versatile.
And the second, it's becomevery useful because we're
actually quite surprised by someof the output actually, by some
(19:36):
of the output actually because,again, because the whole system
now is sort of a you know, youcan see, I can visualize the
problem space the team can solvegrows every time we add a new
agent or the agents becomebetter.
So sometimes we get surprisedby the response.
For example, you know we can'ttest all the use cases, but I
(19:59):
remember recently, when we'redoing a live demo, the customer
just said example, we can't testall the use cases, but I
remember recently, when we weredoing a live demo, the customer
just said okay, can you show methat?
Whenever J-PAL said some phrasein a press conference in the
last five years, how did the UStreasury market behave in the
next week?
I've never tried it before.
Literally we're just typingwhatever the customer said on
(20:23):
the call and it just workedright.
You know they're the agentanalyzing, the Fed speak and the
different agents get hold ofthe timestamp of the speech and
figure out what's the right weekto fetch the numbers, and
everything just kind of worked.
Right week to fetch the numbersand everything just kind of
worked.
So we sometimes surpriseourselves.
(20:44):
But the beautiful thing is, Ithink, every time there's a new
model coming along, if there's abig jump in the model
capabilities, all these agentsalmost automatically will become
smarter.
Speaker 1 (20:55):
So you know, obviously these agents are
finishing tasks in seconds ascompared to humans who have got
to sit and read transcripts andoverlay.
You know deep thinking and dealwith disruptions of email and
phone calls, and you know somecolleague needing some sugar for
his coffee.
But you know, perhaps you couldelaborate on this vision and
(21:17):
how these agents could be a gamechanger for financial
institutions.
Speaker 2 (21:22):
Yeah, I think.
First, we do not believe thatthe AI agents will replace front
office users.
We simply don't think that willhappen, certainly not in the
near term.
Because I think, you know,because financial services are
heavily regulated and so theregulators demand that there's a
responsible person for the job,so human always has to be in
(21:45):
the loop.
I guess a good analogy would bedespite how little commercial
pilot actually flies the plane,we still have two.
We still have two, we stillhave two.
And, if anything, if you lookat the evolution of the airplane
navigation technology over thedecades, as the technology gets
(22:07):
better, instead of having fewerpilots, we have more pilots
because the cost of flying hascome down, so the scale of the
airline business goes up andactually the pie has become
bigger.
There are more humans in theloop, more people are employed,
we fly more customers, even ifthe per-person cost has come
down.
So it's quite interesting.
I think we'll probably seesomething similar in finance,
(22:30):
because you know we hire verysmart people in the industry.
Who actually wants to sit thereon a Friday, spend six hours
reading through all the speechesgiven by Jay Park in the last
five years?
I mean, is that interesting?
No, it's not really interesting.
Is that intellectuallystimulating?
Probably not.
Do you want to type into aGoogle Doc or Microsoft Word or
(22:52):
Samurai which will write overthose speeches?
Probably not.
So I think a lot of theseessential but repetitive, boring
tasks can certainly beautomated.
That gives people more room formore creative, deeper and more
interesting jobs.
I think we probably didn't havethe bandwidth to even think
(23:14):
about, or even do.
I think that will open up newpossibilities.
Speaker 1 (23:17):
Well, that's definitely going to change the
role of an analyst job.
Coming out of college, you knowone of the things, and you'd
mentioned highly regulatedindustries.
You know financial institutions, obviously banking much more
higher regulated than, say, thebuy side.
You know, and traditionallythere's always been, say, the
(23:45):
buy side.
You know, and traditionallythere's always been an adoption
curve for the buy side greaterthan the sell side.
There just seems to be a lotslower.
I mean, how do you see, are youseeing that tradition of buy
side being more advanced thansell-side continue to play out,
or are both sides of the fence,if you will, adopting AI at a
(24:06):
similar speed?
Speaker 2 (24:09):
I think the smaller buy-side tends to have much
smaller institutions and smallerinstitutions do tend to move
faster.
And on the buying side theadoption is certainly much
faster than the sell side,because a buying side is
normally under a lot of pressureto beat the market, generate
(24:31):
returns.
It's competitive.
If they don't perform on aquarterly basis, the investor
may pull the money.
It's just a lot morecompetitive pressure on the buy
side, so they are moreopen-minded and therefore
they're also quite keen to lookinto anything they give them.
On the sell side, the nature ofthe business is you are
(24:54):
providing a service, right, sothe competitive nature exists,
but it's much less brutal.
So I think the catalyst foradopting technology like AI is
more coming from operationalefficiencies.
Because of banking, I didn'tday investors are asking about
(25:16):
return on equity, your profitmargins, so it's more of a
business performance levelmetrics.
So something like AI can helpyou with the cost control.
I think, especially when, forexample, in a down cycle, budget
gets cut but people still needto deliver the same amount of
(25:38):
output, you know there are notmany other alternative knobs you
can turn to actually make thathappen.
People can work harder, but howmuch harder can people work.
So AI technology may give thatboost, whereas in upcycle, in
bull market, what tends tohappen is the banking industry
can't hire enough people to dothe business, so it's the
(26:02):
cyclical nature of the businessversus how expensive the human
capitals are that does suggestthat AI technology can be one of
the very few things that canactually modulate this mismatch
between how much you need andhow much cost you have to
monitor.
So, I think, just verydifferent dynamics.
Speaker 1 (26:25):
You know we touched on regulation, being highly
regulated.
You know the concept ofmulti-agent type systems seems
to be the path forward, right?
A lot of conversations I haveyou know it's no longer just
this one big self-service largelanguage model, it's specialized
, vertical multi-agents that caninteract.
(26:48):
What challenges in that model?
Is that going to overlay?
Because it seems like a lot ofregulators still haven't even
figured out the monolithic largelanguage models.
But now you're dealing with awhole series of them.
Is this going to even curtail,perhaps, adoption further, even
(27:11):
though the systems are smarter?
Speaker 2 (27:14):
This is super interesting, I think.
First let me describe why themulti-agent architecture is the
future.
I do believe it's the future.
I think actually divide andconquer in this case is
certainly the way to go, becauseit's like a modular design for
software.
Nobody is going to write themost complicated system in one
(27:38):
file with like 1 million linesof C code.
You have to break it intomodules.
Each one does a specific thingso you can test it.
So if we have this one giant orlarge-language model that does
everything, it means A.
This model has to be very bigso that it can be an expert in
(28:02):
so many different domains.
And then this model has to behighly advanced because now it
has to learn, for example, howto use 10,000 different tools.
Give them a question how toselect the right tools out of,
say, 10 potential tools to useand how to use them in the right
order, which is verycomplicated because the
(28:22):
complexity is like it goes upquadratically.
And then I think, and alsobecause it's one lab, the bigger
the model, the more difficultto interpret the output, because
basically the black box hasjust got bigger.
And then the last bit is likethe bigger the model and it
(28:45):
becomes substantially moreexpensive for the inference to
run, the inference becomes moreexpensive and the latency
becomes much higher.
So it just doesn't scale verywell, whereas when we break it
down into specialists, each canbe run by a much smaller model.
So it's faster, it's cheaper,it can test it faster, it's
(29:10):
cheaper, it can test it and youcan actually have more
possibility in terms ofexplaining how each one of them
is doing.
And what we do here at Sigtek isthat we actually keep track of
all the interactions among theseagents.
So when I ask a question and Ican see my team working on it,
we track everything.
We track all the tasks theywork on.
We track all the actions eachagent takes to work on each task
(29:35):
.
We track all the contacts, theconversations they have among
themselves.
With every output there's awhole graph of team interactions
all recorded.
We can actually go back andreview and rewind and see how
the decisions are arrived bydifferent agents at different
times and how they come together.
So that actually offers somemore visibility into into um
(30:01):
what happened all right, youjust gave me 15 more questions.
Speaker 1 (30:05):
Um so and my producer is going to be yelling at me,
but I always like to argueeverything behind everything new
is something old right?
So you know, arguably we'retalking the same basic tenants
of a microservice typearchitecture, except with agents
.
But you know, within thatmicroservice architecture it's,
(30:25):
you know, it's easy to find failpoints With agents.
To what extent can youdetermine if one's hallucinating
or not?
And then a second part of thatquestion, as it relates to your
monitoring and looking at thedecision-making, as arguably a
(30:47):
lot of professionals right nowin AI are more let's call it IT
developer-based, how much domainknowledge overlay is critical
for that software engineer orprofessional to be able to
understand that decision-making,especially within a specialized
agent?
Speaker 2 (31:08):
Yes, it's fascinating .
I think hallucination iscertainly one of the most
important topics for largelanguage models.
I do personally find the wordhallucination a bit unfortunate
because I feel like the wordsuggests something that's
different from what actuallyhappens.
What happens with so-calledhallucination with large
(31:29):
language models is that I ask aquestion.
It happens that the largelanguage model is not equipped
to answer my question, but it'sunder pressure to give me a
response because I'm trying tohave a conversation here.
So it's under the pressure tosay, hey, let me give it
something by that at leastsounds or looks coherent and
(31:51):
logical.
So it's trying its best to comeup with a response with its
limited knowledge base.
So that's basically whathappens.
If, to be frank, I feel likehallucination is not a right
word for this, we actually havea perfect word for this.
It's called bullshit.
That's the perfect word.
(32:15):
If you ask me something I don'tknow, I feel like I'm under the
pressure to give you something.
I have to say something.
I'm just going to try my bestto BS.
Sometimes I do it better,sometimes I do it worse, but
that's how I give it a go.
Sometimes you catch me,sometimes you don't.
But that's exactly whathappened with large, with large
language mode.
I think there are a few thingshere we can do, and we have done
(32:37):
that.
Um, the one is, um, very basic.
You know, in the prompt you cansay hey, if you don't know what
to do or what to say, don't tryto respond.
Instead, ask me, ask me forclarification.
It's like say, it's like youcan change the system,
instructions to give thisguideline so that that can help
with some cases.
And then the second thing wecan do is so-called grounding.
(33:00):
So grounding is like pretendI'm the large language model.
You ask me a question, doesn'tmatter whether I know the answer
or not.
From my pre-training.
Every question you ask me, jim,we are going to do a semantic
search in an expert database tofetch some relevant documents or
words or paragraphs as thecontext.
(33:22):
So whenever I answer yourquestions, I'm always given a
list of references.
So therefore, that extracontact retrieved from an expert
database is going to ground me,so that even if I know the
extra context cannot be harmful.
(33:43):
If I don't know, that extracontact can be the difference
between me giving you somethingcompletely made up or something
more actually useful.
So, and the fourth is, I thinkthe users, especially in
financial services.
They want to be able to decidewhether I can trust the output
of large language models.
So citation or references are avery important part of the
(34:06):
response.
So we have tuned our models toalways try to provide
comprehensive references,citations, wherever it's
possible, so the user can reallydo some verification themselves
if they want.
I think those are the steps.
(34:26):
I think the hallucination ingeneral the better the model,
the bigger the model, the morekind of trillions of tokens are
used in the pre-training, theless likely, the less ignorance,
so to speak, in alarge-language model.
Therefore there's lesshallucination.
But there are other things wetalk about that can be done to
(34:46):
improve it.
Speaker 1 (34:47):
Well, it's good to know that, whether it's human or
computer, now I still have todeal with bullshit.
So sadly, we've reached thefinal question of the pod, which
we call the trend drop.
It's like a desert islandquestion, and if you could only
watch or track one trend in AIand financial services, what
(35:09):
would it be?
Speaker 2 (35:11):
I think in AI, I think for me the most important
thing is the so-called scalinglaw.
So what happened in the lastthree, four years?
Different companies have, allof them have agreed on the
so-called scaling law, which isthis interesting graph that says
, given this graph, it's likealmost like everybody can fit
(35:31):
their all trained models on thisgraph.
So it's like a collectivelydecided scaling law.
It's like a graph.
The graph basically says if wescale the compute, the model
size and the input training datasize by this amount, how the
(35:52):
performance of the largelanguage model will be.
So we actually know today, forexample, we can say hey, tell me
that if we scale the compute by10 times, input data by six
times, model size by five times,that's optimal.
There's a relative ratio tothese parameters.
(36:15):
Where that performance?
You know what does theperformance look like?
We actually know, okay, weactually know.
We know this graph and so far,all the models delivered by
different companies all followthis.
We saw it just go up and up andup and they follow this almost
(36:37):
exactly.
It's actually more interesting,probably more fundamental than,
for example, the Moore's Law.
Moore's Law is more of anobservation of the productivity
of the semiconductor industry,but this one actually is more
mathematical, it's morefundamental.
I would be absolutelyinterested in observing in the
(37:00):
coming months, quarters andyears whether the models
released by different companiescan still follow this scaling
law.
Are we going to hit a plateau,hit a ceiling, or actually it's
going to accelerate more?
So that's, I think, the onething I really want to track in
AI.
Speaker 1 (37:20):
Well, two words quantum computing.
Quantum computing yes.
Speaker 2 (37:25):
Maybe the exactly and in terms of financial services,
I think the one thing I wouldbe interested in would be, I
think, the younger generation.
If I think about therelationship between more
intimate relationship with thebank, because they go to the
(37:52):
bank branches, they speak topeople face-to-face, they worry
about oh, how do I plan this,plan that where the younger
generation, we no longer havethat relationship with the bank.
Everything we do is throughmaybe an application on my
iPhone or on a website.
(38:13):
It's entirely digital.
So I think all the financialservices is already very much,
over the years, transitioningfrom this in-person,
relationship-driven businessmodel to a much larger scale
digital transformation, digitaldistribution, especially given
(38:34):
these days, a lot of thefinancial services and products,
arguably, are quite fungible,similar.
So it's very much aboutdistribution.
And then, today, digitaldistribution.
So I think about digitalbanking, I think about digital
distribution, I think aboutdigital banking, I think about
digital distribution, I thinkabout digital wealth advisory
(38:59):
and how AI can play differentroles in those places.
So, yeah, those would be thethings that would be most
interesting.
Speaker 1 (39:08):
Well, I'd be happy to stay out of the branch, but TD
hasn't figured out how to giveme a free pen and a lollipop via
the app, so maybe that's wherethe drones come in.
I go to the app and they dropit off.
All right, ben, thank you somuch for your time.
What a great conversation.
I enjoyed it.
I know our audience has, sothank you so much.
(39:28):
Thank you, jim, it's a pleasure.
Thanks so much for listening totoday's episode and if you're
enjoying Trading Tomorrow,navigating trends and capital
markets, be sure to like,subscribe and share, and we'll
see you next time.
Welcome to Trading Tomorrow Navigating Trends in
Capital Markets the podcastwhere we deep dive into
technologies reshaping the worldof capital markets.
I'm your host, jim Jockle, aveteran of the finance industry
with a passion for thecomplexities of financial
technologies and market trends.
In each episode, we'll explorethe cutting-edge trends, tools
and strategies driving today'sfinancial landscapes and paving
(00:29):
the way for the future.
With the finance industry at apivotal point, influenced by
groundbreaking innovations, it'smore crucial than ever to
understand how thesetechnological advancements
interact with market dynamics.
In today's episode, we'reexploring one of the most
(00:57):
transformative forces in financeartificial intelligence.
Ai is reshaping capital markets, with buy-side and sell-side
firms adopting innovativetechnologies to gain an edge.
But with all this innovationcomes the need for a robust data
foundation.
Today, we're going to explorehow AI is not only boosting
productivity, but also changingthe roles of financial
(01:19):
professionals and what thefuture holds as AI becomes more
deeply integrated into thefinancial landscape.
To help us navigate this topic,we're joined by Bin Ren, founder
and CEO of SigTech, a companyat the forefront of AI
innovation in finance.
Before starting SigTech in 2019, bin was the chief investment
officer of the systemicinvestment group at Brevin
(01:39):
Howard, where he led thequantitative investment funds.
He began his career at Barclaysas an equity exotics trader.
Welcome, ben.
Thank you for having me today,jim, absolutely Pleasure.
So you know what.
Just to start off, why don't wetalk about SigTech?
Speaker 2 (01:53):
So SigTech we have been around since 2019, so over
five years.
We spun up from a hedge fundcalled Brevin Howard and we
really focus on building andshipping the best productivity
tools for people who work in thefront office of the capital
markets.
(02:13):
So our users tend to make veryimportant financial decisions,
such as investment management ortrading or risk management.
So in their daily life, thespeed and quality of the
decisions matter.
So that's why we want to buildthe best tools to help them.
Speaker 1 (02:34):
And what is the key role of AI in the platform?
Speaker 2 (02:39):
So we started five years ago.
So a big part of our product isto help people to do data-driven
analysis.
So in finance, it's very muchabout numbers, their time series
, but also textual data you'regetting from different sources
on a daily basis.
So AI has really helped us inthe last two years to lower the
(03:02):
hurdle in terms of how muchknowledge a user has to have in
terms of programming and dataanalysis and to actually do what
used to be the specialist jobof a data analyst.
So I think AI has made a hugedifference in the last two years
to be able to write veryhigh-quality code on behalf of a
(03:23):
user and be able to actuallyprocess a lot of the fairly
complicated so-called naturallanguage processing problems.
For example, Jay Powell justgave a speech at the Jackson
Hole and talking about making itactually very important and
saying central banks are notready to cut rates, and what
happened is that actually thespeech was published while he
(03:46):
started speaking and our usersare able to immediately, through
large language models, trainand fine-tune by us to be able
to ask questions like summarizea speech for me right now, focus
on potential for rate cut infive seconds, even way before
the speech is finished.
I think this is a sort of oneof the many interesting use
(04:11):
cases that AI has been able todeliver.
Speaker 1 (04:14):
I guess gone are the days of everybody huddling
around CNBC and the floor goingquiet.
That's amazing.
So you know the AI product isbacked by robust data and you
mentioned that you know.
Perhaps you can explain theimportance of having strong data
foundation, especially whenbuilding AI-driven systems.
Speaker 2 (04:35):
I think if we think about how the launch language
models work, I think there aremultiple stages of training.
Okay, there's the first stage.
Actually, actually it's calledthe pre-training.
Pre-training is where we have ahuge corpus of data we're
talking about trillions ofso-called tokens covering as
(04:57):
many different domains aspossible to train a very large
neural network to essentiallybecome a very competent and
knowledgeable generalist.
And then there's the secondstage, which is called
post-training is to align thislarge language model with human
objectives.
So when we interact with themodels, we want the model to
(05:24):
respond to us in a certain waythat aligns with our style of
conversation, of our way ofthinking, rather than just
predicting the next token, whichis what the pre-training is
about.
So in those two stages, clearlythe input, which is the data,
is absolutely essential.
If anything, I would say thatthe architecture of the neural
(05:44):
networks are very well known.
Everybody is using roughly thesame architecture the bigger one
, there are smaller ones, butroughly the same same family.
So really the difference interms of performance really
comes down to three things whichis the size of the model and
then the amount of compute thatyou have to use to train the
(06:07):
models.
Actually, also, the larger themodel requires more compute.
And then there's, finally,which is the data and how much
data you have.
It's like how much knowledgeyou can actually gather and
clean to train the AI, andthat's how we just get to the
large language models.
(06:28):
But to build applications ontop of it, we have to again
provide domain-specific data,such as financial markets, like
news or documents or research,but then we're talking about
time series.
So everything, frankly, isabout data.
Ai model is the engine, buteverything else is it's really
(06:48):
about data.
Data is the field, data is thefoundation, everything really
you know it's funny, you make methink about.
Speaker 1 (06:55):
You know ongoing training, right.
So you know, back back in theold days and I'm an old guy um,
you know when and I started outin municipal finance and when we
had issues in the market.
You know we would go back tothe New York City bankruptcy of
1978 or the Texas mortgagedefaults in history to get a
(07:18):
better understanding of howmarkets are going to perform
today in a particular crisis.
Given the dynamic of themarkets, the volatility, the
speed in which markets aretraining, how are these models
getting updated?
Are they just constantlylearning through data input and
(07:41):
data flow on a regular basis andthe interaction, or how do you
continuously?
Speaker 2 (07:46):
train?
That's a great question.
So, again, if we go back tokind of the stages of training,
so the pre-training, there's acut-off time.
So if anyone had used ChaiGBT,you would remember the cut-off
time originally was likeNovember 2022 and then updated
to like June 2023.
So the pre-training corpus ofknowledge has a cutoff time.
(08:09):
But what's going on is thesedays the AI companies, they do
not just do one kind of one-timepre-training anymore, they are
actually continuously doingpre-training, just with
different checkpoints.
Every day For example, today'straining, or every hour or
something they will have acheckpoint of the entire model.
(08:29):
So they are generatingversion-controlled large
language models continuously andthey will be updating this
corpus of data for training on a, say, maybe monthly basis.
Because, to be honest, maybethere are a lot of news, but the
fundamental knowledge doesn'tgrow on a daily basis.
So they probably can update iton a weekly or monthly basis.
(08:51):
But even given, for example,when we use a large language
model with a cutoff time of sixmonths ago, what we can do is
because the large language modelalso has a context length.
So these days a model has it'slike the memory, it's like the
size of the memory.
So today open-air models havesomething like 128K tokens,
(09:15):
which is quite big.
It's not that big but it'sdecent.
It can put quite a few researchpapers into it, like the
short-term memory, so you canactually just get the latest
data you're interested in, putit into the context window,
(09:36):
which is equivalent to theshort-term memory, and ask the
large language model to doinference, combining the
pre-trained knowledge and thein-context knowledge.
So that's how people do it.
And then there's another waywhich is actually very important
for finance, which is the toolcalling, which is a lot of
knowledge can be only accessiblebehind the API services.
(10:00):
It could be some calculation oftime series, could be like some
more complicated like datacrunching, say generating a risk
report.
So what happened is the systemkeep running.
So we're not making all the ITsystem obsolete.
But you can say, hey, when weask about certain specific
(10:21):
problems, instead of trying tofigure out yourself, which is
impossible by yourself I meanthe large language models try
calling this API on the fly andfetch the result returned by
that service and use it as partof the response.
So in that case, response isalso always up to and in terms
(10:41):
of prompts.
Speaker 1 (10:42):
Right, I always think about, you know, the layperson.
You know, myself included inthat regard.
Everyone I meet has differentlevels of prompting skills, some
very good, some absolutelyhorrible.
You know, how do you manageprompting.
(11:05):
Is it an education issue?
Are you doing some secretprompting in the behind so your
front office users can ask somereally dumb questions and get
meaningful answers?
How are you dealing with that?
Speaker 2 (11:19):
Yeah, that's such a great question, jim.
There are two parts to it.
The first part is we do a lotof prompting in terms of the
system instructions, becausewhen we build the AI agents to
specialize in different parts ofthe capital markets, we have to
make sure that specialistbehaves like a specialist.
So we use very elaborate systeminstructions to make sure that
(11:43):
they do that.
And actually, you know, peoplemay think that prompting is just
a few sentences, but actuallyin some of the more elaborate AI
agents we built, the promptscan be several pages.
There are a structure to theprompts, there's a description,
there's an objective, there arestyles and then we have to give
examples, both positive andnegative examples.
(12:04):
It's pretty comprehensive.
It's less like a prompt.
There are styles and then wehave to give examples, both
positive and negative examples.
It's pretty comprehensive.
It's less like a prompt, it'smore like a little mini course
to behave this way.
It's more like an agenthandbook, follow this handbook
and then the user has to prompt.
And I think that's actuallyvery underappreciated, because
the one thing we have seen inthat the AI system is actually
(12:28):
very ironic.
The AI system is supposed to bequite user-friendly because the
user interface is just having achat.
It cannot be simpler thanhaving a chat.
But what happens is peoplesometimes because before large
language models, the way wethink about software is entirely
deterministic.
You know, we click a button, weknow exactly what that button
(12:50):
is supposed to do.
We click it.
It does the same thing over andover again.
If we click it once it doesn'twork, we think it's broken.
So that's our intuition aboutsoftware.
But with large language modelsit's non-deterministic, it's
statistical.
You ask the same question, youcan get slightly different
answers, even if the gist of theresponse are the same.
(13:11):
But when people speak to thelarge language model, they sort
of get confused.
They're like okay, should I tryto be clever in my question?
Should I be more specific or bemore open-ended?
Or sometimes they justintuitively try to test the
intelligence of the system.
It's like, okay, let me ask themost difficult question I can
(13:35):
think of, just to see whathappens.
And then so we tell them that,look, I think you know this is a
tool.
So think about the question youwould ask a human analyst or
human junior analyst, so thatyou can get something useful and
productive out of thatconversation.
Don't try to make it difficult.
(13:57):
You know you're not here tryingto embarrass the system.
The system will to make itdifficult.
You're not here to try toembarrass the system.
The system will get clever overtime, but the objective here is
to try to get somethingproductive and useful out of
this conversation.
So, for example, in general,just use common sense.
The more specific you ask, themore specific the questions are.
If you assign a task to yourcolleague, you're not going to
(14:25):
ask some very open-endedphilosophical question because
most likely the colleague willask what do you mean exactly
right?
So I think those are just kindof the good practice to follow
when we use large languagemodels-powered application.
Just use common sense.
A better, more specificquestion will get us a better,
more specific answer.
Speaker 1 (14:42):
That's how my staff responds to my emails when I
give them a task.
What do you mean, jim?
Come on, you do make me thinkof one question.
Right, and you said many of thearchitectures are well known.
They're the same.
You know, data is the fuel,right, but what is the
(15:02):
differentiator?
What makes one AI systemproprietary to another?
Is it the training?
Is it the data?
You know, because a lot of thetechnology is open source.
So where's the magic.
Speaker 2 (15:18):
The magic, I think.
If the audience wants to knowwhere is the magic, I recommend
reading the latest technicalreport by Meta, when they
released Lama 3.1 open sourcemodel.
So there's this 200 or 100, 200page paper describing exactly
how they did it.
The magic is there's no magic,you just have to do it at a huge
(15:42):
, huge scale.
We're talking about, you know,like 30 trillion tokens.
We're talking about like 33,000GPUs in a data center and the
data center gets overheating andevery day the GPU falls over.
It's more like an engineeringproblem now than in terms of the
(16:08):
neural network architecture,than a research problem.
So it's literally just likebuilding it bigger and bigger.
So people are not talking about.
You know, up until now, thebiggest data center is about 100
or 200 megawatts.
And people are thinking aboutOK, how do we build a gigawatt
on data center?
Where do we get the power?
(16:29):
How do we do the cooling?
It's like, you know, 5x thecapacity does not mean 5x the
complexity.
It's probably much higher thanthat.
So I think now we're justthinking about it's more of an
engineering problem.
Speaker 1 (16:39):
Well, I would also argue it's three words small
fusion reactors, Absolutely.
It's certainly a very hardtopic.
So let's stay on magic for asecond.
So in August, SigTechintroduced a new AI-backed
product called Magic.
Can you tell us a little bitmore about what your?
Speaker 2 (16:58):
magic does Our magic coincidentally stands for
multi-agent generativeinvestment co-pilot.
So it's our application, whichis made up of a team of AI
agents.
Each agent is a specialist in aspecific domain of the
financial market.
So we have an agent analyzing,say, central bank's statements,
(17:20):
press conferences and speeches.
We have an agent analyzing theequity market, while analyzing
the microeconomic indicators.
One is like a quant strategistthat can turn your trading or
investment ideas into Pythoncode.
Do the backtesting, give youthe results on the fly.
So we build all these differentagents each doing specific
(17:41):
things, and try to make surethat they do it well.
And then what happens is theuser is like having a group chat
with this team of AI expertsand you ask a question, and then
they will first come up with aplan.
They will say, oh, to answeryour question, Jim, we need a
plan which is made up of 12steps.
Step one this agent is assignedto do it.
(18:05):
And step two, maybe a differentagent is assigned to do it.
So, step by step, it breaksdown your question, your problem
, and then each agent or theexpert is assigned to the right
one and they work incollaboration and then
synthesize the entire teamoutput and then give it to you.
So this is the product.
Can the agents talk to eachother?
Yeah, agents talk to each other.
(18:27):
I think what we managed tofigure out are two things.
One is how to build thesespecialist agents in finance,
and the second thing we figuredout is how to orchestrate, aka
how to manage your AI team.
How to manage your team?
How can they pass informationback and forth, what kind of a
(18:49):
context they have to share?
How do you assign the rightagent to the right task?
So that's what SIG, Tech Magic,is about.
Speaker 1 (18:59):
Wow, that's amazing.
And have the agents startedtheir own language yet, or taken
it over the world, or buildingtheir own portfolios, or
anything weird happen.
Speaker 2 (19:16):
I think today, given they are powered by the current
large language models probablynot AGI yet.
So we can't just ask a questionlike oh, make me a billion
dollars, do it now.
But it's certainly proving tobe very, first, very versatile.
And the second, it's becomevery useful because we're
actually quite surprised by someof the output actually, by some
(19:36):
of the output actually because,again, because the whole system
now is sort of a you know, youcan see, I can visualize the
problem space the team can solvegrows every time we add a new
agent or the agents becomebetter.
So sometimes we get surprisedby the response.
For example, you know we can'ttest all the use cases, but I
(19:59):
remember recently, when we'redoing a live demo, the customer
just said example, we can't testall the use cases, but I
remember recently, when we weredoing a live demo, the customer
just said okay, can you show methat?
Whenever J-PAL said some phrasein a press conference in the
last five years, how did the UStreasury market behave in the
next week?
I've never tried it before.
Literally we're just typingwhatever the customer said on
(20:23):
the call and it just workedright.
You know they're the agentanalyzing, the Fed speak and the
different agents get hold ofthe timestamp of the speech and
figure out what's the right weekto fetch the numbers, and
everything just kind of worked.
Right week to fetch the numbersand everything just kind of
worked.
So we sometimes surpriseourselves.
(20:44):
But the beautiful thing is, Ithink, every time there's a new
model coming along, if there's abig jump in the model
capabilities, all these agentsalmost automatically will become
smarter.
Speaker 1 (20:55):
So you know, obviously these agents are
finishing tasks in seconds ascompared to humans who have got
to sit and read transcripts andoverlay.
You know deep thinking and dealwith disruptions of email and
phone calls, and you know somecolleague needing some sugar for
his coffee.
But you know, perhaps you couldelaborate on this vision and
(21:17):
how these agents could be a gamechanger for financial
institutions.
Speaker 2 (21:22):
Yeah, I think.
First, we do not believe thatthe AI agents will replace front
office users.
We simply don't think that willhappen, certainly not in the
near term.
Because I think, you know,because financial services are
heavily regulated and so theregulators demand that there's a
responsible person for the job,so human always has to be in
(21:45):
the loop.
I guess a good analogy would bedespite how little commercial
pilot actually flies the plane,we still have two.
We still have two, we stillhave two.
And, if anything, if you lookat the evolution of the airplane
navigation technology over thedecades, as the technology gets
(22:07):
better, instead of having fewerpilots, we have more pilots
because the cost of flying hascome down, so the scale of the
airline business goes up andactually the pie has become
bigger.
There are more humans in theloop, more people are employed,
we fly more customers, even ifthe per-person cost has come
down.
So it's quite interesting.
I think we'll probably seesomething similar in finance,
(22:30):
because you know we hire verysmart people in the industry.
Who actually wants to sit thereon a Friday, spend six hours
reading through all the speechesgiven by Jay Park in the last
five years?
I mean, is that interesting?
No, it's not really interesting.
Is that intellectuallystimulating?
Probably not.
Do you want to type into aGoogle Doc or Microsoft Word or
(22:52):
Samurai which will write overthose speeches?
Probably not.
So I think a lot of theseessential but repetitive, boring
tasks can certainly beautomated.
That gives people more room formore creative, deeper and more
interesting jobs.
I think we probably didn't havethe bandwidth to even think
(23:14):
about, or even do.
I think that will open up newpossibilities.
Speaker 1 (23:17):
Well, that's definitely going to change the
role of an analyst job.
Coming out of college, you knowone of the things, and you'd
mentioned highly regulatedindustries.
You know financial institutions, obviously banking much more
higher regulated than, say, thebuy side.
You know, and traditionallythere's always been, say, the
(23:45):
buy side.
You know, and traditionallythere's always been an adoption
curve for the buy side greaterthan the sell side.
There just seems to be a lotslower.
I mean, how do you see, are youseeing that tradition of buy
side being more advanced thansell-side continue to play out,
or are both sides of the fence,if you will, adopting AI at a
(24:06):
similar speed?
Speaker 2 (24:09):
I think the smaller buy-side tends to have much
smaller institutions and smallerinstitutions do tend to move
faster.
And on the buying side theadoption is certainly much
faster than the sell side,because a buying side is
normally under a lot of pressureto beat the market, generate
(24:31):
returns.
It's competitive.
If they don't perform on aquarterly basis, the investor
may pull the money.
It's just a lot morecompetitive pressure on the buy
side, so they are moreopen-minded and therefore
they're also quite keen to lookinto anything they give them.
On the sell side, the nature ofthe business is you are
(24:54):
providing a service, right, sothe competitive nature exists,
but it's much less brutal.
So I think the catalyst foradopting technology like AI is
more coming from operationalefficiencies.
Because of banking, I didn'tday investors are asking about
(25:16):
return on equity, your profitmargins, so it's more of a
business performance levelmetrics.
So something like AI can helpyou with the cost control.
I think, especially when, forexample, in a down cycle, budget
gets cut but people still needto deliver the same amount of
(25:38):
output, you know there are notmany other alternative knobs you
can turn to actually make thathappen.
People can work harder, but howmuch harder can people work.
So AI technology may give thatboost, whereas in upcycle, in
bull market, what tends tohappen is the banking industry
can't hire enough people to dothe business, so it's the
(26:02):
cyclical nature of the businessversus how expensive the human
capitals are that does suggestthat AI technology can be one of
the very few things that canactually modulate this mismatch
between how much you need andhow much cost you have to
monitor.
So, I think, just verydifferent dynamics.
Speaker 1 (26:25):
You know we touched on regulation, being highly
regulated.
You know the concept ofmulti-agent type systems seems
to be the path forward, right?
A lot of conversations I haveyou know it's no longer just
this one big self-service largelanguage model, it's specialized
, vertical multi-agents that caninteract.
(26:48):
What challenges in that model?
Is that going to overlay?
Because it seems like a lot ofregulators still haven't even
figured out the monolithic largelanguage models.
But now you're dealing with awhole series of them.
Is this going to even curtail,perhaps, adoption further, even
(27:11):
though the systems are smarter?
Speaker 2 (27:14):
This is super interesting, I think.
First let me describe why themulti-agent architecture is the
future.
I do believe it's the future.
I think actually divide andconquer in this case is
certainly the way to go, becauseit's like a modular design for
software.
Nobody is going to write themost complicated system in one
(27:38):
file with like 1 million linesof C code.
You have to break it intomodules.
Each one does a specific thingso you can test it.
So if we have this one giant orlarge-language model that does
everything, it means A.
This model has to be very bigso that it can be an expert in
(28:02):
so many different domains.
And then this model has to behighly advanced because now it
has to learn, for example, howto use 10,000 different tools.
Give them a question how toselect the right tools out of,
say, 10 potential tools to useand how to use them in the right
order, which is verycomplicated because the
(28:22):
complexity is like it goes upquadratically.
And then I think, and alsobecause it's one lab, the bigger
the model, the more difficultto interpret the output, because
basically the black box hasjust got bigger.
And then the last bit is likethe bigger the model and it
(28:45):
becomes substantially moreexpensive for the inference to
run, the inference becomes moreexpensive and the latency
becomes much higher.
So it just doesn't scale verywell, whereas when we break it
down into specialists, each canbe run by a much smaller model.
So it's faster, it's cheaper,it can test it faster, it's
(29:10):
cheaper, it can test it and youcan actually have more
possibility in terms ofexplaining how each one of them
is doing.
And what we do here at Sigtek isthat we actually keep track of
all the interactions among theseagents.
So when I ask a question and Ican see my team working on it,
we track everything.
We track all the tasks theywork on.
We track all the actions eachagent takes to work on each task
(29:35):
.
We track all the contacts, theconversations they have among
themselves.
With every output there's awhole graph of team interactions
all recorded.
We can actually go back andreview and rewind and see how
the decisions are arrived bydifferent agents at different
times and how they come together.
So that actually offers somemore visibility into into um
(30:01):
what happened all right, youjust gave me 15 more questions.
Speaker 1 (30:05):
Um so and my producer is going to be yelling at me,
but I always like to argueeverything behind everything new
is something old right?
So you know, arguably we'retalking the same basic tenants
of a microservice typearchitecture, except with agents
.
But you know, within thatmicroservice architecture it's,
(30:25):
you know, it's easy to find failpoints With agents.
To what extent can youdetermine if one's hallucinating
or not?
And then a second part of thatquestion, as it relates to your
monitoring and looking at thedecision-making, as arguably a
(30:47):
lot of professionals right nowin AI are more let's call it IT
developer-based, how much domainknowledge overlay is critical
for that software engineer orprofessional to be able to
understand that decision-making,especially within a specialized
agent?
Speaker 2 (31:08):
Yes, it's fascinating .
I think hallucination iscertainly one of the most
important topics for largelanguage models.
I do personally find the wordhallucination a bit unfortunate
because I feel like the wordsuggests something that's
different from what actuallyhappens.
What happens with so-calledhallucination with large
(31:29):
language models is that I ask aquestion.
It happens that the largelanguage model is not equipped
to answer my question, but it'sunder pressure to give me a
response because I'm trying tohave a conversation here.
So it's under the pressure tosay, hey, let me give it
something by that at leastsounds or looks coherent and
(31:51):
logical.
So it's trying its best to comeup with a response with its
limited knowledge base.
So that's basically whathappens.
If, to be frank, I feel likehallucination is not a right
word for this, we actually havea perfect word for this.
It's called bullshit.
That's the perfect word.
(32:15):
If you ask me something I don'tknow, I feel like I'm under the
pressure to give you something.
I have to say something.
I'm just going to try my bestto BS.
Sometimes I do it better,sometimes I do it worse, but
that's how I give it a go.
Sometimes you catch me,sometimes you don't.
But that's exactly whathappened with large, with large
language mode.
I think there are a few thingshere we can do, and we have done
(32:37):
that.
Um, the one is, um, very basic.
You know, in the prompt you cansay hey, if you don't know what
to do or what to say, don't tryto respond.
Instead, ask me, ask me forclarification.
It's like say, it's like youcan change the system,
instructions to give thisguideline so that that can help
with some cases.
And then the second thing wecan do is so-called grounding.
(33:00):
So grounding is like pretendI'm the large language model.
You ask me a question, doesn'tmatter whether I know the answer
or not.
From my pre-training.
Every question you ask me, jim,we are going to do a semantic
search in an expert database tofetch some relevant documents or
words or paragraphs as thecontext.
(33:22):
So whenever I answer yourquestions, I'm always given a
list of references.
So therefore, that extracontact retrieved from an expert
database is going to ground me,so that even if I know the
extra context cannot be harmful.
(33:43):
If I don't know, that extracontact can be the difference
between me giving you somethingcompletely made up or something
more actually useful.
So, and the fourth is, I thinkthe users, especially in
financial services.
They want to be able to decidewhether I can trust the output
of large language models.
So citation or references are avery important part of the
(34:06):
response.
So we have tuned our models toalways try to provide
comprehensive references,citations, wherever it's
possible, so the user can reallydo some verification themselves
if they want.
I think those are the steps.
(34:26):
I think the hallucination ingeneral the better the model,
the bigger the model, the morekind of trillions of tokens are
used in the pre-training, theless likely, the less ignorance,
so to speak, in alarge-language model.
Therefore there's lesshallucination.
But there are other things wetalk about that can be done to
(34:46):
improve it.
Speaker 1 (34:47):
Well, it's good to know that, whether it's human or
computer, now I still have todeal with bullshit.
So sadly, we've reached thefinal question of the pod, which
we call the trend drop.
It's like a desert islandquestion, and if you could only
watch or track one trend in AIand financial services, what
(35:09):
would it be?
Speaker 2 (35:11):
I think in AI, I think for me the most important
thing is the so-called scalinglaw.
So what happened in the lastthree, four years?
Different companies have, allof them have agreed on the
so-called scaling law, which isthis interesting graph that says
, given this graph, it's likealmost like everybody can fit
(35:31):
their all trained models on thisgraph.
So it's like a collectivelydecided scaling law.
It's like a graph.
The graph basically says if wescale the compute, the model
size and the input training datasize by this amount, how the
(35:52):
performance of the largelanguage model will be.
So we actually know today, forexample, we can say hey, tell me
that if we scale the compute by10 times, input data by six
times, model size by five times,that's optimal.
There's a relative ratio tothese parameters.
(36:15):
Where that performance?
You know what does theperformance look like?
We actually know, okay, weactually know.
We know this graph and so far,all the models delivered by
different companies all followthis.
We saw it just go up and up andup and they follow this almost
(36:37):
exactly.
It's actually more interesting,probably more fundamental than,
for example, the Moore's Law.
Moore's Law is more of anobservation of the productivity
of the semiconductor industry,but this one actually is more
mathematical, it's morefundamental.
I would be absolutelyinterested in observing in the
(37:00):
coming months, quarters andyears whether the models
released by different companiescan still follow this scaling
law.
Are we going to hit a plateau,hit a ceiling, or actually it's
going to accelerate more?
So that's, I think, the onething I really want to track in
AI.
Speaker 1 (37:20):
Well, two words quantum computing.
Quantum computing yes.
Speaker 2 (37:25):
Maybe the exactly and in terms of financial services,
I think the one thing I wouldbe interested in would be, I
think, the younger generation.
If I think about therelationship between more
intimate relationship with thebank, because they go to the
(37:52):
bank branches, they speak topeople face-to-face, they worry
about oh, how do I plan this,plan that where the younger
generation, we no longer havethat relationship with the bank.
Everything we do is throughmaybe an application on my
iPhone or on a website.
(38:13):
It's entirely digital.
So I think all the financialservices is already very much,
over the years, transitioningfrom this in-person,
relationship-driven businessmodel to a much larger scale
digital transformation, digitaldistribution, especially given
(38:34):
these days, a lot of thefinancial services and products,
arguably, are quite fungible,similar.
So it's very much aboutdistribution.
And then, today, digitaldistribution.
So I think about digitalbanking, I think about digital
distribution, I think aboutdigital banking, I think about
digital distribution, I thinkabout digital wealth advisory
(38:59):
and how AI can play differentroles in those places.
So, yeah, those would be thethings that would be most
interesting.
Speaker 1 (39:08):
Well, I'd be happy to stay out of the branch, but TD
hasn't figured out how to giveme a free pen and a lollipop via
the app, so maybe that's wherethe drones come in.
I go to the app and they dropit off.
All right, ben, thank you somuch for your time.
What a great conversation.
I enjoyed it.
I know our audience has, sothank you so much.
(39:28):
Thank you, jim, it's a pleasure.
Thanks so much for listening totoday's episode and if you're
enjoying Trading Tomorrow,navigating trends and capital
markets, be sure to like,subscribe and share, and we'll
see you next time.
Popular Podcasts
On Purpose with Jay Shetty
I’m Jay Shetty host of On Purpose the worlds #1 Mental Health podcast and I’m so grateful you found us. I started this podcast 5 years ago to invite you into conversations and workshops that are designed to help make you happier, healthier and more healed. I believe that when you (yes you) feel seen, heard and understood you’re able to deal with relationship struggles, work challenges and life’s ups and downs with more ease and grace. I interview experts, celebrities, thought leaders and athletes so that we can grow our mindset, build better habits and uncover a side of them we’ve never seen before. New episodes every Monday and Friday. Your support means the world to me and I don’t take it for granted — click the follow button and leave a review to help us spread the love with On Purpose. I can’t wait for you to listen to your first or 500th episode!
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.