August 27, 2025 • 43 mins
Daniel and Chris sit with Citadel AI’s Rick Kobayashi and Kenny Song and unpack AI safety and security challenges in the generative AI era. They compare Japan’s approach to AI adoption with the US’s, and explore the implications of real-world failures in AI systems, along with strategies for AI monitoring and evaluation.
Featuring:
- Rick Kobayashi – LinkedIn
- Kenny Song – LinkedIn
- Chris Benson – Website, LinkedIn, Bluesky, GitHub, X
- Daniel Whitenack – Website, GitHub, X
Links:
Register for upcoming webinars here!
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Jerod (00:04):
Welcome to the Practical AI podcast, where we break down
the real world applications ofartificial intelligence and how
it's shaping the way we live,work, and create. Our goal is to
help make AI technologypractical, productive, and
accessible to everyone. Whetheryou're a developer, business
leader, or just curious aboutthe tech behind the buzz, you're
(00:24):
in the right place. Be sure toconnect with us on LinkedIn, X,
or Blue Sky to stay up to datewith episode drops, behind the
scenes content, and AI insights.You can learn more at
practicalai.fm.
Now, onto the show.
Daniel (00:49):
Welcome to another episode of the Practical AI
Podcast. This is DanielWitenack. I am CEO at Prediction
Guard, and I am joined as alwaysby my cohost, Chris Benson, who
is a principal AI researchengineer at Lockheed Martin. How
are doing, Chris?
Chris (01:04):
Hey. Doing great today, Daniel. How's it going with you?
Daniel (01:07):
It's going great. I traveled a bit over the weekend
to run a mini marathon, whichwas fun, but I am back safe at
home. And, of course, safety isis something that hopefully we
can talk a little bit abouttoday with our with our guests
who are, Rick Kobayashi, who iscofounder and CEO at Citadel AI,
(01:28):
and Kenny Song, who is cofounderand CTO at Citadel AI. Welcome.
Kenny (01:34):
Thank you for having us.
Rick (01:35):
Yeah. Thank you, Hwal.
Daniel (01:36):
Yeah. It's great to have you both here. Good to get
introduced to you both. I'm I'mreally excited about this
conversation because, of course,security and safety related to
AI is very close to my own work,so it's always great to connect
with people in this space. Yeah,I'm also interested to hear a
(01:58):
little bit, and I even saw somethings on your website, you
know, talking a little bit aboutsome of the AI in Japan, some of
the, I guess, regulations orguidelines for for businesses
that have come out there.
So maybe before we get into thenature of what you're building
and what you're doing, anythoughts for those out there
(02:21):
that might be in The US or inEurope and be constantly exposed
to the things that are going onin AI in those jurisdictions,
Any thoughts about what is thesame or what stands out about
what's happening with AI inJapan?
Rick (02:37):
So, yeah, anyway, thank you very much for joining us. My
name is Rick. And as for theJapanese conditions, basically,
I'm afraid to say it's aroundone, two years behind The US
situation, basically, up to now.But on the other hand,
surprisingly, as for Gen AI, Ithink that Japan is one of the
(03:00):
most advanced countries. Not thedevelopment of the foundation
models, the utilization of theGen AI applications.
And probably it might be relatedto kind of Japanese animation or
cartoon. So, you know, likethere are lots of say robot or
(03:21):
some of the say animations whichspecify some of the AI or robot
technologies and, you know,Doraemon or there are lots of
animation over there. And peopleare very getting familiar with
such a kind of talking withrobot or some of these advanced
computers from the childhood. Soin that sense, I think that many
(03:45):
people do not have anyhesitation to work with or talk
with like a chatbot or anyGenAI. So in that sense, the
hurdle to introduce LNM or GenAIin Japan is very low in that
compared with US or othercountries.
So in essence, development sideor technology side, frankly
(04:05):
speaking, Japan is behind The USsituation. But as for the usage
of such kind of a generic, Ithink, I hope Japan is one of
the most advanced countries.
Daniel (04:17):
Yeah. That's really interesting to hear the sort of
perception side of things. Andof course with more, maybe even
more adoption or adoption that'sahead of maybe other places in
the world, it could be thatusers or usage of the technology
(04:39):
has hit some bumps along theroad or has hit some problems,
which I know is a lot of whatyou all are working on. What has
been the situation in Japanaround regulation and business
usage of AI? Is it on the moreregulated side or less regulated
(05:03):
side in terms of government and,I guess, regulation in terms of
security or safety, privacy,these sorts of things?
Rick (05:13):
I think Japan is in the middle between EU and The US
situation. So they say that it'sa soft approach. So there is no
strict regulation in Japan, butthere are lots of kind of
guidance in Japan. And peoplehave already noticed some of the
(05:34):
issues on the security side orsafety side of AI. And on top of
that, they are trying tointroduce LLM applications for
contact center applications.
So they are concerned about morereputational risks rather than
security risks. So in essence,they understood the importance
(05:58):
of the safety of the AI ortrustworthy of the AI, but more
concerned about reputationalrisks rather than security
risks.
Chris (06:08):
I'm curious as we're kind of talking about the adoption
and the environment around that,as you kind of mentioned that in
Japan, it's very you know,people are really getting into
the utilization of LLMs andgenerative AI. Do you have any
any kind of thoughts around, youknow, what is it what is it
that's driving that, you know,compared to other countries that
(06:31):
you've observed? If you'relooking at that kind of adoption
in Japan versus The US or versusEurope or whatever. Any thoughts
around that? Because I I had Ihad observed that as as someone
in The US that Japan seemed tobe implementing, and I was kinda
curious if there was you know,what was the what was kind of
the driving force there thatthat got the utilization up,
(06:52):
especially among just yourtypical average everyday folks
there, not outside of the AIindustry?
I would love your thoughts aboutthat.
Rick (07:01):
I'm not so sure the the background reasons why Japan is
more aggressive to introducesuch a kind of L and M
applications. Probably as Isaid, people have lower, say,
risk of perception, I must say.In some cases, in Japan, like a
chatbot or kind of, say, newtechnology might be a kind of
(07:25):
friend in a sense. Sure. In USside, such a kind of a new AI is
kind of enemy of a human.
Chris (07:33):
Yeah. So they're kind of a a little bit of a cultural
difference in terms of how thatthat openness to adoption and
stuff. So, yeah, that would makesense. That would make sense to
me.
Daniel (07:43):
Yeah. And I guess sometimes your your friends can
potentially hurt you even ifthey even if they're not not
trying, which I know, you youall are are kind of involved in
both, you know, the evaluationof these systems, you know,
running metrics against thesesystems and kind of building
trust, in in a very real way.Kenny, I'm wondering if we bring
(08:06):
you in here and and maybe just,you know, safety and security
around AI is such a broad topicnow with so many people
addressing it from so manydifferent perspectives. I'm
wondering if you could help uszero in on maybe the kinds of
problems that you all areexploring and how those fit kind
(08:29):
of more generally into thelandscape of, I guess, security
risks or threats as related toAI.
Kenny (08:36):
Definitely. Yeah. So thank you for having us on the
podcast. My name is Kenny. I'mthe co founder and CTO of
Citadel So what we do at Citadelis we build software tools to
help organizations test, monitorand govern their AI systems.
And in the world of LLMs, whencustomers come to us, they
usually have some proof ofconcept that they've been
(09:00):
developing internally. They'vebeen using some foundation
model, they're building somechatbot or some agentic
workflow, and they come to uswhen they want to make that
proof of concept productionready. Usually their main
problem is they want to mitigatesome of the risks that they see.
Things like hallucinations,toxicity, or users trying to
(09:21):
prompt inject the system.They're looking for a solution
to these types of problems.
That's where we come in as atool provider.
Daniel (09:30):
Yeah, and how does that, I guess, how are you seeing when
you're interacting with yourcustomers or they're coming to
you with these problems? What isthe impact of those problems
like hallucination or injection?Is that something that is
causing real problems orsomething that they've sort of
heard about that they'reconcerned about, but it's maybe
(09:53):
not causing real problem? Whatare you seeing there in terms
of, I guess, the impact? Let'ssay I just want to throw caution
to the wind and ignore thesethings.
What's the bad side of this thatcould happen were I to take that
more loose approach? Anythoughts?
Kenny (10:13):
Yeah, I think it's usually a mix of both. It also
depends on the risk appetite ofthe company that's developing
the system. And typically, ourcustomers are larger enterprise
companies, both inside of Japanand outside of Japan. And so
they have very mature riskmanagement practices. And before
(10:34):
they, you know, launch thesePOCs into a production service,
whether that's internal facingor external facing, they want to
make sure that they, you know,have appropriate controls in
place to manage the risk andthey've properly identified the
potential risks, reputationaldata security, so on.
(10:54):
So yeah, I think for thecustomers we talk to, it's
generally a big concern forthem. And they come to us with
the problem of mitigating someof these risks that they've
already identified.
Chris (11:07):
I'm curious, as you've kind of identified, you know,
what that kind of the strata ofcustomers that you're looking at
there, What what is kind of isthere is there something that
definitively kind of separatesthat kind of mature larger
organization from some of thesmaller ones? Do they have a
different set of problems thatthey're coping with or maybe
(11:30):
just haven't gotten far enoughalong in terms of maturity on
risk management? What like, atwhat point do you see the uptake
kind of falling off within maybesmaller organizations? You know,
what's what does that look likeas you as you move from that
upper strata into the mid tier?
Kenny (11:48):
I think my my general perspective is that there's two
ends of the spectrum of likestartups to enterprise company.
And for startups, the risk isfairly low because you don't
have a brand to protect, youdon't have an existing business
to think about. You can justlaunch these POCs out to your
customers very quickly and youcan iterate quickly. For
(12:10):
enterprise customers, they tendto be more guarded. They have a
closer eye on potential risks tothe business.
Those are the customers thatreally want pretty robust
testing before deployment, andalso monitoring and potentially
real time guardrails filteringafter deployment. I think it
(12:32):
depends a bit on the size of thecompany.
Daniel (12:35):
I'm wondering with that, as you're looking to those types
I'm curious a little bit of thebackstory of Citadel. Maybe
Rick, you could tell us a littlebit of kind of how you came to
these problems because you seemlike you have customers now,
you've developed some of thesethings. When did that happen?
(12:57):
How early was that in kind ofWas that before GenAI? Was that
as GenAI was coming up?
How did things develop in termsof your thinking around these
problems and how you would bringsomething to the market in terms
of addressing them?
Rick (13:14):
So in that sense, was before Gen AI. And actually, the
person who came up with such akind of idea is not me, but
Kenny. So Kenny has worked inGoogle brains, and he was one of
the core member of theTensorFlow team. So he's taking
a leadership to develop the mostadvanced AI technologies in
(13:36):
Google and found that thereshould be a lots of risk about
the trustworthiness or safety ofthe AI. And he believes that
probably he is the best personto explain by himself.
But anyway, he reached the ideathat some of the tools which
protect sort of kind of safetyor security issues should become
(14:01):
popular, especially in theenterprise companies where they
may not have a bunch of AIengineers inside. So that is a
background. Our company nameCitadel itself is shown what we
are focusing. So Citadel is, asyou know, saying like a vault or
castle. So we are the company tosay, protect such a kind of
(14:26):
risk, human from AI risks.
Such kind of a concept is say,based on our company name.
Daniel (14:35):
Yeah, anything to add there, Kenny?
Kenny (14:37):
Yeah, that's a pretty good overview of the background.
When we started the company, itwas at the 2020. So it was
before the era of large languagemodels. And We were initially
really focused on helpingorganizations monitor their
traditional predictive AImodels, so tabular models,
vision models, that kind ofthing. About two years ago, we
(14:59):
started getting a lot moreinterest from our customers in
LLMs.
And how do we reliably test thisnew technology? How do we
integrate it into our workflowsand our business applications?
And so these days, I think a lotof our new customers come to us
with LLM types of questions. Andthen we still have a lot of
(15:21):
existing customers that workmore on the predictive AI side.
Chris (15:25):
So Kenny, I'd like to follow-up as you guys were kind
of developing the idea goingback to that a moment for the
company. And Rick mentioned thatyou had been there at Google
Brain and that you were on theTensorFlow team. I'm I'm curious
how as a you know, for me as asa someone who's used TensorFlow
(15:46):
a bit. And so that was like,wow, you know, one of the people
who helped put that together.I'm curious what parts of the
experiences you had there ledinto the formation of Citadel in
your mind is, as you had some ofthe the insights that you might
have developed in your previousemployment.
Did did any of that what youknow, how how did that lead to
(16:09):
what Citadel does? How did thatkind of bring forward when you
and Rick started the company up?
Kenny (16:14):
Sure. So a bit more about my personal background. So in
2017 to 2020, I was working atGoogle Brain as a product
manager. And my team wasresponsible for building machine
learning infrastructure atGoogle. This included TensorFlow
and also other platforms likeTFX, TensorFlow Extended, and
(16:36):
some of the work on GoogleCloud's AI platforms and so on.
So sort of the softwarefoundations that power a lot of
Google's machine learningapplications. Think Google tends
to be pretty ahead of the curvein AI adoption. Think back then
it was more called machinelearning rather than AI. But
basically what we worked on wasmaking models at Google more
(16:59):
reliable at Google productionscale. That usually meant
building these prettysophisticated pipelines that not
only target training the models,but also serving them in
production.
We had pretty robust systems formonitoring data drift and
validating data that entersproduction models, monitors the
(17:20):
output of these models. And, youknow, at Google, you can afford
to have hundreds of platformengineers build out this kind of
infrastructure to useinternally. But for most other
types of organizations, theycan't make that level of
investment in internalplatforms. And we felt that
there was an opportunity to sortof build some of these model
(17:41):
monitoring and data validationtools for other companies. And
so that's where the idea ofCitadel AI kind of started.
And when we started the companyoriginally, we focused a lot on
adversarial attacks, actuallyback in 2020, since it was a
very hot research topic.Basically, designing noise and
(18:04):
images and other types of inputdata to trick models into making
the wrong predictions. We foundthat after a few months, this
wasn't really a problem thatcompanies were interested in.
It's a very interesting researchproblem, but less interesting
commercially. After that, wepivoted towards more
observability and testing ofthese predictive models.
(18:25):
These days we focus a lot moreon LLM testing and monitoring.
Daniel (18:29):
Maybe you could just give us, going back to you,
Rick, maybe you could just giveus a few examples that stand out
of kinds of companies and thekinds of models that they're
running and the way that or someof the things that they would be
interested in tracking ordetecting or observing. A few
(18:50):
concrete examples might help ouraudience kind of grasp some of
the cases that you're workingwith.
Rick (18:56):
One of the largest customers in Japan is the
financial industry,surprisingly. Like banks or
insurance companies or securitycompanies are very aggressive to
introduce LLM applications intotheir core operations, such as
like a contact center orinternal applications. And when
(19:22):
they say they can go through POCstage and when they get into
commercial operations, theystart caring about some of the
kind of safety issues. Sobecause those financial industry
is governed by the governmentitself. So in essence, different
(19:43):
from regulation related AI, theyhave already been kind of
regulated by traditional, sayfinancial regulations.
So if say LM applications behavebadly, that damage their core
businesses. So to introduce sucha kind of AI application is to
(20:07):
differentiate or advance theirservices compared with their
competitors. But by introducingnew technologies, if their core
business is damaged, that wouldbe very, say, huge negative
impact. And also, if AI behavebadly, as I said, the Japanese
(20:31):
government may say, in somecases, call or punish them. So
that is one of the most largestrisks for them to introduce L
and M generic applications.
So what we are doing is tomaking sure that such kind of a
bad behavior will not happen inthe application or not, and
(20:52):
making sure that they can safelyintroduce Sego into commercial
operation. So some of theleading the bank company or
insurance company is our say ourcustomers.
Chris (21:05):
Gotcha. I'm curious if I could follow-up on that for a
second. We've kind of talkedabout the the notion of risk
management in general that thatindustry is is dealing with. How
do they given the risks for alarge, you know, organization to
implement LLMs? Do you do youhave any insight into how
(21:26):
they're making the the riskmanagement judgment on the
benefits of implementing maybe anew chatbot versus the potential
downside where, you know, as youpointed out, could be damaged to
brand, damage to coreoperations, damage from a
regulatory standpoint.
That's as I'm listening to you,that does sound very risky, you
know, from that perspective. Howdo they make how do they
(21:49):
evaluate that? Like, say thatthey know that they wanna come
to Citadel and get that help.What do you have a sense from
your customers what a typicalbalance on that is of the
benefit of the LLM utilizationversus the downside if things go
off? How do they what are theythinking when they come to you
in that way?
Rick (22:07):
So in that sense, they fully understand that balancing
is very important. So to protectthe risk so much may say delay
the service advancement. On theother hand, say, without having
any security or safety test,they may get into bad situation.
(22:30):
So they need to know how tobalance such a kind of risks and
benefit. And they have set upseveral, like, internal
organization to manage such akind of risk insight.
So, yeah, as you can easilyimagine, financial industries
they are very structuredcompany. So there are some of
(22:53):
like risk management departmentinside. And in some cases, are
like AI governance team insideor something like that. So they
jointly work together and try tomanage such a kind of a risk.
And at the same time, they tryto introduce the advanced system
so that they can differentiatethemselves from others.
(23:16):
And it's true as for the contactcenter, probably the same
situation in The US. But whenthe consumer end user try to
reach out the banks or insurancecompany and call it free dial,
In many cases, they can't easilyreach to the contact center
(23:37):
person. They have to wait thirtyminutes or something like that.
So because of that situation,they are so aggressive introduce
such kind of applications intocontact center and internal
purposes.
Daniel (23:51):
And I'm just thinking about this scenario where you
have the contact center, you'reintroducing whatever it is, a
chatbot, a voice assistant, thatsort of thing. Then I'm thinking
back, to what Kenny was talkingabout kind of how the company
started evaluating models thatwere not GenAI models yet. I'm
(24:12):
wondering, Kenny, if you couldhelp us think about you know,
what needs because if I'munderstanding, you know, part of
what you all are doing, part ofit is evaluating the risks of a
particular model or system, andpart of it is observing those
and monitoring those in realtime. And if I think about a
(24:34):
traditional model, let's say amodel that detects tumors in
medical imagery, right? I canhave a very nice sort of ground
truth dataset and Maybe it'shard to get because there's some
privacy concerns, but I canstill get it.
I need to get it to train mymodel. I have very specific
(24:54):
metrics around whatever it is,accuracy, F1 score, etcetera,
can sort of grasp what theperformance of that model is,
maybe even compare it to humanperformance. With something like
a call center, in some wayspeople might struggle to connect
that to real metrics that makesense, right? Because it's like,
(25:18):
Oh, well, people could sayanything to the chatbot. How do
I know, one, what's going tocome in either from a normal
usage or malicious usage orwhatever, and how do I connect
that to any sort of metricaround a model?
I think sometimes peoplestruggle with this idea of
metrics and Gen AI models or GenAI systems. Could you help maybe
(25:43):
clarify, what are some of therelevant metrics that people
could think about in terms ofthese systems that might help
them understand how the systemsare behaving?
Kenny (25:55):
Sure. Yeah, that's a very spot on question. I guess before
I talk about specific metrics,I'll just take a step back
first. If we sort of think at ahigh level, what is the same
between predictive AI andgenerative AI? I think the
structure of how you maintainreliability is basically the
same, right?
You need testing beforedeployment, and you need
(26:16):
monitoring after deployment. Andit's also very similar to like
traditional softwareapplications, right, where you
have automated tests andautomated monitoring. And so I
think that part is the same. Butthe part that's much trickier
for generative AI is thatusually, as you mentioned, you
don't have ground truth in thesame way that you do for a
(26:38):
classification data set, forexample. And so the metrics that
you use for evaluation are notas well defined.
So you can't measure accuracy,you can't measure position or
recall. And the output of agenerative AI model is also much
more complex than just like aprobability score. And so in
(26:58):
that environment, it's very hardto determine how do we actually
evaluate these things in aquantitative and objective way.
So the approach that most of ourcustomers take and most of the
industry has gone in isbasically using LLM as a judge.
So you can craft theseevaluation prompts that ask an
(27:21):
LLM to evaluate some quality ofsome generated text.
So it could be, you know, a verysimple example is sentiment. So
you evaluate the sentiment of,of some text, you could do that
with traditional, like asentiment classifier as well.
But there are more sophisticatedmetrics such as, you know,
(27:42):
detecting hallucinations againstsome ground truth document, or
measuring the relevance of theanswer relative to the question.
Or you might have more, we callthem custom metrics that are
designed to be domain specific.So if you have like a refund
chatbot, you can design a metricthat measures if the chatbot
(28:05):
adheres to your company's refundpolicy.
And so these metrics, they'revery flexible, because you can
design the evaluation prompt innatural language. In our tools
and our open source libraries,we have a set of built in
metrics. It's like a library ofmetrics you can choose from. But
for many of our customers, theyalso extend those built in
(28:26):
metrics to customize them to fittheir business applications.
Daniel (28:30):
So Kenny, you were mentioning this sort of idea of
LM as a judge, which is using amodel to evaluate the model in
some sort of axis of performanceor some quality, which
definitely seems like a flexibleoption. But also some people
(28:54):
might be thrown off by this sortof circular using a model to
evaluate a model. Also, there'ssort of this you then have a
model that's evaluating themodel, so how do you evaluate
the model that evaluates themodel? You kind of get in this.
How have you all navigated thatside of things, both in terms of
(29:16):
using the larger model, makingsure that the evaluations are
sound and also transferable, onemodel to the other, and maybe
benchmarking the system overtime?
Because also the models youmight want to use as evaluators
might change over time.
Kenny (29:34):
Yeah, also a very good question. It's a question that
we get from our customers quitea lot as well. And the way that
we generally approach theevaluation workflow, which
includes designing these metricsis not usually human judgment
and taste is treated as the goldstandard. But the problem with
(29:54):
having humans evaluate everyexperiment with your LLM system
is that it's very expensive andit's very slow. In the ideal
world, you would design theseLLM as a judge metrics that can
mimic human preferences.
And so in our software tooling,this is what we design specific
workflows to help users to do.Usually when they when a
(30:16):
customer starts on a valuationproject, they'll, you know, of
course, think about theevaluation criteria that's
important. But then they'll alsohave humans do a small set of
that evaluation. So maybe like50 to 100 of these manual
annotations. And from there, youcan design LLM automated metrics
(30:37):
and measure their correlationand accuracy against the human
judgment.
You usually need to iterate afew times to get that custom LLM
metric as close as possible tothe human judgment. But then
once you have that, it's verypowerful, right? You have this
automated metric that is a verygood proxy for human judgment
and it's automated, which meansyou can run it at scale, you can
(31:00):
deploy it during evaluation, butalso in monitoring as well, and
you can also potentially usethat as a production guardrail
in our firewall.
Chris (31:09):
Rick, I was wondering, I know in you know, we've kind of
alluded to it that, you know,the the two sides of the
equation in terms of testing themodels and then monitoring.
Could you talk a little aboutCitadel Lens and Citadel Radar?
How you bring them to customersand what the relationship is
between those two products thatyou're bringing to your
(31:32):
customers and you know, how doyou how do you go to how do you
present them when when somebodyis interested in, in being able
to bring this level of securityto the models that they're
interested in?
Rick (31:44):
Sure. First of all, now we are merging radar features into
lens. Lens can provide bothtesting function and the
monitoring function together atthis stage. And, as for the the
balancing or difference betweentesting and monitoring, as Kenny
mentioned, especially in thecase of LLM, how to customize
(32:08):
human, say, say, setting back.So so our system is in the
concept that the system shouldfollow the human rather than
human has to follow the system.
In essence, we set the humanannotation or human judge as a
(32:30):
first priority and try to ourmetrics to be customized to the
human judge. So that is a veryimportant point. And to make it
happen, we need to or thecustomer need to go through
testing phase first so that allthe metrics should be aligned
(32:53):
with the human judgment so thatand we can make use of the same
custom metrics during the, say,the monitoring phase or fire
firewall stage. So in essence,even though final goal might be
a monitoring or firewall, butthe two before going to that
(33:13):
stage, how to test and customizea metrics is very important I
mean, very critical, to protectthe safety and security and
reputational risks. So werecommend strongly recommend to
start from testing phase first.
So testing phase is not justtesting, but customize your
(33:34):
metrics into your, say,professional, the person's. So
that is a testing phase. Andafter that, they can go into
monitoring phase. So that is ourapproach to the customers.
Daniel (33:47):
And I'm wondering, either one of you could answer
this, but why is it and this maybe obvious to maybe more of the
software engineering type crowd,but maybe less so to some others
outside of that crowd. Why is itimportant to once you've tested
your model to actually monitorit online for maybe the things
(34:10):
that are are potentiallyproblematic inputs, whether
that's a security thing like aprompt injection or, maybe
something outside of the youknow, some type of input that
you wanna filter out like IPgoing in or something that
doesn't fit your policy. Why isit important to have that
(34:31):
monitoring piece and not justthe testing piece? Because if I
test my model and I convincemyself that it can't be prompt
injected, which I'm saying thatsort of in jest because, you
know, as you all know, everymodel there's no perfectly
aligned model. There's no everymodel is vulnerable to various
things.
But let's say that I convincemyself of high performance in in
(34:55):
one of these areas. Why then isit useful and necessary then to
monitor that over time or inreal time?
Rick (35:04):
Technically, probably Kenny is also again the best
person. But even if thecustomers can go through the
testing phase, the marketcondition or say, the human
reaction may change over time.So if we do are safe right now,
(35:25):
but if anything, say somethingnew happens today, what we say
guaranteed today may not applytomorrow. So it's a very general
things. But say, in that sense,keep monitoring is very
important to protect ourcustomers, even if the market
(35:46):
condition or the world conditionchange or economic condition
change.
Kenny (35:51):
Yeah, and just to give a concrete example of why you may
want monitoring, so we reallyview them as complementary, and
you really need both, if youwant to make a system reliable.
So for example, if you have likean answer quality metric that
measures how high quality andanswers, you should of course
(36:12):
use that for testing to makesure that it meets some like,
you know, 8090% bar. But then inmonitoring, you actually want to
measure quality of the realanswers that your chatbot is
giving to real customers, right.So from a quality perspective,
it makes a lot of sense fromlike a safety and risk reduction
perspective. Another example isthat, you know, as you
(36:32):
mentioned, Dan, during testing,you might test a bunch of prompt
injections against your system.
But then in deployment, havereal users, some of them are
adversarial, some of them areactually trying to prompt
inject. They may do it increative ways that you haven't
tested before. You may want someguardrail that will
automatically detect thoseattempts and filter them out,
(36:55):
even if you're sure that themodel is robust to 90% of these
attacks.
Chris (37:00):
I'm curious, you guys have some open source available
out there. I know one of thetools is lang check. Can you
talk a little bit about yourapproach to open source?
Kenny (37:11):
For context, lang check is our open source Python
library that contains a suite ofmetrics that are built in that
you can use for evaluating thequality of text. One of our
motivations for creating thislibrary is that I think we
launched it in October 2023,roughly. And around that time,
(37:34):
there weren't a lot of sort ofindustry standard metrics and
practices for evaluating text,particularly in non English
languages as well. So there wassome focus on, you know, these
metrics in English, but we workwith a lot of customers that
have Japanese texts or Chineseor German and these other
languages. We wanted to make alibrary of these metrics that
(37:57):
anyone can use.
We view this as a pretty goodstarting point. If you just need
one or two metrics, you'recomfortable writing code, you
can use LangeCheck and integratethat into your test pipeline or
your monitoring system. But thenif you want something production
scale, and you want an easyworkflow to design custom
metrics and test them againstmanual annotations, that's where
(38:21):
our commercial product comes in.
Daniel (38:23):
Makes sense, yeah. And as we get a little bit closer to
the end here, there's so manythings and this is such a, so
many in-depth areas to go in,which is why it's great that
there's wonderful people likeyourselves exploring, the topic.
But I'm wondering, if we couldmaybe talk just a little bit as
(38:45):
we close out here about whatyou're excited about, about kind
of, yes, Citadel, but maybe thekind of general ecosystem that
that you're a part of. As as youlook to the future, you know,
what's what's exciting to eachof you about how the ecosystem
is developing? What's becomingpossible with the technology?
(39:05):
What's inspiring to you or whatare you thinking about in terms
of the future? Maybe I'll startwith you, Rick.
Rick (39:13):
Okay. So now the chat GPT-five is released. But when
we say look back today from sayfive years later, I believe
that, oh, this is a verypremature say model or something
like that. So in that sense, thetechnology advancement in the AI
(39:35):
field is so rapid. And in thatsense, yeah, there may be a lot
of risks coming in.
But on the other hand, there aremostly infinite opportunities.
So you can find that hugevariety of possibilities, not
only just say the AI directlyrelated technologies, but I
(40:01):
believe some of the materialproducts or machines or anything
will change maybe within five orten years. So in essence, we are
in the midst of the say periodwhere anything can change in
that sense, especiallytechnology related. So there are
(40:23):
lots of say risks may come inand we like to protect such a
kind of risk as a company. Buton the other hand, people can
find many possibilities,opportunities for you to try.
So I strongly believe that soeven though there are lots of
issues in the world that peoplecan enjoy or say, make best use
(40:48):
of this opportunity, everybody.
Daniel (40:51):
Yeah, that's great. What about yourself, Kenny?
Kenny (40:54):
Yeah, I think as a consumer of AI in both my
personal life and work, from aconsumer perspective, it's
really exciting to benefit fromall the advancements in these
new AI tools and models. Ireally loved Oh, three as a
model in chat GBT. And I loveusing cursor. I'm excited for
(41:14):
these tools to become more andmore agentic over time. I think
that's the trend that you see.
If you just look at ChatGPT,originally it was just 3.5 and
GPT-four that just answer aquestion based on forward pass
of the model. But now thesemodels will search the internet
and it'll sort of reason andthink about what to search next.
(41:35):
As a result, the outputs havebecome a lot better. So really
excited for that to improve evenmore from a consumer
perspective. And then from abusiness perspective, I'm really
excited to help bring thesecapabilities to our business
customers and help them use AImore reliably and more
effectively in in theirbusiness.
Daniel (41:56):
That's great. Yeah. Well, thank you both for taking
time to to chat with us today.And thank you both for the work
and thought that you're puttinginto the tools that you're
building and the open sourceprojects that you're putting out
there. It's a great benefit tothe to the community and to the
business world, of course.
So thank you for the work thatyou're doing. And, yeah, we'll
(42:19):
look forward to, to keepingkeeping, an eye on on what you,
what you evaluate and andprotect us from next. So,
appreciate you both, hope youhave a great evening. Thank you
for joining.
Rick (42:32):
Thank you very much.
Kenny (42:34):
Thank you for the conversation.
Jerod (42:42):
Alright. That's our show for this week. If you haven't
checked out our website, head topracticalai.fm, and be sure to
connect with us on LinkedIn, X,or Blue Sky. You'll see us
posting insights related to thelatest AI developments, and we
would love for you to join theconversation. Thanks to our
partner Prediction Guard forproviding operational support
for the show.
Check them out atpredictionguard.com. Also,
(43:05):
thanks to Breakmaster Cylinderfor the beats and to you for
listening. That's all for now,but you'll hear from us again
next week.
Welcome to the Practical AI podcast, where we break down
the real world applications ofartificial intelligence and how
it's shaping the way we live,work, and create. Our goal is to
help make AI technologypractical, productive, and
accessible to everyone. Whetheryou're a developer, business
leader, or just curious aboutthe tech behind the buzz, you're
(00:24):
in the right place. Be sure toconnect with us on LinkedIn, X,
or Blue Sky to stay up to datewith episode drops, behind the
scenes content, and AI insights.You can learn more at
practicalai.fm.
Now, onto the show.
Daniel (00:49):
Welcome to another episode of the Practical AI
Podcast. This is DanielWitenack. I am CEO at Prediction
Guard, and I am joined as alwaysby my cohost, Chris Benson, who
is a principal AI researchengineer at Lockheed Martin. How
are doing, Chris?
Chris (01:04):
Hey. Doing great today, Daniel. How's it going with you?
Daniel (01:07):
It's going great. I traveled a bit over the weekend
to run a mini marathon, whichwas fun, but I am back safe at
home. And, of course, safety isis something that hopefully we
can talk a little bit abouttoday with our with our guests
who are, Rick Kobayashi, who iscofounder and CEO at Citadel AI,
(01:28):
and Kenny Song, who is cofounderand CTO at Citadel AI. Welcome.
Kenny (01:34):
Thank you for having us.
Rick (01:35):
Yeah. Thank you, Hwal.
Daniel (01:36):
Yeah. It's great to have you both here. Good to get
introduced to you both. I'm I'mreally excited about this
conversation because, of course,security and safety related to
AI is very close to my own work,so it's always great to connect
with people in this space. Yeah,I'm also interested to hear a
(01:58):
little bit, and I even saw somethings on your website, you
know, talking a little bit aboutsome of the AI in Japan, some of
the, I guess, regulations orguidelines for for businesses
that have come out there.
So maybe before we get into thenature of what you're building
and what you're doing, anythoughts for those out there
(02:21):
that might be in The US or inEurope and be constantly exposed
to the things that are going onin AI in those jurisdictions,
Any thoughts about what is thesame or what stands out about
what's happening with AI inJapan?
Rick (02:37):
So, yeah, anyway, thank you very much for joining us. My
name is Rick. And as for theJapanese conditions, basically,
I'm afraid to say it's aroundone, two years behind The US
situation, basically, up to now.But on the other hand,
surprisingly, as for Gen AI, Ithink that Japan is one of the
(03:00):
most advanced countries. Not thedevelopment of the foundation
models, the utilization of theGen AI applications.
And probably it might be relatedto kind of Japanese animation or
cartoon. So, you know, likethere are lots of say robot or
(03:21):
some of the say animations whichspecify some of the AI or robot
technologies and, you know,Doraemon or there are lots of
animation over there. And peopleare very getting familiar with
such a kind of talking withrobot or some of these advanced
computers from the childhood. Soin that sense, I think that many
(03:45):
people do not have anyhesitation to work with or talk
with like a chatbot or anyGenAI. So in that sense, the
hurdle to introduce LNM or GenAIin Japan is very low in that
compared with US or othercountries.
So in essence, development sideor technology side, frankly
(04:05):
speaking, Japan is behind The USsituation. But as for the usage
of such kind of a generic, Ithink, I hope Japan is one of
the most advanced countries.
Daniel (04:17):
Yeah. That's really interesting to hear the sort of
perception side of things. Andof course with more, maybe even
more adoption or adoption that'sahead of maybe other places in
the world, it could be thatusers or usage of the technology
(04:39):
has hit some bumps along theroad or has hit some problems,
which I know is a lot of whatyou all are working on. What has
been the situation in Japanaround regulation and business
usage of AI? Is it on the moreregulated side or less regulated
(05:03):
side in terms of government and,I guess, regulation in terms of
security or safety, privacy,these sorts of things?
Rick (05:13):
I think Japan is in the middle between EU and The US
situation. So they say that it'sa soft approach. So there is no
strict regulation in Japan, butthere are lots of kind of
guidance in Japan. And peoplehave already noticed some of the
(05:34):
issues on the security side orsafety side of AI. And on top of
that, they are trying tointroduce LLM applications for
contact center applications.
So they are concerned about morereputational risks rather than
security risks. So in essence,they understood the importance
(05:58):
of the safety of the AI ortrustworthy of the AI, but more
concerned about reputationalrisks rather than security
risks.
Chris (06:08):
I'm curious as we're kind of talking about the adoption
and the environment around that,as you kind of mentioned that in
Japan, it's very you know,people are really getting into
the utilization of LLMs andgenerative AI. Do you have any
any kind of thoughts around, youknow, what is it what is it
that's driving that, you know,compared to other countries that
(06:31):
you've observed? If you'relooking at that kind of adoption
in Japan versus The US or versusEurope or whatever. Any thoughts
around that? Because I I had Ihad observed that as as someone
in The US that Japan seemed tobe implementing, and I was kinda
curious if there was you know,what was the what was kind of
the driving force there thatthat got the utilization up,
(06:52):
especially among just yourtypical average everyday folks
there, not outside of the AIindustry?
I would love your thoughts aboutthat.
Rick (07:01):
I'm not so sure the the background reasons why Japan is
more aggressive to introducesuch a kind of L and M
applications. Probably as Isaid, people have lower, say,
risk of perception, I must say.In some cases, in Japan, like a
chatbot or kind of, say, newtechnology might be a kind of
(07:25):
friend in a sense. Sure. In USside, such a kind of a new AI is
kind of enemy of a human.
Chris (07:33):
Yeah. So they're kind of a a little bit of a cultural
difference in terms of how thatthat openness to adoption and
stuff. So, yeah, that would makesense. That would make sense to
me.
Daniel (07:43):
Yeah. And I guess sometimes your your friends can
potentially hurt you even ifthey even if they're not not
trying, which I know, you youall are are kind of involved in
both, you know, the evaluationof these systems, you know,
running metrics against thesesystems and kind of building
trust, in in a very real way.Kenny, I'm wondering if we bring
(08:06):
you in here and and maybe just,you know, safety and security
around AI is such a broad topicnow with so many people
addressing it from so manydifferent perspectives. I'm
wondering if you could help uszero in on maybe the kinds of
problems that you all areexploring and how those fit kind
(08:29):
of more generally into thelandscape of, I guess, security
risks or threats as related toAI.
Kenny (08:36):
Definitely. Yeah. So thank you for having us on the
podcast. My name is Kenny. I'mthe co founder and CTO of
Citadel So what we do at Citadelis we build software tools to
help organizations test, monitorand govern their AI systems.
And in the world of LLMs, whencustomers come to us, they
usually have some proof ofconcept that they've been
(09:00):
developing internally. They'vebeen using some foundation
model, they're building somechatbot or some agentic
workflow, and they come to uswhen they want to make that
proof of concept productionready. Usually their main
problem is they want to mitigatesome of the risks that they see.
Things like hallucinations,toxicity, or users trying to
(09:21):
prompt inject the system.They're looking for a solution
to these types of problems.
That's where we come in as atool provider.
Daniel (09:30):
Yeah, and how does that, I guess, how are you seeing when
you're interacting with yourcustomers or they're coming to
you with these problems? What isthe impact of those problems
like hallucination or injection?Is that something that is
causing real problems orsomething that they've sort of
heard about that they'reconcerned about, but it's maybe
(09:53):
not causing real problem? Whatare you seeing there in terms
of, I guess, the impact? Let'ssay I just want to throw caution
to the wind and ignore thesethings.
What's the bad side of this thatcould happen were I to take that
more loose approach? Anythoughts?
Kenny (10:13):
Yeah, I think it's usually a mix of both. It also
depends on the risk appetite ofthe company that's developing
the system. And typically, ourcustomers are larger enterprise
companies, both inside of Japanand outside of Japan. And so
they have very mature riskmanagement practices. And before
(10:34):
they, you know, launch thesePOCs into a production service,
whether that's internal facingor external facing, they want to
make sure that they, you know,have appropriate controls in
place to manage the risk andthey've properly identified the
potential risks, reputationaldata security, so on.
(10:54):
So yeah, I think for thecustomers we talk to, it's
generally a big concern forthem. And they come to us with
the problem of mitigating someof these risks that they've
already identified.
Chris (11:07):
I'm curious, as you've kind of identified, you know,
what that kind of the strata ofcustomers that you're looking at
there, What what is kind of isthere is there something that
definitively kind of separatesthat kind of mature larger
organization from some of thesmaller ones? Do they have a
different set of problems thatthey're coping with or maybe
(11:30):
just haven't gotten far enoughalong in terms of maturity on
risk management? What like, atwhat point do you see the uptake
kind of falling off within maybesmaller organizations? You know,
what's what does that look likeas you as you move from that
upper strata into the mid tier?
Kenny (11:48):
I think my my general perspective is that there's two
ends of the spectrum of likestartups to enterprise company.
And for startups, the risk isfairly low because you don't
have a brand to protect, youdon't have an existing business
to think about. You can justlaunch these POCs out to your
customers very quickly and youcan iterate quickly. For
(12:10):
enterprise customers, they tendto be more guarded. They have a
closer eye on potential risks tothe business.
Those are the customers thatreally want pretty robust
testing before deployment, andalso monitoring and potentially
real time guardrails filteringafter deployment. I think it
(12:32):
depends a bit on the size of thecompany.
Daniel (12:35):
I'm wondering with that, as you're looking to those types
I'm curious a little bit of thebackstory of Citadel. Maybe
Rick, you could tell us a littlebit of kind of how you came to
these problems because you seemlike you have customers now,
you've developed some of thesethings. When did that happen?
(12:57):
How early was that in kind ofWas that before GenAI? Was that
as GenAI was coming up?
How did things develop in termsof your thinking around these
problems and how you would bringsomething to the market in terms
of addressing them?
Rick (13:14):
So in that sense, was before Gen AI. And actually, the
person who came up with such akind of idea is not me, but
Kenny. So Kenny has worked inGoogle brains, and he was one of
the core member of theTensorFlow team. So he's taking
a leadership to develop the mostadvanced AI technologies in
(13:36):
Google and found that thereshould be a lots of risk about
the trustworthiness or safety ofthe AI. And he believes that
probably he is the best personto explain by himself.
But anyway, he reached the ideathat some of the tools which
protect sort of kind of safetyor security issues should become
(14:01):
popular, especially in theenterprise companies where they
may not have a bunch of AIengineers inside. So that is a
background. Our company nameCitadel itself is shown what we
are focusing. So Citadel is, asyou know, saying like a vault or
castle. So we are the company tosay, protect such a kind of
(14:26):
risk, human from AI risks.
Such kind of a concept is say,based on our company name.
Daniel (14:35):
Yeah, anything to add there, Kenny?
Kenny (14:37):
Yeah, that's a pretty good overview of the background.
When we started the company, itwas at the 2020. So it was
before the era of large languagemodels. And We were initially
really focused on helpingorganizations monitor their
traditional predictive AImodels, so tabular models,
vision models, that kind ofthing. About two years ago, we
(14:59):
started getting a lot moreinterest from our customers in
LLMs.
And how do we reliably test thisnew technology? How do we
integrate it into our workflowsand our business applications?
And so these days, I think a lotof our new customers come to us
with LLM types of questions. Andthen we still have a lot of
(15:21):
existing customers that workmore on the predictive AI side.
Chris (15:25):
So Kenny, I'd like to follow-up as you guys were kind
of developing the idea goingback to that a moment for the
company. And Rick mentioned thatyou had been there at Google
Brain and that you were on theTensorFlow team. I'm I'm curious
how as a you know, for me as asa someone who's used TensorFlow
(15:46):
a bit. And so that was like,wow, you know, one of the people
who helped put that together.I'm curious what parts of the
experiences you had there ledinto the formation of Citadel in
your mind is, as you had some ofthe the insights that you might
have developed in your previousemployment.
Did did any of that what youknow, how how did that lead to
(16:09):
what Citadel does? How did thatkind of bring forward when you
and Rick started the company up?
Kenny (16:14):
Sure. So a bit more about my personal background. So in
2017 to 2020, I was working atGoogle Brain as a product
manager. And my team wasresponsible for building machine
learning infrastructure atGoogle. This included TensorFlow
and also other platforms likeTFX, TensorFlow Extended, and
(16:36):
some of the work on GoogleCloud's AI platforms and so on.
So sort of the softwarefoundations that power a lot of
Google's machine learningapplications. Think Google tends
to be pretty ahead of the curvein AI adoption. Think back then
it was more called machinelearning rather than AI. But
basically what we worked on wasmaking models at Google more
(16:59):
reliable at Google productionscale. That usually meant
building these prettysophisticated pipelines that not
only target training the models,but also serving them in
production.
We had pretty robust systems formonitoring data drift and
validating data that entersproduction models, monitors the
(17:20):
output of these models. And, youknow, at Google, you can afford
to have hundreds of platformengineers build out this kind of
infrastructure to useinternally. But for most other
types of organizations, theycan't make that level of
investment in internalplatforms. And we felt that
there was an opportunity to sortof build some of these model
(17:41):
monitoring and data validationtools for other companies. And
so that's where the idea ofCitadel AI kind of started.
And when we started the companyoriginally, we focused a lot on
adversarial attacks, actuallyback in 2020, since it was a
very hot research topic.Basically, designing noise and
(18:04):
images and other types of inputdata to trick models into making
the wrong predictions. We foundthat after a few months, this
wasn't really a problem thatcompanies were interested in.
It's a very interesting researchproblem, but less interesting
commercially. After that, wepivoted towards more
observability and testing ofthese predictive models.
(18:25):
These days we focus a lot moreon LLM testing and monitoring.
Daniel (18:29):
Maybe you could just give us, going back to you,
Rick, maybe you could just giveus a few examples that stand out
of kinds of companies and thekinds of models that they're
running and the way that or someof the things that they would be
interested in tracking ordetecting or observing. A few
(18:50):
concrete examples might help ouraudience kind of grasp some of
the cases that you're workingwith.
Rick (18:56):
One of the largest customers in Japan is the
financial industry,surprisingly. Like banks or
insurance companies or securitycompanies are very aggressive to
introduce LLM applications intotheir core operations, such as
like a contact center orinternal applications. And when
(19:22):
they say they can go through POCstage and when they get into
commercial operations, theystart caring about some of the
kind of safety issues. Sobecause those financial industry
is governed by the governmentitself. So in essence, different
(19:43):
from regulation related AI, theyhave already been kind of
regulated by traditional, sayfinancial regulations.
So if say LM applications behavebadly, that damage their core
businesses. So to introduce sucha kind of AI application is to
(20:07):
differentiate or advance theirservices compared with their
competitors. But by introducingnew technologies, if their core
business is damaged, that wouldbe very, say, huge negative
impact. And also, if AI behavebadly, as I said, the Japanese
(20:31):
government may say, in somecases, call or punish them. So
that is one of the most largestrisks for them to introduce L
and M generic applications.
So what we are doing is tomaking sure that such kind of a
bad behavior will not happen inthe application or not, and
(20:52):
making sure that they can safelyintroduce Sego into commercial
operation. So some of theleading the bank company or
insurance company is our say ourcustomers.
Chris (21:05):
Gotcha. I'm curious if I could follow-up on that for a
second. We've kind of talkedabout the the notion of risk
management in general that thatindustry is is dealing with. How
do they given the risks for alarge, you know, organization to
implement LLMs? Do you do youhave any insight into how
(21:26):
they're making the the riskmanagement judgment on the
benefits of implementing maybe anew chatbot versus the potential
downside where, you know, as youpointed out, could be damaged to
brand, damage to coreoperations, damage from a
regulatory standpoint.
That's as I'm listening to you,that does sound very risky, you
know, from that perspective. Howdo they make how do they
(21:49):
evaluate that? Like, say thatthey know that they wanna come
to Citadel and get that help.What do you have a sense from
your customers what a typicalbalance on that is of the
benefit of the LLM utilizationversus the downside if things go
off? How do they what are theythinking when they come to you
in that way?
Rick (22:07):
So in that sense, they fully understand that balancing
is very important. So to protectthe risk so much may say delay
the service advancement. On theother hand, say, without having
any security or safety test,they may get into bad situation.
(22:30):
So they need to know how tobalance such a kind of risks and
benefit. And they have set upseveral, like, internal
organization to manage such akind of risk insight.
So, yeah, as you can easilyimagine, financial industries
they are very structuredcompany. So there are some of
(22:53):
like risk management departmentinside. And in some cases, are
like AI governance team insideor something like that. So they
jointly work together and try tomanage such a kind of a risk.
And at the same time, they tryto introduce the advanced system
so that they can differentiatethemselves from others.
(23:16):
And it's true as for the contactcenter, probably the same
situation in The US. But whenthe consumer end user try to
reach out the banks or insurancecompany and call it free dial,
In many cases, they can't easilyreach to the contact center
(23:37):
person. They have to wait thirtyminutes or something like that.
So because of that situation,they are so aggressive introduce
such kind of applications intocontact center and internal
purposes.
Daniel (23:51):
And I'm just thinking about this scenario where you
have the contact center, you'reintroducing whatever it is, a
chatbot, a voice assistant, thatsort of thing. Then I'm thinking
back, to what Kenny was talkingabout kind of how the company
started evaluating models thatwere not GenAI models yet. I'm
(24:12):
wondering, Kenny, if you couldhelp us think about you know,
what needs because if I'munderstanding, you know, part of
what you all are doing, part ofit is evaluating the risks of a
particular model or system, andpart of it is observing those
and monitoring those in realtime. And if I think about a
(24:34):
traditional model, let's say amodel that detects tumors in
medical imagery, right? I canhave a very nice sort of ground
truth dataset and Maybe it'shard to get because there's some
privacy concerns, but I canstill get it.
I need to get it to train mymodel. I have very specific
(24:54):
metrics around whatever it is,accuracy, F1 score, etcetera,
can sort of grasp what theperformance of that model is,
maybe even compare it to humanperformance. With something like
a call center, in some wayspeople might struggle to connect
that to real metrics that makesense, right? Because it's like,
(25:18):
Oh, well, people could sayanything to the chatbot. How do
I know, one, what's going tocome in either from a normal
usage or malicious usage orwhatever, and how do I connect
that to any sort of metricaround a model?
I think sometimes peoplestruggle with this idea of
metrics and Gen AI models or GenAI systems. Could you help maybe
(25:43):
clarify, what are some of therelevant metrics that people
could think about in terms ofthese systems that might help
them understand how the systemsare behaving?
Kenny (25:55):
Sure. Yeah, that's a very spot on question. I guess before
I talk about specific metrics,I'll just take a step back
first. If we sort of think at ahigh level, what is the same
between predictive AI andgenerative AI? I think the
structure of how you maintainreliability is basically the
same, right?
You need testing beforedeployment, and you need
(26:16):
monitoring after deployment. Andit's also very similar to like
traditional softwareapplications, right, where you
have automated tests andautomated monitoring. And so I
think that part is the same. Butthe part that's much trickier
for generative AI is thatusually, as you mentioned, you
don't have ground truth in thesame way that you do for a
(26:38):
classification data set, forexample. And so the metrics that
you use for evaluation are notas well defined.
So you can't measure accuracy,you can't measure position or
recall. And the output of agenerative AI model is also much
more complex than just like aprobability score. And so in
(26:58):
that environment, it's very hardto determine how do we actually
evaluate these things in aquantitative and objective way.
So the approach that most of ourcustomers take and most of the
industry has gone in isbasically using LLM as a judge.
So you can craft theseevaluation prompts that ask an
(27:21):
LLM to evaluate some quality ofsome generated text.
So it could be, you know, a verysimple example is sentiment. So
you evaluate the sentiment of,of some text, you could do that
with traditional, like asentiment classifier as well.
But there are more sophisticatedmetrics such as, you know,
(27:42):
detecting hallucinations againstsome ground truth document, or
measuring the relevance of theanswer relative to the question.
Or you might have more, we callthem custom metrics that are
designed to be domain specific.So if you have like a refund
chatbot, you can design a metricthat measures if the chatbot
(28:05):
adheres to your company's refundpolicy.
And so these metrics, they'revery flexible, because you can
design the evaluation prompt innatural language. In our tools
and our open source libraries,we have a set of built in
metrics. It's like a library ofmetrics you can choose from. But
for many of our customers, theyalso extend those built in
(28:26):
metrics to customize them to fittheir business applications.
Daniel (28:30):
So Kenny, you were mentioning this sort of idea of
LM as a judge, which is using amodel to evaluate the model in
some sort of axis of performanceor some quality, which
definitely seems like a flexibleoption. But also some people
(28:54):
might be thrown off by this sortof circular using a model to
evaluate a model. Also, there'ssort of this you then have a
model that's evaluating themodel, so how do you evaluate
the model that evaluates themodel? You kind of get in this.
How have you all navigated thatside of things, both in terms of
(29:16):
using the larger model, makingsure that the evaluations are
sound and also transferable, onemodel to the other, and maybe
benchmarking the system overtime?
Because also the models youmight want to use as evaluators
might change over time.
Kenny (29:34):
Yeah, also a very good question. It's a question that
we get from our customers quitea lot as well. And the way that
we generally approach theevaluation workflow, which
includes designing these metricsis not usually human judgment
and taste is treated as the goldstandard. But the problem with
(29:54):
having humans evaluate everyexperiment with your LLM system
is that it's very expensive andit's very slow. In the ideal
world, you would design theseLLM as a judge metrics that can
mimic human preferences.
And so in our software tooling,this is what we design specific
workflows to help users to do.Usually when they when a
(30:16):
customer starts on a valuationproject, they'll, you know, of
course, think about theevaluation criteria that's
important. But then they'll alsohave humans do a small set of
that evaluation. So maybe like50 to 100 of these manual
annotations. And from there, youcan design LLM automated metrics
(30:37):
and measure their correlationand accuracy against the human
judgment.
You usually need to iterate afew times to get that custom LLM
metric as close as possible tothe human judgment. But then
once you have that, it's verypowerful, right? You have this
automated metric that is a verygood proxy for human judgment
and it's automated, which meansyou can run it at scale, you can
(31:00):
deploy it during evaluation, butalso in monitoring as well, and
you can also potentially usethat as a production guardrail
in our firewall.
Chris (31:09):
Rick, I was wondering, I know in you know, we've kind of
alluded to it that, you know,the the two sides of the
equation in terms of testing themodels and then monitoring.
Could you talk a little aboutCitadel Lens and Citadel Radar?
How you bring them to customersand what the relationship is
between those two products thatyou're bringing to your
(31:32):
customers and you know, how doyou how do you go to how do you
present them when when somebodyis interested in, in being able
to bring this level of securityto the models that they're
interested in?
Rick (31:44):
Sure. First of all, now we are merging radar features into
lens. Lens can provide bothtesting function and the
monitoring function together atthis stage. And, as for the the
balancing or difference betweentesting and monitoring, as Kenny
mentioned, especially in thecase of LLM, how to customize
(32:08):
human, say, say, setting back.So so our system is in the
concept that the system shouldfollow the human rather than
human has to follow the system.
In essence, we set the humanannotation or human judge as a
(32:30):
first priority and try to ourmetrics to be customized to the
human judge. So that is a veryimportant point. And to make it
happen, we need to or thecustomer need to go through
testing phase first so that allthe metrics should be aligned
(32:53):
with the human judgment so thatand we can make use of the same
custom metrics during the, say,the monitoring phase or fire
firewall stage. So in essence,even though final goal might be
a monitoring or firewall, butthe two before going to that
(33:13):
stage, how to test and customizea metrics is very important I
mean, very critical, to protectthe safety and security and
reputational risks. So werecommend strongly recommend to
start from testing phase first.
So testing phase is not justtesting, but customize your
(33:34):
metrics into your, say,professional, the person's. So
that is a testing phase. Andafter that, they can go into
monitoring phase. So that is ourapproach to the customers.
Daniel (33:47):
And I'm wondering, either one of you could answer
this, but why is it and this maybe obvious to maybe more of the
software engineering type crowd,but maybe less so to some others
outside of that crowd. Why is itimportant to once you've tested
your model to actually monitorit online for maybe the things
(34:10):
that are are potentiallyproblematic inputs, whether
that's a security thing like aprompt injection or, maybe
something outside of the youknow, some type of input that
you wanna filter out like IPgoing in or something that
doesn't fit your policy. Why isit important to have that
(34:31):
monitoring piece and not justthe testing piece? Because if I
test my model and I convincemyself that it can't be prompt
injected, which I'm saying thatsort of in jest because, you
know, as you all know, everymodel there's no perfectly
aligned model. There's no everymodel is vulnerable to various
things.
But let's say that I convincemyself of high performance in in
(34:55):
one of these areas. Why then isit useful and necessary then to
monitor that over time or inreal time?
Rick (35:04):
Technically, probably Kenny is also again the best
person. But even if thecustomers can go through the
testing phase, the marketcondition or say, the human
reaction may change over time.So if we do are safe right now,
(35:25):
but if anything, say somethingnew happens today, what we say
guaranteed today may not applytomorrow. So it's a very general
things. But say, in that sense,keep monitoring is very
important to protect ourcustomers, even if the market
(35:46):
condition or the world conditionchange or economic condition
change.
Kenny (35:51):
Yeah, and just to give a concrete example of why you may
want monitoring, so we reallyview them as complementary, and
you really need both, if youwant to make a system reliable.
So for example, if you have likean answer quality metric that
measures how high quality andanswers, you should of course
(36:12):
use that for testing to makesure that it meets some like,
you know, 8090% bar. But then inmonitoring, you actually want to
measure quality of the realanswers that your chatbot is
giving to real customers, right.So from a quality perspective,
it makes a lot of sense fromlike a safety and risk reduction
perspective. Another example isthat, you know, as you
(36:32):
mentioned, Dan, during testing,you might test a bunch of prompt
injections against your system.
But then in deployment, havereal users, some of them are
adversarial, some of them areactually trying to prompt
inject. They may do it increative ways that you haven't
tested before. You may want someguardrail that will
automatically detect thoseattempts and filter them out,
(36:55):
even if you're sure that themodel is robust to 90% of these
attacks.
Chris (37:00):
I'm curious, you guys have some open source available
out there. I know one of thetools is lang check. Can you
talk a little bit about yourapproach to open source?
Kenny (37:11):
For context, lang check is our open source Python
library that contains a suite ofmetrics that are built in that
you can use for evaluating thequality of text. One of our
motivations for creating thislibrary is that I think we
launched it in October 2023,roughly. And around that time,
(37:34):
there weren't a lot of sort ofindustry standard metrics and
practices for evaluating text,particularly in non English
languages as well. So there wassome focus on, you know, these
metrics in English, but we workwith a lot of customers that
have Japanese texts or Chineseor German and these other
languages. We wanted to make alibrary of these metrics that
(37:57):
anyone can use.
We view this as a pretty goodstarting point. If you just need
one or two metrics, you'recomfortable writing code, you
can use LangeCheck and integratethat into your test pipeline or
your monitoring system. But thenif you want something production
scale, and you want an easyworkflow to design custom
metrics and test them againstmanual annotations, that's where
(38:21):
our commercial product comes in.
Daniel (38:23):
Makes sense, yeah. And as we get a little bit closer to
the end here, there's so manythings and this is such a, so
many in-depth areas to go in,which is why it's great that
there's wonderful people likeyourselves exploring, the topic.
But I'm wondering, if we couldmaybe talk just a little bit as
(38:45):
we close out here about whatyou're excited about, about kind
of, yes, Citadel, but maybe thekind of general ecosystem that
that you're a part of. As as youlook to the future, you know,
what's what's exciting to eachof you about how the ecosystem
is developing? What's becomingpossible with the technology?
(39:05):
What's inspiring to you or whatare you thinking about in terms
of the future? Maybe I'll startwith you, Rick.
Rick (39:13):
Okay. So now the chat GPT-five is released. But when
we say look back today from sayfive years later, I believe
that, oh, this is a verypremature say model or something
like that. So in that sense, thetechnology advancement in the AI
(39:35):
field is so rapid. And in thatsense, yeah, there may be a lot
of risks coming in.
But on the other hand, there aremostly infinite opportunities.
So you can find that hugevariety of possibilities, not
only just say the AI directlyrelated technologies, but I
(40:01):
believe some of the materialproducts or machines or anything
will change maybe within five orten years. So in essence, we are
in the midst of the say periodwhere anything can change in
that sense, especiallytechnology related. So there are
(40:23):
lots of say risks may come inand we like to protect such a
kind of risk as a company. Buton the other hand, people can
find many possibilities,opportunities for you to try.
So I strongly believe that soeven though there are lots of
issues in the world that peoplecan enjoy or say, make best use
(40:48):
of this opportunity, everybody.
Daniel (40:51):
Yeah, that's great. What about yourself, Kenny?
Kenny (40:54):
Yeah, I think as a consumer of AI in both my
personal life and work, from aconsumer perspective, it's
really exciting to benefit fromall the advancements in these
new AI tools and models. Ireally loved Oh, three as a
model in chat GBT. And I loveusing cursor. I'm excited for
(41:14):
these tools to become more andmore agentic over time. I think
that's the trend that you see.
If you just look at ChatGPT,originally it was just 3.5 and
GPT-four that just answer aquestion based on forward pass
of the model. But now thesemodels will search the internet
and it'll sort of reason andthink about what to search next.
(41:35):
As a result, the outputs havebecome a lot better. So really
excited for that to improve evenmore from a consumer
perspective. And then from abusiness perspective, I'm really
excited to help bring thesecapabilities to our business
customers and help them use AImore reliably and more
effectively in in theirbusiness.
Daniel (41:56):
That's great. Yeah. Well, thank you both for taking
time to to chat with us today.And thank you both for the work
and thought that you're puttinginto the tools that you're
building and the open sourceprojects that you're putting out
there. It's a great benefit tothe to the community and to the
business world, of course.
So thank you for the work thatyou're doing. And, yeah, we'll
(42:19):
look forward to, to keepingkeeping, an eye on on what you,
what you evaluate and andprotect us from next. So,
appreciate you both, hope youhave a great evening. Thank you
for joining.
Rick (42:32):
Thank you very much.
Kenny (42:34):
Thank you for the conversation.
Jerod (42:42):
Alright. That's our show for this week. If you haven't
checked out our website, head topracticalai.fm, and be sure to
connect with us on LinkedIn, X,or Blue Sky. You'll see us
posting insights related to thelatest AI developments, and we
would love for you to join theconversation. Thanks to our
partner Prediction Guard forproviding operational support
for the show.
Check them out atpredictionguard.com. Also,
(43:05):
thanks to Breakmaster Cylinderfor the beats and to you for
listening. That's all for now,but you'll hear from us again
next week.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
New Heights with Jason & Travis Kelce
Football’s funniest family duo — Jason Kelce of the Philadelphia Eagles and Travis Kelce of the Kansas City Chiefs — team up to provide next-level access to life in the league as it unfolds. The two brothers and Super Bowl champions drop weekly insights about the weekly slate of games and share their INSIDE perspectives on trending NFL news and sports headlines. They also endlessly rag on each other as brothers do, chat the latest in pop culture and welcome some very popular and well-known friends to chat with them. Check out new episodes every Wednesday. Follow New Heights on the Wondery App, YouTube or wherever you get your podcasts. You can listen to new episodes early and ad-free, and get exclusive content on Wondery+. Join Wondery+ in the Wondery App, Apple Podcasts or Spotify. And join our new membership for a unique fan experience by going to the New Heights YouTube channel now!
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com