Introducing Fiddler Guardrails: The Fastest in the Industry

In this episode, we explore how Fiddler Guardrails helps organizations keep large language models (LLMs) on track by moderating prompts and responses before they can cause damage. We break down its industry best latency, secure deployment options, and how it works with Fiddler’s AI observability platform to provide the visibility and control to adapt to evolving threats.

Read the article to learn more about how Fiddler Guardrails can help safeguard your LLM Applications.

Episode Transcript

(00:02):
Welcome back to Safe and Sound AI.
So get ready because today we're divinginto a major development in AI safety.
It's a brand new tool and it's designedto keep those powerful large language
models, LLMs, on track, the onesthat are generating all this text,
translating languages, even writingdifferent kinds of creative content.
But we've all seen it.
They can go off the rails,sometimes in pretty dramatic ways.

(00:22):
Yeah, it's true.
I mean, we're really seeing LLMsbeing adopted across industries.
But with that, obviously comesa whole new set of risks.
These models can sometimes generateincorrect or misleading information.
Exactly, and that's where I think thisidea of Fiddler Guardrails comes in.
It's like a, you know, asafety net for those LLM
applications we're talking about.
The speed is really impressive.
We're talking under 100milliseconds latency.

(00:43):
I mean, that's thefastest in the industry.
So it can keep pace with even the mostdemanding applications like chatbots
or content generation platforms.
Okay, so let's break this down a littleso it sounds like Guardrails builds
on the existing Fiddler Trust Service.
Can you give us a quick refreshon what that is exactly?
Sure.
So the Fiddler Trust Service,think of it as a foundation.
It provides this comprehensive evaluationof both the prompts that are given to an

(01:05):
LLM and the responses that it generates.
And then it scores these interactionsagainst key trust dimensions.
So things like hallucinations, toxicity.
Kind of like a multifacetedrisk assessment for every
single LLM interaction.
So if I'm understanding this correctly,the Trust Service provides the scoring and
a Guardrails is what steps in to actuallymoderate the prompts and responses.

(01:28):
Exactly.
It takes those scores from theTrust Service and then it uses those
scores to determine, okay, should aparticular prompt or response be allowed
through ? So it's almost like havinga sophisticated security checkpoint
built right into your LLM workflow.
It sounds like a pretty robustsystem, but one thing I'm always
curious about is the level of control.
Can organizations customize Guardrails tomatch their own specific risk tolerance?

(01:52):
Because not every use case is going tohave the same level of sensitivity, right?
Absolutely.
And that's actually one of thekey features of Guardrails.
Okay.
You're not stuck with this likeone size fits all approach.
Right.
Companies can actually definetheir own risk tolerance, right?
It's all about deciding what levelof risk you're comfortable with.
So it's not a one size fits all solution.
Organizations have the power todefine their own safety boundaries.

(02:14):
That's pretty impressive.
Exactly.
That kind of flexibility is amazing.
But, you know, setting up all thoserules and thresholds, it sounds
like it could get pretty complex,especially for larger organizations
with multiple LLM deployments.
You know what?
Fiddler's actually madeit surprisingly easy.
And the integration with theirAI Observability platform
gives you this awesome visualdashboard for managing everything.
So it's not just about setting upthe Guardrails, it's about having

(02:37):
the visibility and control tomanage them effectively over time.
Talk about peace of mind.
Exactly.
You can monitor all those keymetrics we talked about, see if any
flags get tripped, any violations,and even drill down into specific
incidents to see why they happen.
Well said.
You mentioned earlierdeployment environments.
We know a lot of organizations are workingwith very sensitive data and have to meet

(02:57):
some very strict security requirements.
How do Fiddler Guardrailsaddress those concerns?
That's a great question, andit's something Fiddler took very
seriously when designing Guardrails.
They can actually be deployed inlots of different environments,
including like virtual privateclouds, and even air-gapped systems.
So even for those organizations operatingin those highly regulated industries,
you know, like healthcare or finance,they can use Guardrails without having to

(03:21):
compromise on their security protocols.
That's a big relief for anyonedealing with sensitive data.
Exactly.
It makes sure that all that dataprocessing and analysis happens in
a secure and controlled environment.
So you minimize the risk of any, youknow, unauthorized access or breaches.
Okay, I think we've laid apretty solid foundation here
about what Fiddler Guardrailsare and why they matter so much.

(03:42):
I want to go back to something we talkedabout earlier, the importance of speed.
You mentioned that Fiddler Guardrailsare the fastest in the industry with a
response time of under 100 milliseconds
. Right.
That combination of speed andsecurity is only going to become
more important as AI keeps advancing.
We're going to see LLMs beingused in even more sensitive and
mission critical applications.
And having these robust securitymeasures, like Fiddler Guardrails in

(04:05):
place, is going to be essential formaking sure that those deployments
are safe, reliable, and trustworthy.
Absolutely, it's about strikingthat balance between innovation and
responsibility, pushing the boundariesof what's possible while also protecting
against those potential risks.
I love that.
What can organizations working with LLMsactually take away from this conversation?
Okay, first and foremost, I think it'scrucial to understand that, that LLM

(04:28):
security, it's not a one and done thing.
It's this ongoing process, this constantvigilance, this, this need to adapt
and be willing to embrace those newtools and strategies as they emerge.
It's almost like an arms race, right?
A cybersecurity arms race, but for LLMs.
You got it.
Exactly that.
That threat landscape,it never sits still.
It's always changing, always evolving.

(04:49):
And those attackers, they're gettingsmarter, more creative with their methods.
Organizations need to be proactive,always assessing their security,
always tweaking, adjusting as needed.
And that's exactly where solutions likeFiddler come in, they give you that
essential protection, but they alsooffer, you know, that flexibility, that
visibility you need to keep pace withthose, those ever changing threats.

(05:10):
Okay, so , let's say an organization'son board, they're convinced, ready
to implement Fiddler Guardrails.
What are some practical first steps?
Where do they even begin?
I'd say start with a really solidrisk assessment, figure out what LLM
applications you're running, the typesof data they're handling, , and really
think about what are the potentialconsequences , if a breach were to happen.
So basically, know yourweaknesses , before you jump in.

(05:32):
Exactly.
Once you've got that clear picture ofyour risk profile, then you can start
looking at Fiddler Guardrails and seehow they fit into your specific needs.
Fiddler has a ton of resources,documentation, tutorials,
And as you start implementingGuardrails, remember, it's
not, you know, a one shot deal.
It's an iterative process.
Don't be afraid to experiment, tweak thosethresholds, you know, really tailor the

(05:54):
system as your security needs evolve.
So it's not a set it andforget it kind of thing.
You've got to stayengaged, keep refining it.
Yeah, exactly.
And that's why those monitoringand analysis capabilities
of Fiddler are so powerful.
You learn from those incidents,you spot those patterns.
You're constantly improvingyour LLM security posture.
So we've covered a lot of groundhere today, but before we wrap things
up, I want to leave our listenerswith something to think about.

(06:17):
As LLMs become more powerful,more integrated into our lives
, those security stakes, they'reonly getting higher, right?
So what role do you think, individualdevelopers, researchers, even just
everyday users have in making sure thatAI is developed and deployed responsibly?
That's a really deep question and Idon't think there's any easy answer.

(06:37):
I think it starts with, awareness.
We all need to understand thosepotential risks that come with AI,
not just the shiny, exciting benefits.
And we have to be thoughtful aboutthe choices we make, both as, as
the people creating this technologyand as the people using it.
It's like recognizing that AIisn't just , this cold, hard tool.
It's a reflection of us, of ourvalues, our hopes for the future.

(06:58):
And on that note, I thinkit's time to wrap up our deep
dive into Fiddler Guardrails.
It's been a fascinating look at this gamechanging technology that has the potential
to really shape the future of AI security.
This podcast is broughtto you by Fiddler AI.
For more on Fiddler Guardrails,see the article in the description.