October 24, 2024 • 35 mins
Professor Werbach speaks with Shea Brown, founder of AI auditing firm BABL AI. Brown discusses how his work as an astrophysicist led him to and machine learning, and then to the challenge of evaluating AI systems. He explains the skills needed for effective AI auditing and what makes a robust AI audit. Kevin and Shae talk about the growing landscape of AI auditing services and the strategic role of specialized firms like BABL AI. They examine the evolving standards and regulations surrounding AI auditing from local laws to US government initiatives to the European Union's AI Act. Finally, Kevin and Shae discuss the future of AI auditing, emphasizing the importance of independence.
Shea Brown, the founder and CEO of BABL AI, is a researcher, speaker, consultant in AI ethics, and former associate professor of instruction in Astrophysics at the University of Iowa. Founded in 2018, BABL AI has audited and certified AI systems, consulted on responsible AI best practices, and offered online education on related topics. BABL AI’s overall mission is to ensure that all algorithms are developed, deployed, and governed in ways that prioritize human flourishing. Shea is a founding member of the International Association of Algorithmic Auditors (IAAA).
International Association of Algorithmic Auditors
NYC Local Law 144: Automated Employment Decision Tools (AEDT)
Want to learn more? Engage live with Professor Werbach and other Wharton faculty experts in Wharton's new Strategies for Accountable AI online executive education program. It's perfect for managers, entrepreneurs, and advisors looking to harness AI’s power while addressing its risks.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
Shea, thank you so much for joining me on the podcast.
Thank you for having me here.
How does an astrophysicist turn into an AI auditor?
Yeah, that's a really strange story.
my research in astrophysics was primarily in data analysis.
(00:24):
So I worked on detecting black holes in distant galaxies, measuring magnetic fields,things like this.
And the problem with astrophysics is that there's so much data that we're getting from thesky now that literally if every human on earth spent their entire lives looking at it,
they would not
be able to get through it all.
And so it sort of necessitated me getting into machine learning and artificialintelligence as a tool to sift through this data.
(00:51):
And I decided that I was really excited about AI and I wanted to get into it.
And so I sort of started to do a little discovery of like, what, where could I have aunique impact?
And I felt like AI auditing and was kind of a unique place where my brain just sort offit.
you know, the critical scientific approach to let's dig and uncover what the problemswith, with this, these systems are.
(01:17):
And it was around the time of like Cambridge Analytica and ProPublica's article aboutbiased algorithms.
And it just felt like the right time.
So around 2018.
And so I sort of made a switch to sort of spend a lot more time focusing on how do weaudit systems?
What are AI systems and what are the ways we can kind of systematically uncover?
(01:38):
the risks that were beginning to become apparent.
So what are the basic technical skills or capabilities that are needed in order toeffectively audit an AI system?
Yeah.
So I think that a lot of the, there's not that many technical skills per se.
(01:58):
So, so we spent a lot of time trying to train auditors and, and, upskill them into, intothis field.
And it's not like you have to learn to code in Python or be able to build a neural networkor something like that.
It's a much more about understanding the types of systems you're going to interact withthat you might be auditing in terms of how they work roughly.
(02:21):
What kind of data they use, what are the risks associated in particular with that kind oftechnology?
And then a lot of sort of societal risk analysis.
So we have a lot of philosophers actually are really great at thinking through sort ofwhat issues are really germane to this problem, sifting through irrelevant information and
(02:43):
getting down to the core of what's important.
And that's really the...
That's really the skill and of the technical skills, would say statistics or anunderstanding of how many factors can influence a data point that there's a big, we call
(03:06):
it parameter space thinking.
That's really the skill.
It's much more about understanding the little bit of statistics and appreciating thecomplexity of data and how AI processes data.
What does that exactly mean, parameter space thinking?
So parameter space thinking, so a parameter space is just that there are lots of differentfactors that influence a particular result.
(03:33):
Let's say it's a number, let's say it's a behavior that AI system has.
most of the time when you're auditing these systems, some of those parameters areimportant, others are not.
So that's like the first way of thinking, like understanding what the parameter space is.
So an example might be
(03:53):
I have some sort of activity detection in video recordings, right?
So maybe I want to detect from a camera, a video feed, whether someone's getting robbed orwhen someone's stealing something.
So in that scenario, there's a whole bunch of parameters.
It's like, what kind of camera did it use?
What kind of model am I using to predict this?
(04:16):
What kind of training data?
Where did that training data come from?
Is it different than where I'm?
using it now.
Maybe I'm doing it in parking garages, but all my data is from like YouTube channels,really high quality video.
Those are all parameters that are relevant for understanding the outputs of some testingthat you might have done or the risk profile that you might have.
(04:39):
And so being able to parse that out and actually, I don't know if quantifies the rightword, ideally quantify would be great.
But sometimes it just qualitatively parse out what are the things that are important ofthis massive parameter space and then getting down to here are the relevant things and
this is how I can sort of partition what factors are important for risk.
(05:02):
Okay, let's take a step back before we go on.
What exactly is an AI audit and what basically are the kinds of services that your companyBabel provides to organizations?
Yeah.
So in 2018, we started AI audit wasn't really a thing.
And so it's, it's now becoming more of a thing.
(05:24):
So what is it?
it's very akin to, let's say a financial audit, like a typical audit that you wouldexpect.
So what is that?
There's some standard or some normative guidance for like what a company or a person oughtto do or ought to have in place.
And an auditor's job is to come in and to
(05:46):
look at what a company or a person has done and has in place against that standard andweigh in on with some judgment.
there are different types of audits.
It could be like binary where it's like you pass or fail.
It might be more like an assurance engagement where it's like, I have reasonable assurancethat this company adheres to the standard.
(06:08):
So that's exactly what we do, except that it's over AI systems.
And the sort of processes around those AI systems, how they're governed, how they'remanaged, how they're tested.
And that's really it.
The complication here is that you can't just, you wouldn't just get a financial auditor todo this, right?
(06:28):
There's certain capabilities that you need to have to recognize when something is in placeor not.
You know, is that risk management framework that that company has the right one?
Is it actually effective at managing risks?
For instance, is that testing the right kind of testing?
Is that information you get relevant?
(06:49):
And so it's the same as that sort of auditing, except it just has added complexity becausethe complexity of the system is so much higher than just like QuickBooks or some financial
records.
So you mentioned when you got started, there were not a lot of people and organizations inthe space.
(07:11):
Now there's a range, everything from individuals to some of the very biggest professionalservices and consulting firms providing AI audit and assessment services.
And there's also researchers doing audit studies from the outside without firms'permission.
So how do you look at the space?
don't know if industry is too strong a word, but the community that exists around AIauditing today.
(07:36):
So there is a strong ecosystem around AI auditing that's growing and there are differentsort of players and actors and places where people naturally fit.
And in this space, I would say there are kind of the technical assessor space, right?
So these are individual people or firms that are gonna really come in and do some sort oftechnical assessment.
(07:59):
They're gonna test for bias.
They're going to check robustness.
There might be some cybersecurity elements or...
red teaming of large language models would fit into this.
And there are a lot of players in that space from individual consultants, academics thatmight work on the side to all of the sort of large assurance firms, the big four and
(08:21):
consultancies will have some services around this.
And then there's like the, those same people are also going to offer services around likerisk management, things like this.
These are the sort of consulting
capability building players.
Then there are the of the technology players where it's might be a technology platformthat will automatically run tests or might be like a GRC platform that will organize how
(08:50):
you might manage your risk.
And they will kind of have the connective, that will be like the connective tissue, thetech assisted layer for getting this done.
And then there's sort of the
the external sort of third party audit assurance services.
And that's where Babel sits primarily.
(09:10):
We do very little consulting.
We try to avoid it.
We prefer to be like an independent third party auditor.
And there are a number of smaller organizations like Babel.
Eticus is another one.
There are sort of mixed like holistic AI is a platform and they do audits.
And then of course there are the big players that are already in the
(09:33):
sort of regulatory assurance space of the big four plus, you know, the other ones, they'reall sort of getting into this in various levels, but the maturity is still quite low
because there are very few standards around it at the moment.
Yeah, so how does a small firm like yours compete against those giant consultants?
(09:55):
So, well, we start early, which we did.
The way we compete is that we're hyper -focused.
And so we're not doing, we don't do ESG assurance.
We don't do financial auditing.
We don't do cybersecurity auditing.
We do one thing and we do it very well, which is the sort of AI assurance risk managementalso includes technical testing.
(10:25):
we approach most of the sort of senior team, we were professors and other things we knowabout research.
And so we started early on recognizing that we don't know what we're doing.
And so we have to figure out what to do.
so early on we weren't auditing, we were consulting and getting our hands dirty, trying tofigure out how do you actually systematically detect risk in these systems?
(10:49):
What is a good risk assessment?
What kind of technical testing and how should they be informed by each other?
and we published a lot, know, so we're publishing our research, we're really coming at itwith ideally sort of intellectual humility that there are a lot of things to be figured
out and it's that niche aspect and so now I mean we have people from the big four comingthrough our training program to learn from us how to do these things because we're just
(11:16):
focusing on one thing and that's probably
Any, all the business people listening will know that that that's a good strategy ingeneral.
If you're small and don't have resources to niche down and just make sure you're very goodat a very specific thing that hopefully will have a big market at the end, which I think
is becoming apparent.
(11:38):
Can you describe what makes a good AI audit?
So a good AI audit, so this will be my opinion, because some people will think differentthings.
I think a good AI audit is going to have a clear objective, right?
So you have to know what the audit is for.
Like why are you doing this?
(11:59):
Is it because there's a law in place?
Is it for some sort of internal risk management that you want senior management to knowhow their company is performing, for instance?
What's the goal?
So it has to have a very clear goal and a very clear intended audience.
And that's not always obvious.
And so if you have that, then that's a good, good start.
There has to be some kind of standards in place.
(12:23):
Now it doesn't have to be standards.
That's, that's like totally set or that aren't changing or they don't even, they could bestandards that the auditors themselves had put in place, but they have to be there ahead
of time.
And they have to be clear and clearly articulated.
And then there has to be independence.
(12:43):
think this is a, it, can of course have internal audit, which is not fully independent.
There's some independence because they report, say up to the board or something.
But I think independence or impartiality is a pretty important component because whatwe've seen is that it's so easy to manipulate the results of these things like technical
testing.
(13:03):
For instance, I have a big data set.
I'm measuring bias.
I run it through, I don't like the results.
say, what is it that's causing these results to be bad?
Well, maybe this data point is not appropriate.
So I'll just take that out and slowly the results start looking much, much better and theway I want them to look.
That's not a great situation.
(13:24):
so having that impartiality, having the clear goals ahead of time, knowing what theprocess is will avoid those sorts of problems.
and so I think those are kind of the key components.
And then the final thing I'll say is auditor competence because you have to understand.
mean, so we, our audits are kind of assurance engagements.
(13:47):
So we follow sort of assurance, international assurance standards.
And one of the things you need to do is really assess the audit risk or where, where it'sthe risk of material misstatement that could happen.
where they just gave us the wrong information, maybe on purpose, maybe by accident.
If you don't understand the system and how it works, then it's going to be hard to detectwhere those risks are.
(14:14):
so auditor competence is sort of the final thing that like seals the deal.
That feels to me like a good package of things that would make a good audit.
Great, so there's a lot in what you just said to unpack.
Let me start at the beginning with the rationale.
What do you find today are the main reasons that companies come to you for those auditservices?
(14:37):
So there were sort of three stages.
The first stage was reputational risk.
So early on, the people who came to us were under sort of severe kind of reputationscrutiny.
Either the articles were written about them, accusing them of bad practices or AI that wasbad, the Senate was looking at them or something like that.
(15:00):
That was sort of the first phase.
People were just panicked about.
having a bad reputation because a lot was being written about it, let's say five yearsago.
The next is that laws actually got passed.
so things like New York city has a local law one for four, which requires bias audits ofautomated employment decision tools.
So the tools that would like sort through your CV or your resume, when you apply for ajob, there's many of those, almost every job you apply to will have those.
(15:28):
If you're operating in New York, they have to be audited.
So there was like a compliance or regulatory pressure that
brought a new wave of clients to us and to other auditors, because I had to get this donebecause of that.
That's not going to stop because there's a lot of laws coming that are going to requirethat.
But there's a new wave now, which is procurement pressure.
(15:48):
So even before these other laws have come into effect, the majority of people who come tous now are trying to sell to, say a large enterprise.
That large enterprise is, has already worried about compliance and reputational risk frombefore.
is now pushing back on their vendors to get some sort of auditor assurance done.
They need something that's gonna limit their downside risk.
(16:11):
And so that's the pressure we're feeling right now.
That's what's driving most of the activity in this space, from my experience.
Yeah, no, no, no, it's really interesting.
So the second thing you said before was standards.
And sometimes, so if we're about local L -144, specifies an adverse impact report, whatthat test is.
(16:31):
But if you don't have something like that, where do the standards come from that you use?
So that's a really good question.
So there are some emerging standards.
So NIST has their AI risk management framework.
That is a fairly kind of, I wouldn't say comprehensive.
(16:53):
It has a lot of options for people to manage risk.
And so if we are trying to audit somebody for their risk management practices,
That's, it's not really a standard.
It's not written necessarily like a standard, but it does have enough elements and enoughspecificity that we could use it as sort of criteria for auditing.
(17:17):
There are now ISO has now released 42 ,001, which is a, an AI management system standard,which has gotten a lot of interest recently.
And that is because it's an ISO standard is written like a standard that is
meant to be auditable.
even have guidelines.
It's 42 ,006, which are in draft, but they're meant to specify how that should be auditedagainst.
(17:43):
So that's a great framework to work from.
And then there are other laws that like EU AI Act, Colorado now has another law, Illinoispassed some law, California has some new laws that are likely to be signed very soon.
All of those will work, but there are always details
(18:03):
that are complications for an auditor that we have to sort through that does cause,there's extra risk and there's a ton of development work.
So we spend probably half our time working on audits.
The other half of the time working on standard, our own internal processes and internalstandards and risk management and procedures to manage the risk of those audits.
(18:25):
it's R and D, half R and D at the moment.
No, I would think that there is a danger in what you described in the things go the otherway and there are too many different standards.
There's all these different competing options out there with financial accounting.
We've got GAP and international equivalents and financial accounting standards board andstuff and there's some clarity.
(18:48):
Do we need more coordination about what the standards are for doing an AI audit?
I think we do need that.
And there are some organizations that are trying to do that.
I'm a fellow at For Humanity.
That organization is trying to do exactly this, have a central kind of repository ofcertification schemes that might be informed by laws or might be informed by standards,
(19:13):
but are sort of developed in a unique sort of coherent way.
But I don't think that we're going to get there.
I think if you look at cybersecurity, for instance,
There's just so many different certification schemes, so many different types of ways ofauditing types of audits.
You've got SOC 2, you've got ISO, there are other organizations.
(19:35):
Some companies have just had their own certification that they develop.
And I think we're going to see it evolve in that way.
Now, one thing I'll say is that the EU AI Act is so comprehensive in its requirements, orat least so onerous, maybe is the word that some people would use.
that it will probably be the most difficult quote unquote standard to adhere to.
(20:01):
And so if you adhere to the most rigorous standard, it's probably gonna cover you for 95 %of all of the other regulations.
And so I think we'll see people gravitating towards that as a benchmark.
then it'll be just filling in the gaps for some individual local law or individual kindof...
(20:23):
association that might want to put something together.
Although it's interesting you mentioned the work that you have to do internally on R &Dand assessment on standards with something like the AI Act, there's still tremendous gaps
in terms of the interpretive guidance and what those standards mean and so forth.
So do firms like yours really have to fulfill a role for making some of those decisionsunder conditions of uncertainty as these standards are evolving?
(20:52):
Absolutely.
we do have to interpret if there aren't standards and the EU will have, Sensenlack isgoing to come out with more detailed standards and we are sort of keeping track of that
and providing input when we can.
(21:12):
But a lot of times we are going to have to make some calls on our own.
What do we think is good enough given the current language of the law?
But I think that's okay because the level of maturity for most organizations frankly isnot anywhere near even the laws it's written, much less some detailed standards.
(21:33):
And so our sort of general strategy, and I think it's a good strategy for otherorganizations that they want to try to audit, is to fall back on transparency and say,
listen, this is exactly what we are using as a...
as a standard, these are the criteria that we use, these are the procedures we wentthrough to check things and disclose that.
(21:55):
And then people will know exactly what happened and what you did.
And it may not be exactly compliant with the law as the new standards might indicate, butyou can always go back and redo it.
And it just, provides a lot more clarity on what exactly is going on.
(22:16):
about certifications and accreditations for the auditors themselves?
I know that's something you've been involved in, in the AI space.
So that's a difficult thing because there isn't a central body for that.
And so I mentioned for Humanity before, they have some sort of certifications based ontheir certification scheme.
So they'll kind of accredit auditors for their certification schemes.
(22:40):
And so we've, for instance, gotten some of those as our organization.
There's a new organization that I'm one of the founding members of, which is theInternational
Association of algorithmic auditors.
I might've gotten one of those words mixed up, but it's I AAA, I AAA.
And so the goal there is much more about identifying what are the capabilities.
(23:05):
It's in a professional association.
What are the capabilities we think auditors need to have and how could they acquire thesort of bona fides that they need in order to do this?
It might be, we haven't got to the point where we have an examination yet or anything likethat.
It probably will get to that point, but I have a feeling it will be a mix of things fromexperience to education.
(23:28):
We don't have any degrees in this at the moment.
That's something that will come.
I'm sure just like, just like financial auditing has accounting degrees that will lead tofinancial auditing that was going to happen eventually.
But right now it's a little bit of the wild West.
And I think we rely on our own training for our own auditors.
And so we have our own program.
(23:50):
where we've outlined the capabilities and we train internally our own auditors and we justmake sure that we are transparent about that and everyone can see what we are expecting in
terms of capabilities.
But eventually, probably everybody's gonna have to get some sort of certification wheneverone of these organizations kind of rises to the top in terms of credibility.
(24:12):
And then going back to when you answer the question about what makes a good audit, thinkthe third thing you said was independence.
Isn't there though an inherent conflict of interest if a company is paying you to providea service?
How can you provide enough of an assurance of independence in what you respond to them?
Yeah.
So we basically approach this the same way that, that financial auditors approach this orassurance professionals approach this.
(24:39):
So there's a, there's a standard kind of code of conduct.
And we, in fact, the standard we rely on is the same one that the, auditors rely on.
And so we look at, because this happens, of course, you know, a company pays Deloitte tocome audit their financial, they have to pay them, right.
Deloitte is not going to do it for free.
So there, there are.
(25:01):
different ways of mitigating the sort of fundamental issue of impartiality andindependence.
know, Sarbanes -Oxley came along and had some additional requirements there.
We keep an eye on that and make sure that we're following that as well.
But there, it's not easy and it's a difficult task.
(25:22):
And what makes it more difficult is it's not like financial auditing where you can show upand expect that the company understands how
bookkeeping works and understands how they need to track their financial transactions.
That's not the case here.
We are dealing with organizations that it's shifting so much that they don't know whatthey need to do.
(25:47):
And so we have to play a balancing act of education, like what is it that you need to do?
But then we also are gonna come in and ensure that you do that.
And that's not easy.
And I think that the level of independence is by necessity slightly less than it would beif there was a very strict standard that everybody understood and they go off and get it
(26:10):
done and we are only coming in to check their homework.
But I think that that's just a function of the time we're in right now.
And as this becomes more mature as a industry, we're gonna see that independence isactually not gonna be that different than
than financial auditing.
(26:31):
So right now, how do you convince clients or prospective clients to accept your approachto it, to, you know, the value of what you're doing, the way that you put it together for
sometimes it's difficult.
what are the challenges?
The challenges are the way we approach it is that we are not consulting them.
(26:55):
we're, are simply, they have to do all the work, right?
So they're the ones who do testing.
They're the ones who have to manage the risk.
We have governance requirements, like for New York, New York law doesn't require riskassessments or governance or any risk management at all.
We have that in our audits.
And so there's this super augatory nature of some of the things that they have to acceptthat they're going to be doing the work and we are going to be checking their homework and
(27:23):
validating the everything they've done, everything they said they did, they've actuallydone.
That's a tough call.
The way we, we convince them is that every law besides New York, every law that's coming
is going to require robust internal governance and risk management and technical testing.
They all are.
(27:43):
So if you were going to be a mature organization using AI in the near future, you have todo this anyway.
So why not start now where the bar is much lower, get your feet wet, get used to havingpeople hold your feet to the fire.
And that's going to mature you as an organization and then you can grow from there.
That's the, that's the selling point.
(28:05):
Not everybody wants that of course.
But at this point, we're only looking for the people who are willing to do that.
And so there's a filter effect a little bit.
And so there's probably a large swath of the market that is going to get serviced by otherkinds of organizations or other kinds of companies that don't approach audit in the same
way we do.
(28:29):
How broad do you think the practice of AI auditing is going to get?
Because obviously we're seeing now companies of every scale in every industry deploying AIsystems and often many different AI systems.
So do you envision a world where all those things are subject to auditing?
sure that would be good for a firm like yours from a business standpoint, but obviouslythere's some overhead at some point.
(28:53):
Yeah, so I don't think so.
think it's going to evolve into much more of a, and again, I don't know, like I don't havea crystal ball, but my hunch is that a lot of the processes for all of these algorithms
are gonna have to be automated and there will be centralized risk management andgovernance and testing of these systems that will be
(29:22):
more specific and more based on standards and that the role of the auditor is to auditthat system, that quality management system with the extra bits that are really relevant
for AI systems.
think that's going to be, because that's much more manageable.
don't have to check that every, you know, for this particular algorithm that you're usinginternally to do something is compliant.
(29:48):
What I'm checking is you have that program in place.
and you have the processes in place, the controls that are going to manage that.
Now, the caveat here is that some laws like the EU AI Act, there'll be high risk systemswhich have requirements for the AI system itself.
Luckily, a lot of those requirements are also organizational based.
(30:11):
do you have a risk, have you done risk management for that?
Right.
So one risk management system can cover multiple high risk algorithms.
The technical testing is going to be unique to a particular algorithm, but that can becentralized and audited at a sort of organizational level.
so anyway, that's what I see it's emerging as organizations get audited and the individualAI systems are being controlled internally in a robust way by those organizations.
(30:41):
And going forward, what do you see as the greatest challenge for the auditing community inAI or specifically for you at Babel?
So the biggest challenge is right now, as you alluded to, the explosion of standards andlaws.
there are just so many, so we spent a long time for New York City, for instance,developing how are we gonna test, and it's very simple, it's like one metric, disparate
(31:09):
impact.
How are we gonna test that?
How are we gonna provide assurance that they have actually done the right thing?
And it was a lot of development.
And if you look at these laws, like the digital services act, which is already out and,and, and, and people are being audited for that.
The EU AI act, Colorado, all the state laws, some of the federal guidance that is, that iscoming out.
There's just so many kind of normative frameworks for, for what you need to do.
(31:35):
That it's going to be difficult to have that level of development work.
And so the way we're approaching this is that we are focusing on.
our internal process of like, how do we provide assurance?
How do we audit these systems and these organizations in general based on that normativeguidance?
What is, and then it's like, what is the process?
(31:56):
If someone comes to us with this brand new thing, like in Taiwan, we need to, this is theregulation and here are the standards.
What's our process for determining is this auditable?
What are the capabilities we need to audit that?
What are the extra resources we might need to
to execute that audit at what level of assurance.
And that's sort of a more unified framework that we're working on.
(32:20):
And that's the way it's probably going to have to proceed until there's like really nichedown into like, I'm an auditor for just this one law and then just this one industry.
have to be kind of, there has to be that flex.
That's the challenge, that flexibility.
Great, well this has been a really fascinating conversation.
Really interesting to see how the whole space of all is going forward.
(32:43):
Jay, thank you very much for your time.
Thank you so much for having me.
Shea, thank you so much for joining me on the podcast.
Thank you for having me here.
How does an astrophysicist turn into an AI auditor?
Yeah, that's a really strange story.
my research in astrophysics was primarily in data analysis.
(00:24):
So I worked on detecting black holes in distant galaxies, measuring magnetic fields,things like this.
And the problem with astrophysics is that there's so much data that we're getting from thesky now that literally if every human on earth spent their entire lives looking at it,
they would not
be able to get through it all.
And so it sort of necessitated me getting into machine learning and artificialintelligence as a tool to sift through this data.
(00:51):
And I decided that I was really excited about AI and I wanted to get into it.
And so I sort of started to do a little discovery of like, what, where could I have aunique impact?
And I felt like AI auditing and was kind of a unique place where my brain just sort offit.
you know, the critical scientific approach to let's dig and uncover what the problemswith, with this, these systems are.
(01:17):
And it was around the time of like Cambridge Analytica and ProPublica's article aboutbiased algorithms.
And it just felt like the right time.
So around 2018.
And so I sort of made a switch to sort of spend a lot more time focusing on how do weaudit systems?
What are AI systems and what are the ways we can kind of systematically uncover?
(01:38):
the risks that were beginning to become apparent.
So what are the basic technical skills or capabilities that are needed in order toeffectively audit an AI system?
Yeah.
So I think that a lot of the, there's not that many technical skills per se.
(01:58):
So, so we spent a lot of time trying to train auditors and, and, upskill them into, intothis field.
And it's not like you have to learn to code in Python or be able to build a neural networkor something like that.
It's a much more about understanding the types of systems you're going to interact withthat you might be auditing in terms of how they work roughly.
(02:21):
What kind of data they use, what are the risks associated in particular with that kind oftechnology?
And then a lot of sort of societal risk analysis.
So we have a lot of philosophers actually are really great at thinking through sort ofwhat issues are really germane to this problem, sifting through irrelevant information and
(02:43):
getting down to the core of what's important.
And that's really the...
That's really the skill and of the technical skills, would say statistics or anunderstanding of how many factors can influence a data point that there's a big, we call
(03:06):
it parameter space thinking.
That's really the skill.
It's much more about understanding the little bit of statistics and appreciating thecomplexity of data and how AI processes data.
What does that exactly mean, parameter space thinking?
So parameter space thinking, so a parameter space is just that there are lots of differentfactors that influence a particular result.
(03:33):
Let's say it's a number, let's say it's a behavior that AI system has.
most of the time when you're auditing these systems, some of those parameters areimportant, others are not.
So that's like the first way of thinking, like understanding what the parameter space is.
So an example might be
(03:53):
I have some sort of activity detection in video recordings, right?
So maybe I want to detect from a camera, a video feed, whether someone's getting robbed orwhen someone's stealing something.
So in that scenario, there's a whole bunch of parameters.
It's like, what kind of camera did it use?
What kind of model am I using to predict this?
(04:16):
What kind of training data?
Where did that training data come from?
Is it different than where I'm?
using it now.
Maybe I'm doing it in parking garages, but all my data is from like YouTube channels,really high quality video.
Those are all parameters that are relevant for understanding the outputs of some testingthat you might have done or the risk profile that you might have.
(04:39):
And so being able to parse that out and actually, I don't know if quantifies the rightword, ideally quantify would be great.
But sometimes it just qualitatively parse out what are the things that are important ofthis massive parameter space and then getting down to here are the relevant things and
this is how I can sort of partition what factors are important for risk.
(05:02):
Okay, let's take a step back before we go on.
What exactly is an AI audit and what basically are the kinds of services that your companyBabel provides to organizations?
Yeah.
So in 2018, we started AI audit wasn't really a thing.
And so it's, it's now becoming more of a thing.
(05:24):
So what is it?
it's very akin to, let's say a financial audit, like a typical audit that you wouldexpect.
So what is that?
There's some standard or some normative guidance for like what a company or a person oughtto do or ought to have in place.
And an auditor's job is to come in and to
(05:46):
look at what a company or a person has done and has in place against that standard andweigh in on with some judgment.
there are different types of audits.
It could be like binary where it's like you pass or fail.
It might be more like an assurance engagement where it's like, I have reasonable assurancethat this company adheres to the standard.
(06:08):
So that's exactly what we do, except that it's over AI systems.
And the sort of processes around those AI systems, how they're governed, how they'remanaged, how they're tested.
And that's really it.
The complication here is that you can't just, you wouldn't just get a financial auditor todo this, right?
(06:28):
There's certain capabilities that you need to have to recognize when something is in placeor not.
You know, is that risk management framework that that company has the right one?
Is it actually effective at managing risks?
For instance, is that testing the right kind of testing?
Is that information you get relevant?
(06:49):
And so it's the same as that sort of auditing, except it just has added complexity becausethe complexity of the system is so much higher than just like QuickBooks or some financial
records.
So you mentioned when you got started, there were not a lot of people and organizations inthe space.
(07:11):
Now there's a range, everything from individuals to some of the very biggest professionalservices and consulting firms providing AI audit and assessment services.
And there's also researchers doing audit studies from the outside without firms'permission.
So how do you look at the space?
don't know if industry is too strong a word, but the community that exists around AIauditing today.
(07:36):
So there is a strong ecosystem around AI auditing that's growing and there are differentsort of players and actors and places where people naturally fit.
And in this space, I would say there are kind of the technical assessor space, right?
So these are individual people or firms that are gonna really come in and do some sort oftechnical assessment.
(07:59):
They're gonna test for bias.
They're going to check robustness.
There might be some cybersecurity elements or...
red teaming of large language models would fit into this.
And there are a lot of players in that space from individual consultants, academics thatmight work on the side to all of the sort of large assurance firms, the big four and
(08:21):
consultancies will have some services around this.
And then there's like the, those same people are also going to offer services around likerisk management, things like this.
These are the sort of consulting
capability building players.
Then there are the of the technology players where it's might be a technology platformthat will automatically run tests or might be like a GRC platform that will organize how
(08:50):
you might manage your risk.
And they will kind of have the connective, that will be like the connective tissue, thetech assisted layer for getting this done.
And then there's sort of the
the external sort of third party audit assurance services.
And that's where Babel sits primarily.
(09:10):
We do very little consulting.
We try to avoid it.
We prefer to be like an independent third party auditor.
And there are a number of smaller organizations like Babel.
Eticus is another one.
There are sort of mixed like holistic AI is a platform and they do audits.
And then of course there are the big players that are already in the
(09:33):
sort of regulatory assurance space of the big four plus, you know, the other ones, they'reall sort of getting into this in various levels, but the maturity is still quite low
because there are very few standards around it at the moment.
Yeah, so how does a small firm like yours compete against those giant consultants?
(09:55):
So, well, we start early, which we did.
The way we compete is that we're hyper -focused.
And so we're not doing, we don't do ESG assurance.
We don't do financial auditing.
We don't do cybersecurity auditing.
We do one thing and we do it very well, which is the sort of AI assurance risk managementalso includes technical testing.
(10:25):
we approach most of the sort of senior team, we were professors and other things we knowabout research.
And so we started early on recognizing that we don't know what we're doing.
And so we have to figure out what to do.
so early on we weren't auditing, we were consulting and getting our hands dirty, trying tofigure out how do you actually systematically detect risk in these systems?
(10:49):
What is a good risk assessment?
What kind of technical testing and how should they be informed by each other?
and we published a lot, know, so we're publishing our research, we're really coming at itwith ideally sort of intellectual humility that there are a lot of things to be figured
out and it's that niche aspect and so now I mean we have people from the big four comingthrough our training program to learn from us how to do these things because we're just
(11:16):
focusing on one thing and that's probably
Any, all the business people listening will know that that that's a good strategy ingeneral.
If you're small and don't have resources to niche down and just make sure you're very goodat a very specific thing that hopefully will have a big market at the end, which I think
is becoming apparent.
(11:38):
Can you describe what makes a good AI audit?
So a good AI audit, so this will be my opinion, because some people will think differentthings.
I think a good AI audit is going to have a clear objective, right?
So you have to know what the audit is for.
Like why are you doing this?
(11:59):
Is it because there's a law in place?
Is it for some sort of internal risk management that you want senior management to knowhow their company is performing, for instance?
What's the goal?
So it has to have a very clear goal and a very clear intended audience.
And that's not always obvious.
And so if you have that, then that's a good, good start.
There has to be some kind of standards in place.
(12:23):
Now it doesn't have to be standards.
That's, that's like totally set or that aren't changing or they don't even, they could bestandards that the auditors themselves had put in place, but they have to be there ahead
of time.
And they have to be clear and clearly articulated.
And then there has to be independence.
(12:43):
think this is a, it, can of course have internal audit, which is not fully independent.
There's some independence because they report, say up to the board or something.
But I think independence or impartiality is a pretty important component because whatwe've seen is that it's so easy to manipulate the results of these things like technical
testing.
(13:03):
For instance, I have a big data set.
I'm measuring bias.
I run it through, I don't like the results.
say, what is it that's causing these results to be bad?
Well, maybe this data point is not appropriate.
So I'll just take that out and slowly the results start looking much, much better and theway I want them to look.
That's not a great situation.
(13:24):
so having that impartiality, having the clear goals ahead of time, knowing what theprocess is will avoid those sorts of problems.
and so I think those are kind of the key components.
And then the final thing I'll say is auditor competence because you have to understand.
mean, so we, our audits are kind of assurance engagements.
(13:47):
So we follow sort of assurance, international assurance standards.
And one of the things you need to do is really assess the audit risk or where, where it'sthe risk of material misstatement that could happen.
where they just gave us the wrong information, maybe on purpose, maybe by accident.
If you don't understand the system and how it works, then it's going to be hard to detectwhere those risks are.
(14:14):
so auditor competence is sort of the final thing that like seals the deal.
That feels to me like a good package of things that would make a good audit.
Great, so there's a lot in what you just said to unpack.
Let me start at the beginning with the rationale.
What do you find today are the main reasons that companies come to you for those auditservices?
(14:37):
So there were sort of three stages.
The first stage was reputational risk.
So early on, the people who came to us were under sort of severe kind of reputationscrutiny.
Either the articles were written about them, accusing them of bad practices or AI that wasbad, the Senate was looking at them or something like that.
(15:00):
That was sort of the first phase.
People were just panicked about.
having a bad reputation because a lot was being written about it, let's say five yearsago.
The next is that laws actually got passed.
so things like New York city has a local law one for four, which requires bias audits ofautomated employment decision tools.
So the tools that would like sort through your CV or your resume, when you apply for ajob, there's many of those, almost every job you apply to will have those.
(15:28):
If you're operating in New York, they have to be audited.
So there was like a compliance or regulatory pressure that
brought a new wave of clients to us and to other auditors, because I had to get this donebecause of that.
That's not going to stop because there's a lot of laws coming that are going to requirethat.
But there's a new wave now, which is procurement pressure.
(15:48):
So even before these other laws have come into effect, the majority of people who come tous now are trying to sell to, say a large enterprise.
That large enterprise is, has already worried about compliance and reputational risk frombefore.
is now pushing back on their vendors to get some sort of auditor assurance done.
They need something that's gonna limit their downside risk.
(16:11):
And so that's the pressure we're feeling right now.
That's what's driving most of the activity in this space, from my experience.
Yeah, no, no, no, it's really interesting.
So the second thing you said before was standards.
And sometimes, so if we're about local L -144, specifies an adverse impact report, whatthat test is.
(16:31):
But if you don't have something like that, where do the standards come from that you use?
So that's a really good question.
So there are some emerging standards.
So NIST has their AI risk management framework.
That is a fairly kind of, I wouldn't say comprehensive.
(16:53):
It has a lot of options for people to manage risk.
And so if we are trying to audit somebody for their risk management practices,
That's, it's not really a standard.
It's not written necessarily like a standard, but it does have enough elements and enoughspecificity that we could use it as sort of criteria for auditing.
(17:17):
There are now ISO has now released 42 ,001, which is a, an AI management system standard,which has gotten a lot of interest recently.
And that is because it's an ISO standard is written like a standard that is
meant to be auditable.
even have guidelines.
It's 42 ,006, which are in draft, but they're meant to specify how that should be auditedagainst.
(17:43):
So that's a great framework to work from.
And then there are other laws that like EU AI Act, Colorado now has another law, Illinoispassed some law, California has some new laws that are likely to be signed very soon.
All of those will work, but there are always details
(18:03):
that are complications for an auditor that we have to sort through that does cause,there's extra risk and there's a ton of development work.
So we spend probably half our time working on audits.
The other half of the time working on standard, our own internal processes and internalstandards and risk management and procedures to manage the risk of those audits.
(18:25):
it's R and D, half R and D at the moment.
No, I would think that there is a danger in what you described in the things go the otherway and there are too many different standards.
There's all these different competing options out there with financial accounting.
We've got GAP and international equivalents and financial accounting standards board andstuff and there's some clarity.
(18:48):
Do we need more coordination about what the standards are for doing an AI audit?
I think we do need that.
And there are some organizations that are trying to do that.
I'm a fellow at For Humanity.
That organization is trying to do exactly this, have a central kind of repository ofcertification schemes that might be informed by laws or might be informed by standards,
(19:13):
but are sort of developed in a unique sort of coherent way.
But I don't think that we're going to get there.
I think if you look at cybersecurity, for instance,
There's just so many different certification schemes, so many different types of ways ofauditing types of audits.
You've got SOC 2, you've got ISO, there are other organizations.
(19:35):
Some companies have just had their own certification that they develop.
And I think we're going to see it evolve in that way.
Now, one thing I'll say is that the EU AI Act is so comprehensive in its requirements, orat least so onerous, maybe is the word that some people would use.
that it will probably be the most difficult quote unquote standard to adhere to.
(20:01):
And so if you adhere to the most rigorous standard, it's probably gonna cover you for 95 %of all of the other regulations.
And so I think we'll see people gravitating towards that as a benchmark.
then it'll be just filling in the gaps for some individual local law or individual kindof...
(20:23):
association that might want to put something together.
Although it's interesting you mentioned the work that you have to do internally on R &Dand assessment on standards with something like the AI Act, there's still tremendous gaps
in terms of the interpretive guidance and what those standards mean and so forth.
So do firms like yours really have to fulfill a role for making some of those decisionsunder conditions of uncertainty as these standards are evolving?
(20:52):
Absolutely.
we do have to interpret if there aren't standards and the EU will have, Sensenlack isgoing to come out with more detailed standards and we are sort of keeping track of that
and providing input when we can.
(21:12):
But a lot of times we are going to have to make some calls on our own.
What do we think is good enough given the current language of the law?
But I think that's okay because the level of maturity for most organizations frankly isnot anywhere near even the laws it's written, much less some detailed standards.
(21:33):
And so our sort of general strategy, and I think it's a good strategy for otherorganizations that they want to try to audit, is to fall back on transparency and say,
listen, this is exactly what we are using as a...
as a standard, these are the criteria that we use, these are the procedures we wentthrough to check things and disclose that.
(21:55):
And then people will know exactly what happened and what you did.
And it may not be exactly compliant with the law as the new standards might indicate, butyou can always go back and redo it.
And it just, provides a lot more clarity on what exactly is going on.
(22:16):
about certifications and accreditations for the auditors themselves?
I know that's something you've been involved in, in the AI space.
So that's a difficult thing because there isn't a central body for that.
And so I mentioned for Humanity before, they have some sort of certifications based ontheir certification scheme.
So they'll kind of accredit auditors for their certification schemes.
(22:40):
And so we've, for instance, gotten some of those as our organization.
There's a new organization that I'm one of the founding members of, which is theInternational
Association of algorithmic auditors.
I might've gotten one of those words mixed up, but it's I AAA, I AAA.
And so the goal there is much more about identifying what are the capabilities.
(23:05):
It's in a professional association.
What are the capabilities we think auditors need to have and how could they acquire thesort of bona fides that they need in order to do this?
It might be, we haven't got to the point where we have an examination yet or anything likethat.
It probably will get to that point, but I have a feeling it will be a mix of things fromexperience to education.
(23:28):
We don't have any degrees in this at the moment.
That's something that will come.
I'm sure just like, just like financial auditing has accounting degrees that will lead tofinancial auditing that was going to happen eventually.
But right now it's a little bit of the wild West.
And I think we rely on our own training for our own auditors.
And so we have our own program.
(23:50):
where we've outlined the capabilities and we train internally our own auditors and we justmake sure that we are transparent about that and everyone can see what we are expecting in
terms of capabilities.
But eventually, probably everybody's gonna have to get some sort of certification wheneverone of these organizations kind of rises to the top in terms of credibility.
(24:12):
And then going back to when you answer the question about what makes a good audit, thinkthe third thing you said was independence.
Isn't there though an inherent conflict of interest if a company is paying you to providea service?
How can you provide enough of an assurance of independence in what you respond to them?
Yeah.
So we basically approach this the same way that, that financial auditors approach this orassurance professionals approach this.
(24:39):
So there's a, there's a standard kind of code of conduct.
And we, in fact, the standard we rely on is the same one that the, auditors rely on.
And so we look at, because this happens, of course, you know, a company pays Deloitte tocome audit their financial, they have to pay them, right.
Deloitte is not going to do it for free.
So there, there are.
(25:01):
different ways of mitigating the sort of fundamental issue of impartiality andindependence.
know, Sarbanes -Oxley came along and had some additional requirements there.
We keep an eye on that and make sure that we're following that as well.
But there, it's not easy and it's a difficult task.
(25:22):
And what makes it more difficult is it's not like financial auditing where you can show upand expect that the company understands how
bookkeeping works and understands how they need to track their financial transactions.
That's not the case here.
We are dealing with organizations that it's shifting so much that they don't know whatthey need to do.
(25:47):
And so we have to play a balancing act of education, like what is it that you need to do?
But then we also are gonna come in and ensure that you do that.
And that's not easy.
And I think that the level of independence is by necessity slightly less than it would beif there was a very strict standard that everybody understood and they go off and get it
(26:10):
done and we are only coming in to check their homework.
But I think that that's just a function of the time we're in right now.
And as this becomes more mature as a industry, we're gonna see that independence isactually not gonna be that different than
than financial auditing.
(26:31):
So right now, how do you convince clients or prospective clients to accept your approachto it, to, you know, the value of what you're doing, the way that you put it together for
sometimes it's difficult.
what are the challenges?
The challenges are the way we approach it is that we are not consulting them.
(26:55):
we're, are simply, they have to do all the work, right?
So they're the ones who do testing.
They're the ones who have to manage the risk.
We have governance requirements, like for New York, New York law doesn't require riskassessments or governance or any risk management at all.
We have that in our audits.
And so there's this super augatory nature of some of the things that they have to acceptthat they're going to be doing the work and we are going to be checking their homework and
(27:23):
validating the everything they've done, everything they said they did, they've actuallydone.
That's a tough call.
The way we, we convince them is that every law besides New York, every law that's coming
is going to require robust internal governance and risk management and technical testing.
They all are.
(27:43):
So if you were going to be a mature organization using AI in the near future, you have todo this anyway.
So why not start now where the bar is much lower, get your feet wet, get used to havingpeople hold your feet to the fire.
And that's going to mature you as an organization and then you can grow from there.
That's the, that's the selling point.
(28:05):
Not everybody wants that of course.
But at this point, we're only looking for the people who are willing to do that.
And so there's a filter effect a little bit.
And so there's probably a large swath of the market that is going to get serviced by otherkinds of organizations or other kinds of companies that don't approach audit in the same
way we do.
(28:29):
How broad do you think the practice of AI auditing is going to get?
Because obviously we're seeing now companies of every scale in every industry deploying AIsystems and often many different AI systems.
So do you envision a world where all those things are subject to auditing?
sure that would be good for a firm like yours from a business standpoint, but obviouslythere's some overhead at some point.
(28:53):
Yeah, so I don't think so.
think it's going to evolve into much more of a, and again, I don't know, like I don't havea crystal ball, but my hunch is that a lot of the processes for all of these algorithms
are gonna have to be automated and there will be centralized risk management andgovernance and testing of these systems that will be
(29:22):
more specific and more based on standards and that the role of the auditor is to auditthat system, that quality management system with the extra bits that are really relevant
for AI systems.
think that's going to be, because that's much more manageable.
don't have to check that every, you know, for this particular algorithm that you're usinginternally to do something is compliant.
(29:48):
What I'm checking is you have that program in place.
and you have the processes in place, the controls that are going to manage that.
Now, the caveat here is that some laws like the EU AI Act, there'll be high risk systemswhich have requirements for the AI system itself.
Luckily, a lot of those requirements are also organizational based.
(30:11):
do you have a risk, have you done risk management for that?
Right.
So one risk management system can cover multiple high risk algorithms.
The technical testing is going to be unique to a particular algorithm, but that can becentralized and audited at a sort of organizational level.
so anyway, that's what I see it's emerging as organizations get audited and the individualAI systems are being controlled internally in a robust way by those organizations.
(30:41):
And going forward, what do you see as the greatest challenge for the auditing community inAI or specifically for you at Babel?
So the biggest challenge is right now, as you alluded to, the explosion of standards andlaws.
there are just so many, so we spent a long time for New York City, for instance,developing how are we gonna test, and it's very simple, it's like one metric, disparate
(31:09):
impact.
How are we gonna test that?
How are we gonna provide assurance that they have actually done the right thing?
And it was a lot of development.
And if you look at these laws, like the digital services act, which is already out and,and, and, and people are being audited for that.
The EU AI act, Colorado, all the state laws, some of the federal guidance that is, that iscoming out.
There's just so many kind of normative frameworks for, for what you need to do.
(31:35):
That it's going to be difficult to have that level of development work.
And so the way we're approaching this is that we are focusing on.
our internal process of like, how do we provide assurance?
How do we audit these systems and these organizations in general based on that normativeguidance?
What is, and then it's like, what is the process?
(31:56):
If someone comes to us with this brand new thing, like in Taiwan, we need to, this is theregulation and here are the standards.
What's our process for determining is this auditable?
What are the capabilities we need to audit that?
What are the extra resources we might need to
to execute that audit at what level of assurance.
And that's sort of a more unified framework that we're working on.
(32:20):
And that's the way it's probably going to have to proceed until there's like really nichedown into like, I'm an auditor for just this one law and then just this one industry.
have to be kind of, there has to be that flex.
That's the challenge, that flexibility.
Great, well this has been a really fascinating conversation.
Really interesting to see how the whole space of all is going forward.
(32:43):
Jay, thank you very much for your time.
Thank you so much for having me.
Popular Podcasts
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Therapy Gecko
An unlicensed lizard psychologist travels the universe talking to strangers about absolutely nothing. TO CALL THE GECKO: follow me on https://www.twitch.tv/lyleforever to get a notification for when I am taking calls. I am usually live Mondays, Wednesdays, and Fridays but lately a lot of other times too. I am a gecko.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.