Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:01):
This article was publishedin December 2024.
It's titled AlgorithmIntegrity, Audit versus Review.
Very simple background to this.
There's lots of, differencesin the terminology that's used.
Some people call things audits,them reviews, some people call
audits reviews, and otherscall reviews audits, etc.
(00:22):
The terminology It is importantto make sure that we all are
talking the same language,but really clarity about
the deliverables is moreimportant when commissioning an
algorithm integrity assessmentof some sort, whether you
call it an audit or review,making sure that you review.
Understand and document thosedeliverables is probably
the more important thing.
(00:43):
So I just wanted to explain,the norm from other domains
and how that potentiallytranslates into using
algorithms and reviewing them.
Okay, so here we go.
We don't need to always beprecise with our terminology.
But when you're commissioningan assessment to determine
whether your algorithm hasintegrity, you need to know
(01:06):
what to ask for to make surethat you get what you want.
So understanding thedistinction between an audit
and review is important.
But there are no standardizeddefinitions, so this
is often misunderstood.
Or at least understooddifferently by different people.
This article explores thisin more detail, then explains
what to focus on regardlessof the terminology used.
(01:31):
Reviews.
A review typically aims toidentify potential issues
before they become problems.
They are sometimes less formaland may not always follow
a standardized methodology.
So they include thefollowing five as examples.
Number one, self assessments.
Internal teams evaluate theirown AI systems for potential
(01:53):
risks or areas of improvement.
2.
Peer reviews.
Colleagues or other teamswithin the organization
examine the AI system.
3.
Informal external reviews.
Consultants or experts providefeedback without following
a strict audit protocol.
(02:13):
4.
Agreed upon procedures.
More formal but arenot full audits.
They involve specific proceduresas agreed, usually conducted
by an external party butcould be internal, a report of
findings without an opinion.
And five, internal audits,conducted by teams within
(02:34):
the organization that arededicated to conducting internal
audits, . Can be more formalthan other types of reviews.
In practice, they mayalso be considered audits,
as the name suggests.
Could also be agreedupon procedures.
May result in an opinion orconclusion but not necessarily
for external distribution.
(02:56):
In short, any type of assessmentcan be called a review.
In fact, an audit couldbe considered a special
type of review with adistinct type of output.
Let's talk about audits then.
An audit is structured andmost often conducted by
independent external parties.
When conducted by an externalparty, they are sometimes
(03:18):
called external audits.
Before we get into whatan external audit is.
Let's explore a coupleof other types of audits
in inverted commas.
internal audits.
An audit is sometimesperformed by independent
internal auditors.
with opinions or aconclusion and reports
that may be distributed.
(03:40):
However, as outlined earlier,not all internal audits
are audits and most areother types of reviews.
Then there areadversarial audits.
This is another type ofreview that is sometimes
called an audit.
These are performed without acontract in place, usually by
non profit entities, withoutengaging with the entity.
(04:02):
Beyond AI or algorithmreviews, these are not
typically called audits.
The terminology is loose.
For example, the EU AIAct expects developers to
include adversarial testingof their own systems.
Importantly, you don't typicallycommission an adversarial
audit of your own systemsby definition, so let's
(04:25):
ignore this item for now.
This all becomesquite confusing.
So let's explain whata real audit, inverted
commas, comprises.
External audits are the mostwidely recognized reference
to audits, are performedwith a contractual agreement
in place between the entityand the auditor, Performed
(04:48):
by independent entities whooffer auditing as a service.
So this includes traditionalconsulting firms and specialized
auditing companies in thepublic sector can be performed
by supreme audit institutions.
So national audit officersor state audit officers.
are often performed forregulatory compliance or
public accountability, aretypically more rigorous and
(05:11):
impartial, and the result isan opinion or conclusion, and
often distributed outside ofthe organization being audited.
This last point is important.
An audit results in anopinion, often with a report
that can be distributed.
So the key differences, whileaudits are really just a special
type of review, there are somekey differences between audits
(05:34):
and other types of reviews.
Formality, so audits followstructured methodologies
and adhere to specificstandards or regulations.
Other types of reviewscan be less formal.
Independence, audits requireindependence and objectivity.
Other types of reviews canbe undertaken internally.
(05:55):
Scope.
Audits tend to be comprehensive,including technical, ethical,
and governance aspects.
Other types of reviews may focuson specific areas of concern.
Documentation.
Audits require extensivedocumentation and
evidence gathering.
Other types of reviewsmay be less rigorous.
(06:15):
The result.
Audits produce opinionsor conclusions.
Other types of reviews mayproduce facts or findings
without an opinion.
Deliverable.
Audits result in formalreports and are usually
used for compliance orcertification purposes.
Other types of reviews typicallylead to internal recommendations
(06:37):
with reports for internal use.
Again, the result anddeliverable are the most
important distinctions.
Audits produce formal reportswith opinions or conclusions.
In some cases these reportsare shared with other
organizations like clients oreven out in the public domain.
The bottom line, if you forgetthe rest of the complexities
(07:00):
in this article, make surethat the deliverable is clearly
understood and documentedbefore the assessment starts.
Whatever you decide tocall them, it is useful
to know what to ask for.
But regardless of what term youuse to describe the assessment,
make sure that the nature ofthe result and deliverable
are clear and documented.
(07:21):
For instance, a bankcommissioning an evaluation
of its AI driven loan approvalsystem would likely require
an audit if the report is tobe shared with regulators.
Here the bank and auditor willagree upfront what to opine
or conclude on and who thereport may be distributed to.
(07:42):
In writing.
On the other hand, Aninsurance company looking
to improve its internalunderwriting process might
opt for a less formal review.
Here the insurer and thereviewer will agree up front
that the report will containfacts or findings And will
not be for distribution.
(08:02):
This may also be in writing.
The fact that there willbe a report may also be in
If you want a formal opinionin a report that can be shared,
you probably want an audit.
In most other cases, adifferent type of review
will meet your needs.
Regardless of what terminologyyou use, make sure that
(08:23):
the deliverable is clearlyunderstood and documented.
Will there be a report?
Will the report detailfindings, facts, an
opinion or a conclusion?
Will the report be forinternal use, for a select
predefined set of stakeholdersand or third parties, or for
publishing in the public domain?
(08:44):
Documenting the answers tothese questions up front
will help ensure that youget what you asked for.
That's the end of the article.
Thanks for listening.