Article 22. Algorithm Integrity: Third party assurance

Spoken by a human version of this article.

One question that comes up often is “How do we obtain assurance about third party products or services?”

Depending on the nature of the relationship, and what you need assurance for, this can vary widely.

This article attempts to lay out the options, considerations, and key steps to take.

TL;DR (TL;DL?)

• Third-party assurance for algorithm integrity varies based on the nature of the relationship and specific needs, with several options.
• Key factors to consider include the importance and risk level of the service/product, regulatory expectations, complexity, transparency, and frequency of updates.
• Standardised assurance frameworks for algorithm integrity are still emerging; adopt a risk-based approach, and consider sector-specific standards like CPS230(Australia).

To subscribe to the weekly articles: https://riskinsights.com.au/blog#subscribe

About this podcast

A podcast for Financial Services leaders, where we discuss fairness and accuracy in the use of data, algorithms, and AI.

Hosted by Yusuf Moolla.
Produced by Risk Insights (riskinsights.com.au).

Episode Transcript

(00:14):
This article was publishedin February 2025.
It's titled Algorithm Integrity,Third Party Assurance.
The long and short of this isthird party assurance for algorithm
integrity varies based on the natureof the relationship and your specific
needs and there are several options.
Some of the key factors to considerinclude the importance and risk

(00:34):
level of the service or product.
Regulatory expectations, complexity,transparency, and frequency of updates.
Standardized assurance frameworksfor algorithm integrity are still
emerging, so you may want toadopt a risk based approach and
consider sector specific standards.
For example, in Australia we have CPS 230.

(00:56):
Here's the full article.
One question that comes up oftenis, how do we obtain assurance about
third party products or services?
Depending on the natureof the relationship.
And what you need assurance for.
This can vary widely.
out the options, considerations,and key steps to take.

(01:17):
Nature of the relationship.
So this is about what the thirdparty does for you and how.
For example, they may provide oneor more of these seven examples.
Number one, hosting services.
Number two, custom applications.
Number three, off the shelfapplications that you configure.

(01:37):
Number four, custom models or algorithms.
Number five, off the shelfmodels you install and configure.
Number six, off the shelf modelsor algorithms accessed via APIs, so
that's Application Program Interfaces.
7. Developers that work under yourdirection So, these are just examples

(01:58):
and each of these relationshipspresents unique challenges for
ensuring algorithm integrity.
Let's talk about typesof third party assurance.
The assurance methods availablevary in their reliability and depth.
Here are the main types, not allof them, but the main types, from
most to least reliable or usable.

(02:22):
There's five here.
The first is independentthird party assurance reports.
For example, SOC 2.
These provide an independentdetailed assessment of the third
party's controls and processes.
Number two, certifications.
For example, ISO 27001 42001.

(02:45):
These are not ascomprehensive as SOC reports.
and typically not reliedon for external audits.
They indicate a level of compliancewith recognized standards.
Number three, internal audits canprovide insights but lack the true
independence of external assessments.

(03:05):
Number four, self assessments.
These offer a starting point forunderstanding the third party's
approach to algorithm integrity.
And number five, and this is in herefor completeness, it's not quite an
assurance method Vendor questionnaires.
So these are useful for gatheringbasic information and they
are the least reliable formof inverted commas assurance.

(03:29):
Just a note here about SOC 2 reportsor third party assurance reports.
We prefer them because they aremore reliable, comprehensive and
useful than any of the other types.
But while SOC 2 and similar reportsexist for information security and
other areas, SOC There isn't currentlya standardized, widely accepted,

(03:50):
independent assurance report specificallytailored to algorithm integrity.
This is an emerging area and as theimportance of algorithm integrity grows,
we expect to see the development of suchspecialized frameworks in the future.
Here are some factors to consider.
So with the nature of therelationship and the options in

(04:11):
mind, consider these questions.
How important is the service or product?
How risky is the use ofthe service or product?
What are the associated risks?
So for example, potential impacton stakeholders, being customers or
employees, reputational risk, riskof financial loss, etc. What are

(04:33):
the security or privacy concerns?
What, if any, are the historicalissues with the service or product?
What are your peers, that'sothers in your industry, doing?
What do regulators expect you to do?
How complex is the service or product?
How transparent andexplainable is the algorithm?

(04:56):
How often is the algorithmupdated or changed?
These factors or questions areparticularly relevant in banking and
insurance, where algorithms oftenhandle sensitive customer data,
make, or are relied on to make,Critical decisions and are subject
to strict regulatory oversight.

(05:16):
We spoke a little bit aboutwhat regulators expect and
here's a quick note about that.
In Australia, banks and insurancecompanies need to consider various laws.
there are also prudential standardsthat relate to third parties,
like CPS 220, 230, 234, etc.
CPS 230 is quite specific, certainservice providers must be classified

(05:40):
as material service providers unlessthere is a justification for exclusion.
For banks, this includes third partiesthat provide credit assessment,
funding and liquidity management.
and mortgage brokerage services.
For insurers, this includes thirdparties that provide underwriting,
claims management, insurancebrokerage, and reinsurance.

(06:04):
And finally, here arethe key steps to take.
a note, this article is aboutobtaining assurance, but
there are some prerequisites.
Without them, an assurance approachwill be difficult to execute.
There's five steps in this short list,it's not a complete list, but, the first
of those five is policy, so formallydocument your approach to third party

(06:26):
due diligence, risk assessment, riskmitigation, ongoing monitoring and review.
Number two, catalog, keep a recordof all third party arrangements,
risks and risk management practices,including the frequency of reassessment
and the nature of ongoing reviews.
Thank you.
Number three, regulatory,check the specifics of relevant

(06:50):
compliance expectations.
So some of them like CPS230Australia can be quite extensive.
Procurement, before entering into a thirdparty arrangement, make sure that you
can obtain the assurance that you need.
What you need will depend on thenature of the relationship and the
factors to consider that we spokeabout earlier those questions.

(07:11):
And number five, monitor and review.
Get copies of the agreed reports,analyze them, and consider what
steps you may need to take if thereare critical problems highlighted.
That's the end of this article.
Thanks for listening.