Article 15. Algorithm Integrity Documentation - Getting Started

Spoken (by a human) version of this article.

Documentation makes it easier to consistently maintain algorithm integrity.

This is well known.

But there are lots of types of documents to prepare, and often the first hurdle is just thinking about where to start.

So this simple guide is meant to help do exactly that – get going.

About this podcast

A podcast for Financial Services leaders, where we discuss fairness and accuracy in the use of data, algorithms, and AI.

Hosted by Yusuf Moolla.
Produced by Risk Insights (riskinsights.com.au).

Episode Transcript

(00:00):
This article was publishedin November 2024.
It's titled Algorithm IntegrityDocumentation Getting Started.
The background to this is thatseen a few situations recently
where Documentation wasn't inplace, and the question that
seems to come up quite oftenis, do we start with putting

(00:21):
documentation together ifwe've got nothing in place
yet, Now, obviously, there'slots that goes into putting
integrity related documentationtogether, but The intent of
this article was to Identifysome key items that are needed
to get started, and even ifyou've started already, , what

(00:43):
are the types of thingsthat we often, expect to see
So here we go.
Documentation makes iteasier to consistently
maintain algorithm integrity.
This is well known, butthere are lots of types
of documents to prepare.
And often the firsthurdle is just thinking
about where to start.

(01:04):
So this simple guide is meantto help do exactly that.
Get going.
Starting points.
here are five key items toget started on if you don't
yet have anything in place.
If you have already startedor have some in place,
consider whether youhave each of these five.
One, Algorithm inventory.

(01:27):
Risk assessment.
Policies, procedures,standards and controls.
Design and developmentdocumentation.
Technical documentation.
Let's explore eachin a bit more detail.
So 1.
Algorithm inventory.
This is a list of algorithmsin use and or planned.

(01:50):
And it includes a cleardocumented statement of each
algorithm's objective, itspurpose and its intended use.
This provides an overviewof your algorithmic
slash AI landscape.
Number two, risk assessment.
A document outlining theidentified risks and how
these will be managed.

(02:11):
If the algorithm affectscustomers, we may want
to make this a risk andimpact assessment, adding
impact to reflect theimportance of ensuring
customer safety and fairness.
This helps prioritizeefforts, ensure responsible
use, and keep track of thekey matters to focus on to
maintain customer safety.

(02:31):
3.
Policies, Procedures, Standards,and Controls The principles
and expectations for all stagesof the algorithm lifecycle
Together with step by stepinstructions, mandatory rules,
and then measures to ensurethe above are all upheld.
For these, we set a regularreview cycle, so for example,

(02:53):
at least annually, toensure they remain current.
They help ensure consistency.
and reduce ambiguity.
Four, design anddevelopment documentation.
key decisions, methodologies,assumptions and rationales
during algorithmdesign and development.

(03:14):
Keep in mind that thisis not a once off.
Algorithm design and developmentis not a one and done.
As the algorithms willchange regularly, this
ensures transparency andaids in future improvements.
5.
Technical documentation.
Describes the AI system'sdesign, specifications,

(03:35):
algorithms, datasources, testing methods.
Performance metrics, knownlimitations and workarounds.
Ideally, this will includea version history and audit
trail reflecting key changes.
This supports maintainabilityand compliance efforts.
So that's the list of five.
There's a few importantpoints to note.

(03:57):
this is certainly not acomprehensive list, it
is just a starting point.
Involving relevantstakeholders in creating and
maintaining these can helpensure coverage and buy in.
These should be livingdocuments, regularly
reviewed and updated asyour algorithms evolve.
Aim for accessibilityand readability.

(04:18):
There is value in the documentsbeing understandable to both
technical and non technicalstakeholders where relevant.
In some cases, the detail,that is also important,
may mean that the audienceis technical folk, so they
shouldn't be diluted either.
Store them in an accessiblelocation, but without

(04:39):
risking security, privacyand confidentiality.
There will be otherdocumentation.
What exactly these are willdepend on several factors.
The factors determiningwhat these are include
organizational expectations,the nature, context and purpose
of each algorithmic system,and compliance obligations.

(05:01):
Nice and short.
That's the end of the article.
Thanks for listening.