March 28, 2021 • 10 mins
In using data for audits, quality assurance of our work is important.
In this episode, we discuss:
1. The four main QA steps
- Technical QA
- Functional QA
- Business QA (validation)
- Audit QA
2. Why we need a QA Plan
This is explored in more depth in our book, here: https://riskinsights.com.au/the-data-confident-internal-auditor
About this podcast
The podcast for performance auditors and internal auditors that use (or want to use) data.
Hosted by Conor McGarrity and Yusuf Moolla.
Produced by Risk Insights (riskinsights.com.au).
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Narrator (00:19):
You're listening to The Assurance Show.
The podcast for performanceauditors and internal auditors
that focuses on data and risk.
Your hosts are Conor McGarrityand Yusuf Moolla.
Conor (00:20):
Today Yusuf, we're going to be speaking about quality
assurance of our data work orQA, as we like to refer to it in
audit land.
But let's start off with afundamental.
Why is QA over our workimportant?
Yusuf (00:31):
Most auditors will have QA steps that they undertake for
all of the general audit workthat is being conducted.
We've spoken before aboutchecking quality of the data
that we use.
It's equally important to ensurethat any analysis that we do
with the data that we have andoutcomes that we produce that we
have a level of QA over it.
(00:51):
As we've said before and as ourguests have said before, data is
really just another way in whichwe can understand the subject
matter and then produce evidenceto inform our findings or audit
work, but also to back up someof our findings.
And so, if it is just anotherform of evidence or something
that we use as input, then QAover that work that we do
(01:12):
becomes as important as QA ofany other audit work that we do.
One thing that we areconsistently in agreement on-
that QA is important.
We've all had to sit through ordeal with lots of red pens in
the old days, lots of reviewnotes nowadays, comments and
track changes in our documentsfrom people that review our work
(01:33):
so we've all been exposed to it.
There're a few differences inthe way in which data work, QA
works, so we'll talk throughthat.
Conor (01:41):
QA is a pretty broad term and covers a lot, but they're
actually different types of QA.
There are four main types thatwe're going to discuss today.
The first one is technical QA,what that means and when it's
applied.
The second one we'll talk aboutis functional QA and what that
looks like.
And the third one will bebusiness QA.
And lastly, audit QA.
We'll step through each of thosein detail.
So, let's start off withtechnical QA.
(02:03):
What does that mean, and when isit used?
Yusuf (02:05):
Technical QA is usually the first step.
This involves self-review.
So reviewing your own work, thenalso getting a peer to have a
look at what you've done.
Technical QA is ensuring thatwhat you've done technically
makes sense.
So when you complete some of thework that you've done, you want
to make sure that the data thatyou've used is and remains
(02:29):
complete, that it is and remainsaccurate and that it is relevant
to the period that you'vecovered.
A few things that you want tomake sure about that you've used
all of the relevant records,that you've not excluded any
records, that you've reconciledinputs where you are able to.
You haven't altered any of theaccuracy of the data by double
counting any data or conductingany joins that would multiply
the data out.
You have not performed anycalculations that alter accuracy
(02:51):
of the data and you've correctlyperformed transformations, have
considered all of the relevantdates, etc.
y ourself before passing it ontoa colleague to have a look.
Ideally, you'd get somebodyindependent of the work that
you've done, potentiallysomebody at the same level as
yourself to have a look at thatand ask questions around how
you've technically combined thedata, how you've joined it, et
(03:13):
cetera.
The reason you want to getsomebody else involved is just
like with any other QA, youoften get so deep into some of
these technical matters that youcan't always see the mistakes
that you might've made.
Conor (03:24):
It talks about the importance of self-review really
in all of the work that you'vedone.
Every auditor worth their saltwould obviously be already doing
that.
But also the benefit to begained from peer review.
Let's talk about the second QAprocess, and that's functional
QA.
What does that involve?
Yusuf (03:39):
Again, functional QA is performed first by yourself, so
that's the second step that youwould conduct yourself on your
data work.
And then by a peer.
It might not be somebody at thesame level as you, but it would
be somebody that understands thebusiness process or maybe
understand how the broaderbusiness operates.
Because functional QA is aboutensuring that you remain
objective and credible in thework and that also your work
(04:01):
makes sense, Are your resultsbelievable?
Can you believe them and willothers believe them?
Are your results rational?
With an understanding of thebusiness process overall, does
it make sense?
An example of this might besomething like, I've got a whole
bunch of data and I imported itand technically the data makes
sense and I've not excluded anydata that I shouldn't have
excluded, I've got the righttime period, et cetera.
(04:23):
When you do a functional QA,you're looking at things like we
know overall that there might bea particular trend.
It is expected.
There may have been a newproduct that has been introduced
and that new product that wasintroduced may have generated a
certain level of revenue.
Can you see that revenuereflected in the results?
Can you see that particularproduct reflected in the
results?
Is there something else, eitherseasonally or otherwise that
(04:46):
somebody with an understandingof the business process, or the
business more generally, wouldknow about that would then look
for in that data?
So in essence, does it makesense?
Conor (04:55):
Then you ask questions like, this is what we're seeing,
is this important?
Does this meet expectations?
Is there a context here thatwe're missing based on what
we're seeing?
Those types of things.
So that takes us neatly on tothe third important aspect of QA
and that's business QA.
And I think you've led intothat.
So what does that involve?
Yusuf (05:11):
So business QA.
This is quite similar tofunctional QA, but the reason
why you do it, one of the mainreasons that you do it is that
you want to have acceptance fromoutside of the audit team.
That's one of the main reasons.
The other is that there may bethings that the audit team
aren't aware of because they'renot that close to the business
(05:31):
process that you would wantsomebody that is a potential
representative of the businessor a subject matter expert to
help explain.
So again, the questions that youwant to answer here are: are the
results correct?
Are we missing any context wherethe business process may have
changed recently?
Or the audit team just aren'taware of that particular matter.
The key question we want toanswer here is will our findings
(05:54):
be accepted by the business?
So, if the work that we've donedoesn't pass this step, then we
need to think about things alittle bit differently.
Or we need to get other peopleto think about things as in our
stakeholders need to beeducated.
The challenge with skipping thisstep is that we may have a lot
of pushback when we producingour results, and because a lot
(06:15):
of the data work that we dodoesn't necessarily hit the
final result immediately butforms part of the report or
contributes to the report, orthere's further follow up work
that is done after this, we wantto make sure that what we're
saying from a data perspectiveis accepted by the business
early on.
Conor (06:32):
In the business QA step we're trying to again to
understand the context whetheror not there're for example
exceptional circumstances orsentinel events that may have
impacted some of the data thatwe've used or analyzed.
Secondly, we're applying thereasonableness test to help with
acceptance of what we're seeing.
Talking to the business andmaking sure what we're seeing
here is valuable, useful andpractical taking forward.
(06:53):
That takes us on to number fourand that's audit QA, so what are
we talking about here?
Yusuf (06:58):
This is the final bit of QA that you do.
This is"QA-ing" the QA.
This is where somebody withinthe audit team leader in the
team, maybe the head of youraudit function if you're in a
smaller team, or if you're in areally large team that might be
a team member that focuses onquality.
Some teams actually have a wholeQA function that's built into
the internal audit team orperformance audit team.
(07:18):
What you want to do here is makesure that there has been a QA
plan established for the workthat you've done.
We're talking about this last,but before you would start any
QA, you'd put together a QAplan.
The QA plan doesn't have to bevery detailed, but you need to
outline some steps.
The audit QA will check whethera QA plan has been established.
Make sure the QA has beenundertaken at the right time
(07:40):
with the right level of rigor.
And then the key thing that theaudit QA step will cover off is
does the analysis support theaudit result?
The person that's conducting theaudit QA will check whether
technical QA was robust, whetherfunctional QA was robust, and
whether the business validationwas correct.
But also then check against theaudit report.
(08:00):
Is what we're seeing in the datawork that was performed, is that
actually reflected in the auditresult appropriately?
Are we missing anything?
Your normal working papers willhave cross references.
What have you found versuswhat's in the report and making
sure that everything found is inthe report or is explained, and
everything in the report wasfound in the data work.
So similar to that does theanalysis actually support that
audit result?
Conor (08:21):
Is it possible to pass the technical QA then pass the
functional QA of the businessvalidation to the business QA?
And then perhaps still not bemeeting the audit objective, and
that's maybe where the audit QAcan come in at the end.
Is that a possibility?
Yusuf (08:36):
Yes.
The audit objective overall, andthen broken down into the
hypotheses that have beenidentified.
If you only cover let's say youhave an overall objective, and
as part of that you looking atfive hypotheses.
If you only cover four of them,you've technically done the
right thing.
Functionally you've done theright thing.
The business won't know thedifference, but overall, you
(08:57):
haven't covered off on all thehypotheses that need to be
covered off.
So this is that sort of finalcheck.
Conor (09:01):
Okay, fantastic.
Yusuf (09:03):
One final note is we spoke about it briefly is what's
your QA plan?
What's your quality plan?
Do you want to do this initiallyto be able to check off that
each of those steps have beenundertaken.
And it's particularly importantwhen you're doing work
iteratively because you canquite often forget about or miss
a step when you're going throughan iterative style approach.
(09:24):
Doesn't have to be very detailedbut we're just creating an
overall plan that we can use toensure that we are covering off
each of those steps within eachof the iterations that we have,
and that the individuals thatare performing those QA steps
understand what is expected ofthem at each of those steps.
You could be in functional QAstage and the person that's
(09:44):
doing the QA may not remember orunderstand what exactly is
expected of them.
By creating a plan, it makes ita lot easier to explain what is
expected of each of theindividuals undertaking the
different types of QA, that thenjust leads to a more complete
result.
Conor (09:59):
So we covered the four components of QA there.
Technical QA and what that wasand when it's used.
Functional QA and they're movingon to the business validation,
which is business QA and thenthe overall audit QA, which is
your catch all at the end tomake sure that all the previous
steps have been taken to theextent necessary and that you're
meeting the audit objective.
Yusuf (10:17):
That's it.
Thank you.
Conor (10:18):
Thanks Yusuf.
You're listening to The Assurance Show.
The podcast for performanceauditors and internal auditors
that focuses on data and risk.
Your hosts are Conor McGarrityand Yusuf Moolla.
Conor (00:20):
Today Yusuf, we're going to be speaking about quality
assurance of our data work orQA, as we like to refer to it in
audit land.
But let's start off with afundamental.
Why is QA over our workimportant?
Yusuf (00:31):
Most auditors will have QA steps that they undertake for
all of the general audit workthat is being conducted.
We've spoken before aboutchecking quality of the data
that we use.
It's equally important to ensurethat any analysis that we do
with the data that we have andoutcomes that we produce that we
have a level of QA over it.
(00:51):
As we've said before and as ourguests have said before, data is
really just another way in whichwe can understand the subject
matter and then produce evidenceto inform our findings or audit
work, but also to back up someof our findings.
And so, if it is just anotherform of evidence or something
that we use as input, then QAover that work that we do
(01:12):
becomes as important as QA ofany other audit work that we do.
One thing that we areconsistently in agreement on-
that QA is important.
We've all had to sit through ordeal with lots of red pens in
the old days, lots of reviewnotes nowadays, comments and
track changes in our documentsfrom people that review our work
(01:33):
so we've all been exposed to it.
There're a few differences inthe way in which data work, QA
works, so we'll talk throughthat.
Conor (01:41):
QA is a pretty broad term and covers a lot, but they're
actually different types of QA.
There are four main types thatwe're going to discuss today.
The first one is technical QA,what that means and when it's
applied.
The second one we'll talk aboutis functional QA and what that
looks like.
And the third one will bebusiness QA.
And lastly, audit QA.
We'll step through each of thosein detail.
So, let's start off withtechnical QA.
(02:03):
What does that mean, and when isit used?
Yusuf (02:05):
Technical QA is usually the first step.
This involves self-review.
So reviewing your own work, thenalso getting a peer to have a
look at what you've done.
Technical QA is ensuring thatwhat you've done technically
makes sense.
So when you complete some of thework that you've done, you want
to make sure that the data thatyou've used is and remains
(02:29):
complete, that it is and remainsaccurate and that it is relevant
to the period that you'vecovered.
A few things that you want tomake sure about that you've used
all of the relevant records,that you've not excluded any
records, that you've reconciledinputs where you are able to.
You haven't altered any of theaccuracy of the data by double
counting any data or conductingany joins that would multiply
the data out.
You have not performed anycalculations that alter accuracy
(02:51):
of the data and you've correctlyperformed transformations, have
considered all of the relevantdates, etc.
y ourself before passing it ontoa colleague to have a look.
Ideally, you'd get somebodyindependent of the work that
you've done, potentiallysomebody at the same level as
yourself to have a look at thatand ask questions around how
you've technically combined thedata, how you've joined it, et
(03:13):
cetera.
The reason you want to getsomebody else involved is just
like with any other QA, youoften get so deep into some of
these technical matters that youcan't always see the mistakes
that you might've made.
Conor (03:24):
It talks about the importance of self-review really
in all of the work that you'vedone.
Every auditor worth their saltwould obviously be already doing
that.
But also the benefit to begained from peer review.
Let's talk about the second QAprocess, and that's functional
QA.
What does that involve?
Yusuf (03:39):
Again, functional QA is performed first by yourself, so
that's the second step that youwould conduct yourself on your
data work.
And then by a peer.
It might not be somebody at thesame level as you, but it would
be somebody that understands thebusiness process or maybe
understand how the broaderbusiness operates.
Because functional QA is aboutensuring that you remain
objective and credible in thework and that also your work
(04:01):
makes sense, Are your resultsbelievable?
Can you believe them and willothers believe them?
Are your results rational?
With an understanding of thebusiness process overall, does
it make sense?
An example of this might besomething like, I've got a whole
bunch of data and I imported itand technically the data makes
sense and I've not excluded anydata that I shouldn't have
excluded, I've got the righttime period, et cetera.
(04:23):
When you do a functional QA,you're looking at things like we
know overall that there might bea particular trend.
It is expected.
There may have been a newproduct that has been introduced
and that new product that wasintroduced may have generated a
certain level of revenue.
Can you see that revenuereflected in the results?
Can you see that particularproduct reflected in the
results?
Is there something else, eitherseasonally or otherwise that
(04:46):
somebody with an understandingof the business process, or the
business more generally, wouldknow about that would then look
for in that data?
So in essence, does it makesense?
Conor (04:55):
Then you ask questions like, this is what we're seeing,
is this important?
Does this meet expectations?
Is there a context here thatwe're missing based on what
we're seeing?
Those types of things.
So that takes us neatly on tothe third important aspect of QA
and that's business QA.
And I think you've led intothat.
So what does that involve?
Yusuf (05:11):
So business QA.
This is quite similar tofunctional QA, but the reason
why you do it, one of the mainreasons that you do it is that
you want to have acceptance fromoutside of the audit team.
That's one of the main reasons.
The other is that there may bethings that the audit team
aren't aware of because they'renot that close to the business
(05:31):
process that you would wantsomebody that is a potential
representative of the businessor a subject matter expert to
help explain.
So again, the questions that youwant to answer here are: are the
results correct?
Are we missing any context wherethe business process may have
changed recently?
Or the audit team just aren'taware of that particular matter.
The key question we want toanswer here is will our findings
(05:54):
be accepted by the business?
So, if the work that we've donedoesn't pass this step, then we
need to think about things alittle bit differently.
Or we need to get other peopleto think about things as in our
stakeholders need to beeducated.
The challenge with skipping thisstep is that we may have a lot
of pushback when we producingour results, and because a lot
(06:15):
of the data work that we dodoesn't necessarily hit the
final result immediately butforms part of the report or
contributes to the report, orthere's further follow up work
that is done after this, we wantto make sure that what we're
saying from a data perspectiveis accepted by the business
early on.
Conor (06:32):
In the business QA step we're trying to again to
understand the context whetheror not there're for example
exceptional circumstances orsentinel events that may have
impacted some of the data thatwe've used or analyzed.
Secondly, we're applying thereasonableness test to help with
acceptance of what we're seeing.
Talking to the business andmaking sure what we're seeing
here is valuable, useful andpractical taking forward.
(06:53):
That takes us on to number fourand that's audit QA, so what are
we talking about here?
Yusuf (06:58):
This is the final bit of QA that you do.
This is"QA-ing" the QA.
This is where somebody withinthe audit team leader in the
team, maybe the head of youraudit function if you're in a
smaller team, or if you're in areally large team that might be
a team member that focuses onquality.
Some teams actually have a wholeQA function that's built into
the internal audit team orperformance audit team.
(07:18):
What you want to do here is makesure that there has been a QA
plan established for the workthat you've done.
We're talking about this last,but before you would start any
QA, you'd put together a QAplan.
The QA plan doesn't have to bevery detailed, but you need to
outline some steps.
The audit QA will check whethera QA plan has been established.
Make sure the QA has beenundertaken at the right time
(07:40):
with the right level of rigor.
And then the key thing that theaudit QA step will cover off is
does the analysis support theaudit result?
The person that's conducting theaudit QA will check whether
technical QA was robust, whetherfunctional QA was robust, and
whether the business validationwas correct.
But also then check against theaudit report.
(08:00):
Is what we're seeing in the datawork that was performed, is that
actually reflected in the auditresult appropriately?
Are we missing anything?
Your normal working papers willhave cross references.
What have you found versuswhat's in the report and making
sure that everything found is inthe report or is explained, and
everything in the report wasfound in the data work.
So similar to that does theanalysis actually support that
audit result?
Conor (08:21):
Is it possible to pass the technical QA then pass the
functional QA of the businessvalidation to the business QA?
And then perhaps still not bemeeting the audit objective, and
that's maybe where the audit QAcan come in at the end.
Is that a possibility?
Yusuf (08:36):
Yes.
The audit objective overall, andthen broken down into the
hypotheses that have beenidentified.
If you only cover let's say youhave an overall objective, and
as part of that you looking atfive hypotheses.
If you only cover four of them,you've technically done the
right thing.
Functionally you've done theright thing.
The business won't know thedifference, but overall, you
(08:57):
haven't covered off on all thehypotheses that need to be
covered off.
So this is that sort of finalcheck.
Conor (09:01):
Okay, fantastic.
Yusuf (09:03):
One final note is we spoke about it briefly is what's
your QA plan?
What's your quality plan?
Do you want to do this initiallyto be able to check off that
each of those steps have beenundertaken.
And it's particularly importantwhen you're doing work
iteratively because you canquite often forget about or miss
a step when you're going throughan iterative style approach.
(09:24):
Doesn't have to be very detailedbut we're just creating an
overall plan that we can use toensure that we are covering off
each of those steps within eachof the iterations that we have,
and that the individuals thatare performing those QA steps
understand what is expected ofthem at each of those steps.
You could be in functional QAstage and the person that's
(09:44):
doing the QA may not remember orunderstand what exactly is
expected of them.
By creating a plan, it makes ita lot easier to explain what is
expected of each of theindividuals undertaking the
different types of QA, that thenjust leads to a more complete
result.
Conor (09:59):
So we covered the four components of QA there.
Technical QA and what that wasand when it's used.
Functional QA and they're movingon to the business validation,
which is business QA and thenthe overall audit QA, which is
your catch all at the end tomake sure that all the previous
steps have been taken to theextent necessary and that you're
meeting the audit objective.
Yusuf (10:17):
That's it.
Thank you.
Conor (10:18):
Thanks Yusuf.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.