June 20, 2021 • 19 mins
Christine (performance audit) and Xiaoyan (data analytics) are Directors with ANAO, the Australian National Audit Office.
In this episode we discuss:
- What The Australian National Audit Office (www.anao.gov.au) does
- The five ways ANAO uses data in its performance audits
- How performance audit teams work with the central data team
- Challenges with data preparation
- The benefits of a strong focus on data quality
About this podcast
The podcast for performance auditors and internal auditors that use (or want to use) data.
Hosted by Conor McGarrity and Yusuf Moolla.
Produced by Risk Insights (riskinsights.com.au).
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Narrator (00:07):
You're listening to the Assurance show.
The podcast for performanceauditors and internal auditors
that focuses on data and risk.
Your hosts are ConorMcGarrity and Yusuf Moolla.
Conor (00:19):
Today, we're gonna focus on performance auditing and
the use of data in auditing.
And we're going to welcometwo guests on the show from
the Australian National AuditOffice, the ANAO, Xiaoyan
Lu and Christine Chalmers.
Let's start with abrief introduction.
Christine, can you tell us alittle bit about yourself and
your role at the ANAO, please?
Christine (00:39):
I'm an audit manager in performance
audit at the ANAO.
I've been with the ANAOfor about two years.
Prior to that, I worked inresearch and evaluation for
many years in Australia,in Canada and in the UK.
And my background is inpolitical science and
(00:59):
business administration.
Conor (01:00):
Excellent.
And Xiaoyan, can you tellus a little bit about your
role and your background?
Xiaoyan (01:04):
I have been working for the Australian National Audit
Office in the last three years.
Prior to that, I workedfor another government
agency for 11 years.
And before that Iwas a social research
consultant for four years.
I have a background inapplied statistics and
quantitative research andmainly using data analytics
to inform government decision.
Christine (01:26):
Just a little aside on that, Xiaoyan and
I have actually known eachother for about 20 years.
We worked together many, manyyears ago, coincidentally,
probably about 2002in social research.
Conor (01:39):
Can you give us a little bit of a background
on the ANAO and its functions?
Christine (01:44):
The ANAO exists General in delivering audits.
The Auditor General isan independent officer of
the Australian Parliament,whose purpose is to
support accountabilityand transparency in the
Australian government sector.
And he does that throughindependent reporting to
the Parliament and in doingthat driving improvements in
(02:04):
public sector performance.
His work covers three areas,annual financial statements
audits of Commonwealth entities.
Secondly conducting performanceaudits of Commonwealth entities.
And then finally auditingthe annual performance
statements of Commonwealthentities, on request.
Our role is that oncea performance audit is
(02:26):
completed and approved bythe Auditor General, we
will table that report inthe Australian parliament.
The ANAO also shares keylearnings from performance audit
in quarterly audit insights,publications that are also
placed on the ANAO website.
Most people, ifnot everybody, will be
really familiar withfinancial statements audits.
But let's have a little bit of afocus on the performance audits.
(02:48):
How does the ANAO select whichperformance audits it will do?
Every year, the ANAO publishes something called
the annual audit work program.
And that is essentially a listof potential performance audits
that might be conducted over thecourse of the financial year.
The selection of the potentialaudits is based on firstly,
(03:12):
the interests and prioritiesof the Parliament of Australia.
The audit office or the auditorgeneral will attempt to provide
a balanced program of activityacross different factors.
And that includes the properuse and management of public
resources across the four E'sof efficiency, effectiveness,
economy, and ethics.
So we'll try to achieve abalance across those four E's.
(03:34):
It'll look at planning anddelivery in major areas
of public investment.
So examples from 2020,21 might include defense
capability, large scaleinfrastructure such as the
national broadband network,programs targeting Indigenous
Australians, for example.
We'll also look at themeasurement of performance
and impact against agreedprogram objectives.
(03:55):
So including in relation tothings like probity, integrity.
And this year, there was alsoa COVID-19 audit strategy.
And that's looking at specificrisks that the Australian public
sector faced during COVID-19,and the specific challenges
to policy design and servicedelivery during COVID-19.
So in terms of the process,generally, what happens is
(04:17):
there's an environmentalscan that's done by the
performance audit teamsin October to November.
That's developed intopotential topics in
December through February.
And then that's sent outfor consultation to the Joint
Committee of Public Accountsand Audit, and the individual
entities in March through June.
And the program istypically finalized in July.
Yusuf (04:39):
You mentioned that towards the end of the calendar
year, you would identify auditsand then you have approval
mid-year, are there anychanges that happen to that
for emergency or other reasons?
Christine (04:49):
Yeah.
So it's really just a listof potential audits that over
the course of the year couldbe adapted to emerging risks.
And that happens.
There can also bespecific requests made by
parliamentarians, for example,for the Auditor General to
examine certain topics and thoserequests will be considered.
Conor (05:09):
Can you give us a little bit more information
about how data is currentlybeing used by the ANAO for
for its performance audits?
Christine (05:16):
The first thing I would say is that we want to
define what we mean by datawhen you ask that question.
So we generally classifydata into two broad
types, unstructuredand structured data.
So by unstructured data, whatwe mean there is information
that's normally very text-heavy.
Although it might contain somenumerical data such as dates
(05:36):
and quantities it's uncollated.
It requires a lotof interpretation.
Some of it's available publicly.
So for example transcriptsfrom parliamentary or
public inquiries or mediaor corporate documents.
But most of it's not publicand that might include things
like correspondence, meetingminutes, organizational
charts, contracts, thosesorts of things or ministerial
(05:58):
briefs and cabinet documents.
And it's very timeconsuming to analyze.
We're fortunate at the ANAOto have access to a program
that we use to assist us withclassifying, searching and
storing that information.
That type of unstructureddata is probably the main
type of data that's usedby performance audit.
(06:19):
Structured data we woulddefine as data that's
generally collated.
It adheres to some kindof predefined data model.
And it's usually in a tabularformat with rows and columns
that are related to eachother, and it's usually
number heavy, although it cancontain text data as well.
So that kind of data is usedas well in performance audit.
There's five differentways in which we use data.
(06:41):
The first one is justfor descriptive purposes.
But secondly, we'll look atdata as a means of validating
an entity's public reporting.
So for example, what theymight have said in their annual
performance statements or intheir annual reports, we'll
use it to test hypothesisor explain why certain
outcomes may have occurred.
We will often use itto identify outliers.
(07:02):
And then finally we useit as a sampling tool.
So we might identifystratified random or targeted
sample based on some riskcriteria that we'll then
test in a more in-depth way.
Xiaoyan (07:15):
The consideration of using data for audit has
already been embedded inANAO's audit workflow for
both performance audit andfinancial statements audit.
For example for the performanceaudits during the audit's
scoping and planning phase, theaudit manager will be the first
point of contact, to coordinateand discuss any potential
(07:36):
involvement and the input fromthe data analytics expertise.
Christine (07:40):
What we find is that it can be quite complex when
data needs to be transformedinto something that's useful
and reliable for audit purposes.
So the majority of the workwe find is in relation to
data extraction, preparation,transformation, and cleansing,
that is really the huge effortthat needs to go into the data.
(08:02):
So there's a big cost benefitanalysis that needs to be done
by performance audit wheneverwe're looking to use structured
data, that's derived fromother systems just to determine
whether that extraction andpreparation process is going
to provide enough benefitto the audit in terms of
meeting the objective andanswering the audit criteria.
Yusuf (08:20):
We've seen the same thing in terms of data preparation.
Where do you findmore of a challenge?
Is it in preparing opendata for use or preparing
proprietary data for use?
Xiaoyan (08:30):
I think it's both.
Accessing business datainvolves both technical
challenge and non-technicalchallenges for us.
Just because you can accessit, it doesn't mean that
it is in a usable format.
Sometimes you get lots ofrelational data sets, but
as there is no data modelto help you identify the
relationships and sometimes youneed a specialist to transform
(08:51):
data into ways that can beeasily used by our auditors.
Christine (08:56):
With respect to data quality from performance audit
perspective, one of the problemsthat we encounter is that
process of preparing the data.
Yeah.
Given the amount of workthat's involved in doing that
can take so long that by thetime we're in a position to
draw a conclusion about howcomplete and reliable and valid
(09:18):
that data is it can be quitelate in the audit lifecycle.
So the lack of qualitydata is a real issue in
performance audit particularlywhere the data hasn't been
created for audit purposes.
And that can actually be anaudit finding in and of itself.
And the other thing too, isthat the data that we might
collect from one system, forexample, a case management
(09:40):
system is really only trulyuseful if it can be linked to
data from another system suchas financial management system.
It's very hard to predict at theoutset of an audit, the extent
of the data preparation, linkingchallenge and the extent of
the data validation challenge.
That is particularlydifficult for performance
auditors, who may notunderstand the technical data.
(10:01):
So it becomes a realchallenge, I guess, for
performance auditors to workout how to integrate data
analysis into their audit.
Conor (10:09):
So we've got all these challenges, but
obviously the benefits ofgetting some good data prep
and analysis are manifoldand can have great impact.
With that in mind, can youtell us about some of the
upcoming audits or projectsthat the ANAO is working
on involving data analysis?
Xiaoyan (10:25):
We announced last month that we are going to
do an information reporton Australian government
expenditure on grants.
Information report is notan audit and it is not
generally developed by anyperformance audit approach.
It is mainly based ondata available publicly,
and this is going to beour second information
(10:45):
report following anotherinformation report published
two years ago on Australianexpenditure on procurement.
Probably 90% of this informationreport work will be relying
on data analysis work.
We really look forwardto share this with the
public later this year.
Conor (11:02):
We've spoken there about the challenges, and we've
talked about some excitingupcoming projects using data.
What about your organization'soverall future ambitions
for data analytics?
Where do you see that going?
Xiaoyan (11:14):
At the moment, our office is in the process
of bringing Microsoft 365into our IT environment.
It will come with enterprisetools for our auditors
and analysts to usefor data analytics and
visualization functions.
This will hopefully equipour auditors in general
with technology for selfservicing analytics solutions
(11:36):
and our System Assurance andData Analytics group will
be focusing on providingmore advanced analytics
support and standardizedsolutions for our auditors.
Christine (11:45):
I think going forward performance audit really needs
to think more creatively aboutwhen and how to involve data
analytics and SADA in its work.
Both in terms of improving theefficiency and the effectiveness
of what we're doing.
The onus, I think, is onperformance auditors to
integrate data analystsinto their performance
audit team better.
Such that the data analyststruly understand the audit
(12:08):
objective, the audited entityand the business of that entity.
And a large challenge is findinga way for the performance audit
team and the data analyticsteam to work together well,
because there often, wefind, can be a bit of a chasm
between the two teams that'sdifficult to fill because the
performance audit team maynot have a good understanding
of the principles of dataextraction and preparation.
(12:31):
And conversely, the dataanalytics team can lack
knowledge of the auditobjectives the auditee
or the business area.
So I think that's one ofthe biggest hurdles that
we face is just bringingthose two teams together to
work effectively together.
Recently the ANAO moved fromassigned desks to activity-based
working arrangements to tryand facilitate that cross
(12:54):
collaboration between thedifferent teams physically
locating them near us byensuring that the data analytics
team is involved in keymeetings and conversations right
through the audit lifecycle,not just brought in mid audit.
So those are some of the thingswe're trying to do to increase
that collaboration between dataanalytics and performance audit.
Xiaoyan (13:12):
And our next goal is really to identify if there are
any opportunities to improvethe performance audit efficiency
by using data by starting torequest similar data early and
shortening the time on datadiscovery and acquisition.
Another opportunity is touse some of the high value,
common data sets which ispotentially relevant to
(13:34):
improve our efficiency inbuilding that understanding
and running analysis.
Narrator (13:39):
The Assurance Show is produced by Risk Insights.
We work with performanceauditors and internal auditors.
Delivering audits, helpingaudit teams use data
and coaching auditors toimprove their data skills.
You can find out more aboutour work at datainaudit.com.
Now, back to the conversation.
Yusuf (13:59):
Open data and improving the quality of
data that is available.
What role have you playedas an audit office in
helping to improve thatlevel of data quality?
Xiaoyan (14:10):
We did two information reports and following
our first release on ourreporting , We can definitely
see improvement in the dataquality in the Austender data.
I think the maturity ofgovernment agencies is improving
in producing and preparingdata over the last few years.
Most government entities havea data strategy in place and
(14:33):
as they are implementing datagovernance on how they prepare
and publish their data.
Through our informationreports we're promoting and
advocating for better quality.
Yusuf (14:45):
You mentioned that there's opportunities for
performance auditors to usedata more directly as opposed
to asking for everythingfrom the central team.
Do you have a view as to,where the gap has arisen?
So where it is that our useof data as auditors has not
progressed to the extent thatmaybe it should have and now
we almost have to play a littlebit of catch up in building
those skills within audit teams?
Xiaoyan (15:06):
This is a very good question.
The way I say this is youneed to vizualize a matrix.
On one axis you have thecapability of the audit
teams and whether auditorsalready have analytic skills.
And the other axis is thecomplexity of the analysis.
Some of the typical statisticalanalysis can be done using
(15:27):
Excel or visualization tools.
That can definitely behandled by the audit teams.
And there is no question abouttheir ability to do that.
But on the other side,these days because we use
more and more populationdata from the IT systems,
sometimes we are talkingabout millions of records.
If the data is not really usablethat's where my team comes in
(15:50):
to transform and prepare datafor the audit team to use.
And on the other sideSometime it's not about
their capability whetherthey can analyse data or not.
It's all about efficiency.
If we can do the samething much quicker.
And in our view, that's whereyou need to bring an analyst in
rather than using traditionalExcel to handle a large
(16:10):
volume of population data.
Christine (16:12):
One of the other ways in which I
think SADA and performanceaudit will work together
in future is identifyingopportunities for using data.
That might be an areawhere performance auditors
need more support.
They may not necessarilyunderstand, or be able
to visualize how data canbe extracted and used.
(16:34):
So that sort of collaborationright from the outset of
audit planning is reallyimportant, so that SADA can
help performance auditors,identify those opportunities.
Conor (16:45):
You mentioned there identifying those high value
data sets that maybe can beused for multiple projects.
And the audit teams wespeak to a lot of them are
going through that sameprocess or have done that.
What are some of the othertips and traps that other
audit agencies shouldconsider when using data?
Xiaoyan (17:00):
With the publicly available data sets, the
quality of them really varies.
For auditors, you reallyneed to make sure the
data has a high quality,is complete and accurate.
Do you have sufficientinformation for you to
verify the quality of databefore you can use it?
If we talk to our colleaguesacross, the audit sector,
(17:22):
there may be something wecan learn from each other.
Christine (17:24):
I have five tips for performance auditors.
The first one is to rememberthat a huge proportion of
the data analytics time forperformance audit is going
to be in data preparation andbudgets and timeframes need to
take that into consideration.
Secondly, I believe thatthe quality assurance and
validation of the data needsto be the first priority and
(17:48):
it needs to take place priorto any analysis that's done.
So establishing face validity ofthe data early on in the audit
process is really essential.
I'd also recommend maintaininga single source of truth.
So trying to integrate all yourdata into a single linked data
set will be the key to avoidingerror and other inefficiencies
(18:08):
in using that data.
Maintaining full recordsof all data preparation
and analysis procedures toenable quality assurance
work and internal audit.
And then finally just thepoint I was making before,
thinking about how you canfully integrate a data analyst
into the performance auditteam as early as possible
in the audit lifecycle.
Xiaoyan (18:28):
Last one is if analysts are not auditors, we need to
make sure whoever used theanalysis, has the capability
to interpret the data properly.
Conor (18:37):
Thank you, Christine and Xiaoyan for your time.
A few key takeaways for me.
When you're accessing datafor your projects, get your
request into the entity early.
The second thing was quality.
Do not skip any stepsaround quality because it
is so important and willpay dividends in the end.
Thirdly, when youget to your analysis.
Make sure you've documentedyour logic and that it's
(18:58):
reviewable by a peer or bya quality assurance person.
And lastly, and thiswas something that came
through quite strong wascollaboration as between the
data professionals and theperformance audit professionals
need to get that collaborationhappening as early as possible
at the start of every project.
Great conversationwith the ANAO.
Thanks again.
Xiaoyan (19:16):
Thanks to both of you.
Narrator (19:17):
If you enjoyed this podcast, please share
with a friend and rateus in your podcast app.
For immediate notificationof new episodes, you can
subscribe at assuranceshow.com.
The link is in the show notes.
You're listening to the Assurance show.
The podcast for performanceauditors and internal auditors
that focuses on data and risk.
Your hosts are ConorMcGarrity and Yusuf Moolla.
Conor (00:19):
Today, we're gonna focus on performance auditing and
the use of data in auditing.
And we're going to welcometwo guests on the show from
the Australian National AuditOffice, the ANAO, Xiaoyan
Lu and Christine Chalmers.
Let's start with abrief introduction.
Christine, can you tell us alittle bit about yourself and
your role at the ANAO, please?
Christine (00:39):
I'm an audit manager in performance
audit at the ANAO.
I've been with the ANAOfor about two years.
Prior to that, I worked inresearch and evaluation for
many years in Australia,in Canada and in the UK.
And my background is inpolitical science and
(00:59):
business administration.
Conor (01:00):
Excellent.
And Xiaoyan, can you tellus a little bit about your
role and your background?
Xiaoyan (01:04):
I have been working for the Australian National Audit
Office in the last three years.
Prior to that, I workedfor another government
agency for 11 years.
And before that Iwas a social research
consultant for four years.
I have a background inapplied statistics and
quantitative research andmainly using data analytics
to inform government decision.
Christine (01:26):
Just a little aside on that, Xiaoyan and
I have actually known eachother for about 20 years.
We worked together many, manyyears ago, coincidentally,
probably about 2002in social research.
Conor (01:39):
Can you give us a little bit of a background
on the ANAO and its functions?
Christine (01:44):
The ANAO exists General in delivering audits.
The Auditor General isan independent officer of
the Australian Parliament,whose purpose is to
support accountabilityand transparency in the
Australian government sector.
And he does that throughindependent reporting to
the Parliament and in doingthat driving improvements in
(02:04):
public sector performance.
His work covers three areas,annual financial statements
audits of Commonwealth entities.
Secondly conducting performanceaudits of Commonwealth entities.
And then finally auditingthe annual performance
statements of Commonwealthentities, on request.
Our role is that oncea performance audit is
(02:26):
completed and approved bythe Auditor General, we
will table that report inthe Australian parliament.
The ANAO also shares keylearnings from performance audit
in quarterly audit insights,publications that are also
placed on the ANAO website.
Most people, ifnot everybody, will be
really familiar withfinancial statements audits.
But let's have a little bit of afocus on the performance audits.
(02:48):
How does the ANAO select whichperformance audits it will do?
Every year, the ANAO publishes something called
the annual audit work program.
And that is essentially a listof potential performance audits
that might be conducted over thecourse of the financial year.
The selection of the potentialaudits is based on firstly,
(03:12):
the interests and prioritiesof the Parliament of Australia.
The audit office or the auditorgeneral will attempt to provide
a balanced program of activityacross different factors.
And that includes the properuse and management of public
resources across the four E'sof efficiency, effectiveness,
economy, and ethics.
So we'll try to achieve abalance across those four E's.
(03:34):
It'll look at planning anddelivery in major areas
of public investment.
So examples from 2020,21 might include defense
capability, large scaleinfrastructure such as the
national broadband network,programs targeting Indigenous
Australians, for example.
We'll also look at themeasurement of performance
and impact against agreedprogram objectives.
(03:55):
So including in relation tothings like probity, integrity.
And this year, there was alsoa COVID-19 audit strategy.
And that's looking at specificrisks that the Australian public
sector faced during COVID-19,and the specific challenges
to policy design and servicedelivery during COVID-19.
So in terms of the process,generally, what happens is
(04:17):
there's an environmentalscan that's done by the
performance audit teamsin October to November.
That's developed intopotential topics in
December through February.
And then that's sent outfor consultation to the Joint
Committee of Public Accountsand Audit, and the individual
entities in March through June.
And the program istypically finalized in July.
Yusuf (04:39):
You mentioned that towards the end of the calendar
year, you would identify auditsand then you have approval
mid-year, are there anychanges that happen to that
for emergency or other reasons?
Christine (04:49):
Yeah.
So it's really just a listof potential audits that over
the course of the year couldbe adapted to emerging risks.
And that happens.
There can also bespecific requests made by
parliamentarians, for example,for the Auditor General to
examine certain topics and thoserequests will be considered.
Conor (05:09):
Can you give us a little bit more information
about how data is currentlybeing used by the ANAO for
for its performance audits?
Christine (05:16):
The first thing I would say is that we want to
define what we mean by datawhen you ask that question.
So we generally classifydata into two broad
types, unstructuredand structured data.
So by unstructured data, whatwe mean there is information
that's normally very text-heavy.
Although it might contain somenumerical data such as dates
(05:36):
and quantities it's uncollated.
It requires a lotof interpretation.
Some of it's available publicly.
So for example transcriptsfrom parliamentary or
public inquiries or mediaor corporate documents.
But most of it's not publicand that might include things
like correspondence, meetingminutes, organizational
charts, contracts, thosesorts of things or ministerial
(05:58):
briefs and cabinet documents.
And it's very timeconsuming to analyze.
We're fortunate at the ANAOto have access to a program
that we use to assist us withclassifying, searching and
storing that information.
That type of unstructureddata is probably the main
type of data that's usedby performance audit.
(06:19):
Structured data we woulddefine as data that's
generally collated.
It adheres to some kindof predefined data model.
And it's usually in a tabularformat with rows and columns
that are related to eachother, and it's usually
number heavy, although it cancontain text data as well.
So that kind of data is usedas well in performance audit.
There's five differentways in which we use data.
(06:41):
The first one is justfor descriptive purposes.
But secondly, we'll look atdata as a means of validating
an entity's public reporting.
So for example, what theymight have said in their annual
performance statements or intheir annual reports, we'll
use it to test hypothesisor explain why certain
outcomes may have occurred.
We will often use itto identify outliers.
(07:02):
And then finally we useit as a sampling tool.
So we might identifystratified random or targeted
sample based on some riskcriteria that we'll then
test in a more in-depth way.
Xiaoyan (07:15):
The consideration of using data for audit has
already been embedded inANAO's audit workflow for
both performance audit andfinancial statements audit.
For example for the performanceaudits during the audit's
scoping and planning phase, theaudit manager will be the first
point of contact, to coordinateand discuss any potential
(07:36):
involvement and the input fromthe data analytics expertise.
Christine (07:40):
What we find is that it can be quite complex when
data needs to be transformedinto something that's useful
and reliable for audit purposes.
So the majority of the workwe find is in relation to
data extraction, preparation,transformation, and cleansing,
that is really the huge effortthat needs to go into the data.
(08:02):
So there's a big cost benefitanalysis that needs to be done
by performance audit wheneverwe're looking to use structured
data, that's derived fromother systems just to determine
whether that extraction andpreparation process is going
to provide enough benefitto the audit in terms of
meeting the objective andanswering the audit criteria.
Yusuf (08:20):
We've seen the same thing in terms of data preparation.
Where do you findmore of a challenge?
Is it in preparing opendata for use or preparing
proprietary data for use?
Xiaoyan (08:30):
I think it's both.
Accessing business datainvolves both technical
challenge and non-technicalchallenges for us.
Just because you can accessit, it doesn't mean that
it is in a usable format.
Sometimes you get lots ofrelational data sets, but
as there is no data modelto help you identify the
relationships and sometimes youneed a specialist to transform
(08:51):
data into ways that can beeasily used by our auditors.
Christine (08:56):
With respect to data quality from performance audit
perspective, one of the problemsthat we encounter is that
process of preparing the data.
Yeah.
Given the amount of workthat's involved in doing that
can take so long that by thetime we're in a position to
draw a conclusion about howcomplete and reliable and valid
(09:18):
that data is it can be quitelate in the audit lifecycle.
So the lack of qualitydata is a real issue in
performance audit particularlywhere the data hasn't been
created for audit purposes.
And that can actually be anaudit finding in and of itself.
And the other thing too, isthat the data that we might
collect from one system, forexample, a case management
(09:40):
system is really only trulyuseful if it can be linked to
data from another system suchas financial management system.
It's very hard to predict at theoutset of an audit, the extent
of the data preparation, linkingchallenge and the extent of
the data validation challenge.
That is particularlydifficult for performance
auditors, who may notunderstand the technical data.
(10:01):
So it becomes a realchallenge, I guess, for
performance auditors to workout how to integrate data
analysis into their audit.
Conor (10:09):
So we've got all these challenges, but
obviously the benefits ofgetting some good data prep
and analysis are manifoldand can have great impact.
With that in mind, can youtell us about some of the
upcoming audits or projectsthat the ANAO is working
on involving data analysis?
Xiaoyan (10:25):
We announced last month that we are going to
do an information reporton Australian government
expenditure on grants.
Information report is notan audit and it is not
generally developed by anyperformance audit approach.
It is mainly based ondata available publicly,
and this is going to beour second information
(10:45):
report following anotherinformation report published
two years ago on Australianexpenditure on procurement.
Probably 90% of this informationreport work will be relying
on data analysis work.
We really look forwardto share this with the
public later this year.
Conor (11:02):
We've spoken there about the challenges, and we've
talked about some excitingupcoming projects using data.
What about your organization'soverall future ambitions
for data analytics?
Where do you see that going?
Xiaoyan (11:14):
At the moment, our office is in the process
of bringing Microsoft 365into our IT environment.
It will come with enterprisetools for our auditors
and analysts to usefor data analytics and
visualization functions.
This will hopefully equipour auditors in general
with technology for selfservicing analytics solutions
(11:36):
and our System Assurance andData Analytics group will
be focusing on providingmore advanced analytics
support and standardizedsolutions for our auditors.
Christine (11:45):
I think going forward performance audit really needs
to think more creatively aboutwhen and how to involve data
analytics and SADA in its work.
Both in terms of improving theefficiency and the effectiveness
of what we're doing.
The onus, I think, is onperformance auditors to
integrate data analystsinto their performance
audit team better.
Such that the data analyststruly understand the audit
(12:08):
objective, the audited entityand the business of that entity.
And a large challenge is findinga way for the performance audit
team and the data analyticsteam to work together well,
because there often, wefind, can be a bit of a chasm
between the two teams that'sdifficult to fill because the
performance audit team maynot have a good understanding
of the principles of dataextraction and preparation.
(12:31):
And conversely, the dataanalytics team can lack
knowledge of the auditobjectives the auditee
or the business area.
So I think that's one ofthe biggest hurdles that
we face is just bringingthose two teams together to
work effectively together.
Recently the ANAO moved fromassigned desks to activity-based
working arrangements to tryand facilitate that cross
(12:54):
collaboration between thedifferent teams physically
locating them near us byensuring that the data analytics
team is involved in keymeetings and conversations right
through the audit lifecycle,not just brought in mid audit.
So those are some of the thingswe're trying to do to increase
that collaboration between dataanalytics and performance audit.
Xiaoyan (13:12):
And our next goal is really to identify if there are
any opportunities to improvethe performance audit efficiency
by using data by starting torequest similar data early and
shortening the time on datadiscovery and acquisition.
Another opportunity is touse some of the high value,
common data sets which ispotentially relevant to
(13:34):
improve our efficiency inbuilding that understanding
and running analysis.
Narrator (13:39):
The Assurance Show is produced by Risk Insights.
We work with performanceauditors and internal auditors.
Delivering audits, helpingaudit teams use data
and coaching auditors toimprove their data skills.
You can find out more aboutour work at datainaudit.com.
Now, back to the conversation.
Yusuf (13:59):
Open data and improving the quality of
data that is available.
What role have you playedas an audit office in
helping to improve thatlevel of data quality?
Xiaoyan (14:10):
We did two information reports and following
our first release on ourreporting , We can definitely
see improvement in the dataquality in the Austender data.
I think the maturity ofgovernment agencies is improving
in producing and preparingdata over the last few years.
Most government entities havea data strategy in place and
(14:33):
as they are implementing datagovernance on how they prepare
and publish their data.
Through our informationreports we're promoting and
advocating for better quality.
Yusuf (14:45):
You mentioned that there's opportunities for
performance auditors to usedata more directly as opposed
to asking for everythingfrom the central team.
Do you have a view as to,where the gap has arisen?
So where it is that our useof data as auditors has not
progressed to the extent thatmaybe it should have and now
we almost have to play a littlebit of catch up in building
those skills within audit teams?
Xiaoyan (15:06):
This is a very good question.
The way I say this is youneed to vizualize a matrix.
On one axis you have thecapability of the audit
teams and whether auditorsalready have analytic skills.
And the other axis is thecomplexity of the analysis.
Some of the typical statisticalanalysis can be done using
(15:27):
Excel or visualization tools.
That can definitely behandled by the audit teams.
And there is no question abouttheir ability to do that.
But on the other side,these days because we use
more and more populationdata from the IT systems,
sometimes we are talkingabout millions of records.
If the data is not really usablethat's where my team comes in
(15:50):
to transform and prepare datafor the audit team to use.
And on the other sideSometime it's not about
their capability whetherthey can analyse data or not.
It's all about efficiency.
If we can do the samething much quicker.
And in our view, that's whereyou need to bring an analyst in
rather than using traditionalExcel to handle a large
(16:10):
volume of population data.
Christine (16:12):
One of the other ways in which I
think SADA and performanceaudit will work together
in future is identifyingopportunities for using data.
That might be an areawhere performance auditors
need more support.
They may not necessarilyunderstand, or be able
to visualize how data canbe extracted and used.
(16:34):
So that sort of collaborationright from the outset of
audit planning is reallyimportant, so that SADA can
help performance auditors,identify those opportunities.
Conor (16:45):
You mentioned there identifying those high value
data sets that maybe can beused for multiple projects.
And the audit teams wespeak to a lot of them are
going through that sameprocess or have done that.
What are some of the othertips and traps that other
audit agencies shouldconsider when using data?
Xiaoyan (17:00):
With the publicly available data sets, the
quality of them really varies.
For auditors, you reallyneed to make sure the
data has a high quality,is complete and accurate.
Do you have sufficientinformation for you to
verify the quality of databefore you can use it?
If we talk to our colleaguesacross, the audit sector,
(17:22):
there may be something wecan learn from each other.
Christine (17:24):
I have five tips for performance auditors.
The first one is to rememberthat a huge proportion of
the data analytics time forperformance audit is going
to be in data preparation andbudgets and timeframes need to
take that into consideration.
Secondly, I believe thatthe quality assurance and
validation of the data needsto be the first priority and
(17:48):
it needs to take place priorto any analysis that's done.
So establishing face validity ofthe data early on in the audit
process is really essential.
I'd also recommend maintaininga single source of truth.
So trying to integrate all yourdata into a single linked data
set will be the key to avoidingerror and other inefficiencies
(18:08):
in using that data.
Maintaining full recordsof all data preparation
and analysis procedures toenable quality assurance
work and internal audit.
And then finally just thepoint I was making before,
thinking about how you canfully integrate a data analyst
into the performance auditteam as early as possible
in the audit lifecycle.
Xiaoyan (18:28):
Last one is if analysts are not auditors, we need to
make sure whoever used theanalysis, has the capability
to interpret the data properly.
Conor (18:37):
Thank you, Christine and Xiaoyan for your time.
A few key takeaways for me.
When you're accessing datafor your projects, get your
request into the entity early.
The second thing was quality.
Do not skip any stepsaround quality because it
is so important and willpay dividends in the end.
Thirdly, when youget to your analysis.
Make sure you've documentedyour logic and that it's
(18:58):
reviewable by a peer or bya quality assurance person.
And lastly, and thiswas something that came
through quite strong wascollaboration as between the
data professionals and theperformance audit professionals
need to get that collaborationhappening as early as possible
at the start of every project.
Great conversationwith the ANAO.
Thanks again.
Xiaoyan (19:16):
Thanks to both of you.
Narrator (19:17):
If you enjoyed this podcast, please share
with a friend and rateus in your podcast app.
For immediate notificationof new episodes, you can
subscribe at assuranceshow.com.
The link is in the show notes.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.