Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:02):
The Institute of Internal Auditors presents all
things internal audit tech.
In this episode, George Barum talks with Mike Kino about
how agile methodologies
and artificial intelligence are
transforming internal audit functions.
Kino shares practical insights on implementing agile
auditing, the challenges and benefits of this approach,
and how AI is being used throughout the
(00:24):
internal audit lifecycle.
Mike, thanks for being with us today.
Thank you guys for having me. Very excited to be here.
So, uh, but I guess
before we get started, could you maybe give us
what you would, uh, consider a definition of agile auditing?
What I think of it, I think of, uh, efficiency
and streamlining processes, but you, you tell me.
How would you define it? So I would define agile as
(00:47):
allowing you to be dynamic in
how you're conducting your audit lifecycle.
Um, being able to conduct procedures in more of a, more
of an iterative way that allows you to be flexible
as certain risks may, may come up.
Whereas typically when you think of auditing,
you would have very structured, sort
of gated things you need to do.
You need a plan, then you need to go through your,
(01:08):
execute your execution phase, then you need to go
through your reporting phase and you need to go
through your monitoring follow-up.
When I think of Agile, I think of doing all of those sort of
in time box components that focus on one piece of scope
for a given audit or for a given body of work.
Okay, Mike, next question.
Um, how are you using the agile mindset
(01:28):
in your internal audit function?
So, from an internal audit standpoint, when we think of
planning and and Agile,
what we do is we prepare our annual audit plan
and we figure out all the work that we're gonna,
that we're gonna be doing in a given year.
That work is then time boxed
into what we would call an epic.
An epic is equivalent to an audit.
(01:49):
So in a given year, we would have the amount of audits
that would be called epics.
Each of those epics would be,
would have sprints allocated to them.
And within those sprints,
you are covering a functional scope area.
If we think about, um, doing a, um, an a procure
to pay audit, for example, in a procure to pay audit in one
of the sprints, we may cover the business process side
(02:13):
of procure to pay in another, um, sprint.
We may cover the TE side of procure to pay.
But in each one of those sprints, in each one
of those two week increments, we're going to plan,
we're gonna have our field work, we're gonna have review,
we're gonna have reporting, we're gonna have feedback,
we're gonna confirm observations,
and then we're going to continuously iterate.
And so we're going to have that continuous feedback loop
(02:36):
with the end user until it's done.
And then once we're done with one component of a sprint,
we're gonna move on to the next scoping area.
So typically where an audit would take three months to,
to complete, by the end of three months,
your report would be fully written, all
of the sprints would be completed in each sprint.
You would have all components
of the audit lifecycle completed,
and then at the end you would have stakeholders
(02:58):
and auditees who are in fully alignment with
what the output is and, and less friction and resistance.
Mike, could you give us an example of the planning
and scoping phase, um, some
of the critical elements that you need to consider?
I do think it's important as you embark on your agile
journey to define the definition of success.
(03:18):
And to me, we look at this in, in a few different areas.
Uh, number one, we look at sprint progress.
So in a two week period,
how much did we get done relative to plan?
We look at sprint velocity, how much can we get done
or what, what do we plan to get done in two week period?
We look at the quality of work.
You know, when you think of a two week time period,
(03:39):
it feels very condensed,
but we don't wanna sacrifice quality.
So that's another thing we assess.
We look at stakeholder satisfaction.
How satisfied are our stakeholders to go
through their all the audit life cycle
components in a two week period?
It may feel that we're pressed for time,
but what we're trying to do is get
through everything in an efficient way
(04:00):
and give them a given finding, if you will,
or give them, um, a certain observation
that they could help address more real time.
And then the last area of continuous success is continuous
improvement, making sure you're reflecting on
what worked well and what didn't,
and incorporating lessons learned
and better ways of working into the next phase
of, of your, your sprint.
(04:22):
Okay. Alright, thanks for that explanation.
How would you say agile auditing has changed
over the past few years?
I know we have AI
and we have different aspects from a technology standpoint,
but maybe if you said, uh,
looking at it maybe 2020 versus 2025,
what would you highlight are,
are some of the major differences?
I think from 2020 to 2025,
(04:43):
the major differences definitely relate to technology
and the advancement of technology.
So if you go back to when we started our journey, um,
which is in between that 2020
and 2025, yeah, it was around 2022.
What Agile looked for us in 2022 was having
a Kanban board using project management tools to figure out
what the year is gonna look like in terms of our audit plan,
(05:04):
and then time boxing each audit into sprints
and figuring out how we're going to sort
of allocate resources and that work.
And also having a Kanban board where we could sort
of know what's not started, what's in progress, um,
what is complete, what is in the backlog.
So Agile in 2022 was very project management oriented.
We had some data and some indicators in terms of
(05:25):
how work was going and how we could reallocate.
And then I think as the years progressed, the big thing
that changed was technology and the data we had.
So now there's a lot more, um, data involved in
how we manage our day-to-day and, and the use of agile.
When we are doing continuous monitoring
and we see there are certain transactions being flagged,
we may have to adjust our audit plan.
(05:46):
Working in an agile environment allows us
to shift priorities on demand
and allows us to really focus on the
highest rated risk areas.
And so the biggest difference is,
biggest difference is still having a project management
discipline, plus listening to the data
and then utilizing the skills
and program that you've developed to help address the,
the highest, uh, rated risk.
(06:06):
Okay. Uh, in, in terms of an organization
or an internal audit function that's considering, uh,
going from a traditional way of auditing, moving to Agile,
what would you say are maybe some readiness type things or,
or what you would do to get prepared for that?
Maybe some of the challenges that, that a, uh,
internal audit function may face in terms of readiness?
(06:27):
I think it starts with education.
Really talking to the team about
how you traditionally would audit, right?
And going through your, your various phases
to really thinking agile
and talking through what the benefits are, uh,
are in, are in that sense.
So when we started our journey, I explained to the team,
Hey, this is gonna feel a little bit unconventional,
but you're gonna reap a few different benefits.
(06:48):
One of the benefits is you're gonna own
to get all the phases done of an audit in one phase,
and the findings and everything that,
that are gonna come out of the audit
are not gonna be outdated.
So you're not gonna do an audit for six months
and then give all your results at the end.
In the first two months, you're gonna deliver results,
you're gonna have those agreed upon,
and your audit is gonna be sort
of completed in increments over time.
(07:08):
And that's more beneficial to the business
because they could be agile in their remediation, they hear
of a finding sort of real time,
and then they could go ahead and address it.
So really educating and getting the buy-in from my team
that this is a step in the right direction, right?
The regulatory landscape is very dynamic.
Business risk is very dynamic. We need to be able
to operate in line with the business
(07:29):
raises a software company.
The company operates under Agile or Scrum methodologies.
So we cannot operate in sort of an elongated audit lifecycle
if we wanted to really address risk
and mitigate things, um, on a day-to-day.
Yeah, it sounds like, uh, responsiveness is,
is a big advantage, just being, uh, able to adapt
(07:50):
and respond to the business, respond to management, right.
So from a roles
and responsibility standpoint, uh,
I know there are Scrum masters and different folks.
Uh, would you say that that is, uh, something
to identify the folks are gonna be involved in
certain roles in the beginning?
Is that just kind of evolve over time
or, uh, could you just talk a little bit about, you know,
(08:11):
how important it's to, to go ahead
and identify folks who are
gonna be in those, those key roles?
Yeah, it, it's definitely important
and it's also important to explain sort of
what those roles entail, right?
So your scrum master is almost your, you,
your lieutenant, right?
Your lead sort of project manager.
They're gonna keep everyone accountable, make sure
that the work is progressing,
make sure your CanBan board is up to date,
getting the relevant updates from the other team members.
(08:33):
We do daily standups.
The expectation when we get
to the daily standups is the scrum master would've spoken
to all the team members, make sure the board is reflected in
terms of what's not sorted, what needs going backlog.
So when we're having the meeting, it's very intentional
and everyone understands what the priorities are.
And if the priorities shift,
it's very clear when we're communicating
and we don't need to spend 45 minutes doing a rundown of
what everyone did, we can move on to the next day
(08:55):
and say, yep, the expectation
for the next day is that we get here.
And so we'll do, um, daily standups, we'll do, um,
every two weeks we'll do some sort of retrospective to see
what do we plan to accomplish versus what did we accomplish?
And then see where some of the, the blockers were, maybe
where there are certain inefficiencies and we look, um,
and lean on our agile framework to continuously improve.
(09:16):
So, Mike, from a, uh, success standpoint,
or maybe from a key performance indicator standpoint,
what are some of the key things that you would highlight on
how we're doing along that journey
and, uh, measuring our success with Agile auditing?
So when, when we're conducting an Agile audit, it's,
it's extremely important for us to work with the end user
and document the user acceptance criteria.
(09:38):
What does this mean in practical terms?
It means we're documenting the requirements that must be met
for a story or, or work item to be completed in the audit.
So understanding what the business risk is, making sure that
that's incorporated into the audit, in addition
to having your standard audit procedures that you're going
through, and Agile is meant to be collaborative,
(09:58):
and you're working with the end user to making sure
that everything is taken into account
as you iterate through the process.
Okay. Um, from a training standpoint, um, just to try to
get your internal audit function up to speed
and make sure that, um, you know, maybe on an annual basis
that, that you're improving
and working towards, uh, being more efficient, would you say
(10:19):
that, uh, that you need to go to training
or is it more on the job training, whereas you go
through different projects and different audits
that you kinda learn and, and evolve from there?
So I would say you need one person that's trained.
Um, I joined Brazen 2021 April of 2021
as the first internal audit hire
to really build out the function.
(10:39):
Later that year in 2021,
I attended an awesome training in Florida hosted
by the I a about agile auditing.
So I learned before I had my team learn,
and that's something that I hold
very near and dear to the heart.
So I'd say you need at least one person who understands
what agile auditing is, goes through, I'd say at least three
to five days of very hands-on tactical training
(11:00):
to see what's gonna work for your, your business needs,
and then be able to train other people.
I don't think it requires everyone on the team,
but you need one person who could then retrain
and get everyone into that, into that methodology
of updating a CanBan board, right?
Working on sprints, understanding what the expectations are
and when you need to finish things and,
and how you, um, iteratively work
(11:21):
through a larger body of work.
Okay. So let's, uh, shift gears a little bit.
Let's talk about artificial intelligence.
Could you maybe, uh, give a couple examples of things have,
uh, maybe over the past year
or so that you've seen, uh, be implemented with the Agile,
uh, approach?
Uh, maybe talk about, uh, you know, some examples,
(11:42):
maybe some challenges that folks have had implementing it.
Maybe just share some stories. Yeah.
So artificial intelligence has definitely expedited the way
that we could carry out
and conduct our agile way of auditing some real use cases
and, and practical ways of thinking about it.
When you think of planning, right,
and when you're under such a small time box component
(12:03):
of two weeks for planning,
the best thing you could have at your discretion is
technology to help you sort
of expedite the planning process.
So pre agile days
and pre AI days, you would spend a lot of time on scoping,
having my planning memos,
getting all the relevant information in there.
Now we leverage AI to help us scope,
help us write the planning memo.
Things that took multiple hours or multiple days,
(12:25):
but now happening in seconds.
And now we just need to do a quality level of review on it
to make sure it's tailored towards what we're doing.
That's planning, right? Then we think of, we move
to your field work phase.
Again, you're pressed on time, you have sort of two weeks
to cover a large amount of scope.
Well, what's the simplest thing to use AI for?
Record a meeting, transcribe notes,
(12:45):
and have sort of a prerequisite for
what your work paper's gonna look like.
And again, in the past, you would have a human
who would've to take very detailed notes.
You then have to get into a room,
did we capture all the relevant information?
Does everyone agree? Well, now we have a pretty accurate,
and I say pretty accurate, right?
Because with, with that ca caveat with ai,
we have a pretty accurate representation
of the meeting output.
(13:05):
We could then summarize from there what your attributes
that you're testing or what are,
what are the key takeaways that you need for your work paper?
And then you've, you've again saved a lot of time.
So it's definitely helped expedite in in that sense.
When you think of data,
data's important in any audit that you're doing.
We use AI to, um, analyze tabular data.
(13:25):
So very structured data.
Um, if we're doing data ana analytics as part of an audit,
we may write the SQL queries and scripts,
but we may use AI to interpret the results.
If we're looking at documentation, um, PDFs
or unstructured data, we may use AI
to help us expedite the review
and give us some of the outputs
or if we're looking for a
specific consideration in a report.
(13:47):
So we try to look at it
and correlate AI to every sort of phase
of the audit lifecycle and where we can get to it.
So then moving on from execution to reporting.
Reporting's great, right?
You, you sort of have AI help you stage your work papers,
and then for a report you may take a first pass, say, Hey,
can you rewrite this section?
(14:07):
So almost looking at it as a sort of assistant
that could help you expedite each part of the process,
but you never wanna sacrifice quality.
So with the use of ai, we make sure we have
diligent review checkpoints to make sure that
what we're putting down is sound and makes sense
and it's aligned with what the stakeholders
have have communicated to us.
And then lastly, monitoring and follow up.
(14:27):
So we use AI to maybe draft a remediation plan.
We may get a written up response in terms of where a,
where an auditee is in their, um, remediation journey.
And we may revise it to make it a little bit more cleaner,
more concise and so forth.
So we've used it at every part of the audit lifecycle
and it's definitely helped allow us to save time
(14:48):
and operate in sort of the,
the two week sort of, uh, time box.
Okay, great, great examples.
Would you say there are any misconceptions with, uh,
with agile auditing, someone who's maybe not done it before
or maybe just has, uh, limited, uh, exposure to it,
what would you say is, is maybe
something that, uh, that's out there?
That's, uh, a misconception?
A misconception about agile auditing is that
(15:11):
it's one size fits all and it's absolutely not.
And you need to understand the principles of agile
and align it with what your intended output is, right?
Whether you do daily standups
or whether you do standups a few times a week,
you still are going with the spirit of communicating,
constantly understanding when things are changing internally
(15:33):
with the business, external factors, um,
whether you're doing your retrospectives
every week versus two weeks.
So the timing, I think there's a lot of flexibility
and variability in what you're doing.
The documentation, there are very, um, structured documents
that the Agile manifesto states, right?
You have artifacts and the, and a few different documents.
(15:53):
My team does not align with those documents one to one.
We may call them totally different things,
but we follow some of the principles to make sure
that we are operating in an agile manner
and we're not a hundred percent compliant with it.
And I, I think anyone
who tells you they're a hundred percent agile is kind
of fooling themselves.
Okay. So last question, Mike.
(16:14):
Uh, maybe this is putting you on the spot a little bit,
but what do you anticipate the next few years hold
for agile auditing?
And maybe if you can, uh, you know, work in any,
any angles from an AI standpoint,
how do you see this evolving?
What do you think are some of the, gonna be the, the key,
um, the items that you, that you look out for in the future?
So I think you're gonna see a lot more adoption of AI
(16:35):
and have that integrated with agile.
I think there may be ai, scrum, masters, right?
Who keep your team accountable, where you had a human
who was sort of monitoring the progress of work
and making sure everything was on track.
You probably have more aid from an AI
scrum master to do that.
I think you're gonna see a lot more in
the data analytics space.
(16:55):
Um, you're gonna see a lot more agentic workflows
to help sort of streamline the way we audit.
Um, so I think those are just a few examples of
how you're gonna see AI be more inserted.
Um, within, within Agile.
Mike, you mentioned, uh, the possibility of ai, uh,
replacing the Scrum master, uh, role.
Could you take a little, uh,
a little bit deeper dive into that?
(17:16):
Maybe give some examples and uh, how that could happen?
So, you know, I have a vision that there could be the use
of ai, you know, to replace a scrum master.
What this, um, AI agent would do would be to skin
or sift through your Kanban board, right?
Understand where there may be, um, too much in one
(17:38):
of the various buckets, whether it be in progress, you know,
not started and automatically figure out a way to
reallocate story cards to end users, to free up allocation
and to get more work done in an efficient manner.
'cause I think right now when you think of Agile,
it's really at the discretion
of the Scrum master communicating with each member
(17:59):
of the team on the amount
of work they could take if things are moving along.
And when you think of AI
and having AI sort of expedite this process,
it could analyze a lot more of this information a lot faster
and figure out if you are time boxing something,
a story card, and you're saying it's gonna take four hours
and it's been sitting on your Kanban board in progress
for multiple weeks, then you as a human have looked at it
(18:20):
and sort of mis sized it,
but perhaps the AI can say, Hey,
over a few days, this still wasn't done.
You've time boxed this to be four hours.
Is this getting done in this sprint
or should it get allocated to the next sprint?
So I think in terms of like ai, scrum, masters,
there's an opportunity for them
to rationalize your Kanban board, give summary outputs of
what we believe we can achieve in a two week sprint when
(18:40):
we're going through planning, and then work with us
as humans, help tell the story
and help drive greater efficiencies
as we progress through the year.
Okay. Good deal. Well, it was great talking to you today.
Thanks for sharing your insights, Mike. Thank you.
Thank you guys for having me.
Really appreciate the conversation.
Hey, audit pros ready to supercharge your skills
(19:00):
and connect with the best in the field.
You absolutely need
to check out the I'S 2025 International Conference happening
July 14th through the 16th in Toronto.
And virtually this is your chance
to dive into emerging risks, cutting edge tech
and global best practices
that will elevate your internal audit game.
Don't get left behind and register now@theia.org.
(19:24):
If you like this podcast, please subscribe and rate us.
You can subscribe wherever you get your podcasts.
You can also catch other episodes on YouTube or@theia.org.
That's THE iia.org R.