March 24, 2023 • 48 mins
This episode we are excited to welcome to the podcast Danielle Duran, Director of GxP Compliance and Training at Aimmune Therapeutics! As fellow advocates for education and continuous enablement in life sciences, Dori and Danielle share their insights on the current state of learning in our industry, and where we can do better to support more effective outcomes in IT and Quality organizations.
Listen in now to learn more a bout leveraging effective learning to support quality culture, shifting towards effectiveness measures of training, and, perhaps most importantly, moving towards a people-centric learning model as part core business operations.
Enjoy the episode and please don't forget to rate, subscribe and share!
*Disclaimer: Podcast guest participated in the podcast as an individual subject matter expert and contributor. The views and opinions they share are not necessarily shared by their employer. Nor should any reference to specific products or services be interpreted as commercial endorsements by their current employer.
This is a production of ProcellaRX
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Dori Gonzalez-Acevedo (00:00):
Well, welcome to another episode of
(00:01):
software quality today. Hi,Danielle, how are you today? I'm
great. Thanks, Tori, for havingme. So I would I am so excited
about today, because we've metover the course of this last
year at a couple differentevents. But this last event in
December, we got to know eachother even more. And I'm so
excited about our newcollaboration and friendship and
(00:23):
all of that stuff. Love it. I'mso excited.
So before we get started, Iwould love for you to tell
everyone about your journey towhere you're at today as a
director of GXP learning.
Danielle Duran (00:37):
Sure, so I, I took a route through learning.
That's how that's how I gotwhere where I am right now. It
started my very first role outof college was teaching math and
science and medieval modernworld history to seventh graders
in South LA. And I was I wasfeeling pretty passionate, by
(01:00):
the end of my first couple ofyears about teaching math and
turning,you know, preteens into math
nerds, that was a mission that Ihad. But given how things worked
in large public school systemsat the time, I was bumped from
my position and was being pushedinto a role to teach more
history, which I was lesspassionate about. And that
(01:21):
caused me to reevaluate what Iwanted to be doing, I ended up
doing some recruitment for theteaching program that I was in
to make sure that my studentswere going to have really great
math teachers. But working inrecruitment taught me that I
wanted longer term relationshipswhere I could coach and support
(01:41):
the growth and success ofothers. And so I ended up in a
role where I was coaching andsupporting new teachers. And
that led me to a place ofmanaging sets of those coaches.
And then I moved into, then Imoved into other roles related
to teaching education for awhile, including some medication
(02:04):
technology, and then I ended upin philanthropy. So on the I've
done every single side ofeducation you can do. And in
that philanthropy role, Istarted doing a lot more
evaluation of effectiveness ofprograms and of education
programs. And, you know, whatare models of doing that and
working. Another role that I hadwas working with the state on
(02:29):
creating an assessment for theCalifornia Community College
system. And so I got to workwith psychometricians. And
that's when I first startedlearning about validation,
testing validation, and thatthat was very interested in
that. So we use little bits ofthings along the way. And I did
study science in college, Ispent a lot of time in labs. And
I think there's always a littlebit of something missing in my
(02:51):
heart for that science, heartmiss in me that, you know, kind
of loss for words for the rightright thing there. But a role
opened up at a biotech companycloser to where I lived. And I
was, I just jumped at theopportunity, it was in corporate
training. And in the compliancearea. So it was, it was great,
(03:16):
because I got to learneverything about GXP. From the
from the highest level, I got tounderstand everything about
compliance and what that means Igot to work with a lot of
auditors. And that gave me areally good view into what I was
doing. That department wasultimately eliminated. And I
moved into technical operationswhere I was overseeing a one of
(03:37):
our manufacturing sites and notsite one of the one of the
manufacturing groups within thelarger site, there were several.
So I worked in gene therapy fora little while supporting that
group, and then ultimately ledour site training team and was
(03:59):
working on global initiatives,and really wanted to just
continue having a larger globalimpact. So that's, and then I
found the role that I'm in rightnow where I get to oversee the
governance of of training Thetraining system, writ large for
the enterprise, and I amresponsible for all of the
(04:19):
regulated training.
Dori Gonzalez-Acevedo (04:23):
So part of your story is very common of
the some of the women that I'vealso interviewed this past year,
right, is that we've had thesekind of very divergent I don't
even mazes of careers, right.
Danielle Duran (04:40):
Oh, yeah. It was not a straight line.
Dori Gonzalez-Acevedo (04:43):
And your mom and all this other stuff are
all those.
Danielle Duran (04:46):
Yeah. And I had a baby during that time. Yep.
Dori Gonzalez-Acevedo (04:50):
So what about that having that diversity
of experience gives you theopportunity to do what You do
today.
Danielle Duran (05:03):
So I, in my experience so far I, when it
comes to my colleagues in, inthe regulated side of training,
I haven't really come acrossthat many that have a full
background in education. And,and really understanding at a
deep level how learning reallyworks within the neurobiology of
(05:25):
that is, and what, just the vasttools that are available for
effective learning. And a lot oftimes, I think folks in my type
of role will come from it orwill come from equality. And
then the folks in l&d that areon the commercial side, often
they're, they're reallysuccessful sales trainers, and
(05:47):
it's a, you know, a step on theway to management. But I think
the advantage that I have isreally understanding what
effective learning is, and howand how it connects to culture,
and how important the culturepart is when it comes to
learning. And then neuro, theneuro stuff matters, because
(06:09):
it's how, how available, yourmind even is in order to learn.
And that's not something that wethink about or talk about a
whole lot.
Dori Gonzalez-Aceved (06:18):
Absolutely 100% agree. And one of the
things in what you're saying,strikes me as a real issue over
the years that I've seen is thatthe effectiveness piece, right,
whether we're talking abouteffectiveness of, of Kappa,
implementation or effectivenessof learning, there has been a
huge gap in within the industry.
And one of the things often hasbeen said, oh, we'll just do if
(06:40):
we didn't learn and understand,right. And I know that that bugs
you out, like really, reallybad, and it bugs me out to you.
But so can you say more aboutthat why this effectiveness
piece is really, reallycritical?
Danielle Duran (06:54):
I think that Well, I also think that this is
something that the healthauthorities are paying attention
to even more. So it's even moreimportant that we have the
conversation. I think in thepast, it was acceptable that a
training activity beingcomplete, meant that it was
(07:15):
effective, but especially withinwhat you're referring to where
training, the word training isequated to an attestation of
understanding, without anymeasure of oh, what's our
evidence, and I find that sointeresting for this field where
everything is evidence based,right? You're not going to take
(07:37):
a product from, you know, acrossthe phases without really strong
evidence that it's got reallyhigh efficacy. So you know,
what, why, how come we're notpracticing the same thing when
it comes to our learning, andit's expensive, conducting
training activities, it takes alot of time, even if it's just
(07:58):
reading SOPs, which is nottraining, just reading. But
that's, you know, it takes time,and people who only have so much
bandwidth, so much mental RAM,you can say, in order to process
new information, and then ifthey're not going to put it to
you. So if you don't have anactual training activity in
(08:18):
there putting it to use applyingit, they're really not going to
learn it effectively. I mean,yeah,
Dori Gonzalez-Acevedo (08:24):
yeah, it's a I think it's a big issue
that we seen in qualitymanagement programs, or are not
such great quality managementprograms. Right. I think we've
also, historically have been ledto think that if we do the
checklist, and that we providethe checklist to the auditors or
the regulators, then then thatif as long as that checklist is
(08:46):
done, and all those signaturesare there as that adaptation is
there, then then we're good. Andwe can walk away, and we've done
our job. Right. And and that'snot necessarily the case, right?
I think there's plenty ofexamples out there where, you
know, we've put as a cap item,right, that it's due to
(09:09):
training, right? Human error,human error is retraining and
retrain. And, and how many timesdo we do that right over and
over again, and simplify it inthat sort of way, but yet, get
to an end result where somethingis reproducible, and effective,
and learned from and added valueto write all of those things.
(09:34):
And then the other part of thatis walking the walk, like saying
saying, Oh, go do this training,but actually invest in the
dollars associated to that. So Iknow you have some thoughts
around how how, you know, whereis learning and in the scale of
organizational hierarchy?
Danielle Duran (09:54):
Yeah. Well, I think I think the most the most
critical thing that any One cando and a lot of a lot of what I
recommend is balsley. Like,quote free in that it's, it's
really just how are youleveraging the resources you
already have primarily peopleand their time. And I remember I
(10:16):
found a quote some time ago thatI wish I knew who said it, maybe
someone one of the listenerswill know. But it's something
like, If you don't have time todo it right. Now, when will you
have time to fix it? And I know,they're stuck somewhere that
I've read that, you know, talkabout the cost of quality and
the cost of preventing an error,as opposed to fixing an error.
(10:39):
There's something I talked to mycentral daughter about
literally, like two days ago, Iwas like, okay, but is it? Is it
easier if we prevent this giantmess from happening? Or is it
you know? And what is what?
What's the investment? How am Igoing to, you know, what, how
much
Dori Gonzalez-Acevedo (10:53):
time I think so you're talking to your
seven year old around that Ithink most folks know that
organically at some level thatthat is the right thing to do.
Danielle Duran (11:02):
But we don't do it. So this so this? So I'll
start even with this idea ofhuman error and kappa. So
there's a human error? To whatextent are we holding ourselves
accountable to spending the timeto get to a really good,
specific root cause? And if thatroot cause really is why I
didn't understand what I wassupposed to do? That's different
(11:25):
than I didn't follow theirdirections. Right? Like, those
are two very differentconversations. I could see
retrain, sure, if you justdidn't do it, then like, Okay,
we need to have a little heavierhand. But if I didn't know what
to do, that's very different.
But if you're if your kappa isretrained the exact same way you
did the first time, then what'syour confidence that it's going
(11:45):
to work the second time, so itdidn't work the first time. And
so it's, I think a lot of timesin our field, things are so
fast, and no one wants to slowdown, because we're all under so
much pressure. Right.
Dori Gonzalez-Acevedo (12:00):
I would argue that a lot of many
verticals. It's not just likesciences. Right. I mean, I think
that's also kind of a the theawakening out of COVID. Right?
Right. I think now we canglobally and universally say, we
don't have enough brain mentalload to go around?
Danielle Duran (12:23):
Oh, yeah. Now for all the things that were
demanding of everybody. So andthat's, that's where? So using
that as the little kernelexample. But thinking bigger of
if, if someone if a manager saysmy people it, what does it look
like to be qualified? What doesit mean? And then take that time
(12:45):
to really define? What does itmean to be qualified that I feel
really confident in my peoplebeing successful later? And then
if they think that someone needssome additional training, then
then how is the manager going toknow that that training work?
How do they know that it was agood investment in time? What
should they be looking forlater, to see? Oh, that that
training did have an impact? AndI think that's where I really
(13:11):
wish one of the things I wish Isaw more of was managers being
super engaged in what's in mypeople's curriculum? What what
do they need to do, whether it'sSOPs, or on the job trainings,
or, you know, some other type ofelearning? Is that really the
thing that's gonna get me mypeople to where they need to go?
Dori Gonzalez-Acevedo (13:31):
Well, so let's, let's pause there for a
moment. Because do we think thatthe managers at those levels
actually have the skills thatthey need in order even to make
those assessments?
Danielle Duran (13:40):
Right? Well, that's a very good question. And
I'm not sure they're, I don't Idon't know. I think a lot of
times, not necessarily, but Idon't know if they're ever
taught what that looks like.
Right? We oftentimes, especiallythe mid level managers, they're
not, that's not somethingthey've ever even seen. So how
are they supposed to know how todo this? That's right. And, and
then there's the, you know, verycommon leadership training,
(14:03):
right that people might get, butthat's not teaching you this
very technical skill of how am Idefining qualification for my
people? How will I know what agap is in their performance and
how to match that with theright, I think I've just seen
some really, like heartbreakingexamples of, quote, performance
management when someone isn'tperforming, but the manager
doesn't know how to coach theminto higher performance. Because
(14:25):
it's it's not it's notstraightforward. And it's not
it's not simple. And sometimesit takes a lot of that
Dori Gonzalez-Acevedo (14:33):
so complicated. I mean, yeah, I
know, even with my teenager,right? Like, there's so many
complexities to learning styles,my own myself, just like sick,
and it is not something that Ireally understood until much
(14:53):
much later on in my life. Nowthat I know that I know some
things that I can do in order tohelp mitigate that. Right? But
there's there's a whole spectrumout here for for how one can
consume information in general.
Danielle Duran (15:10):
Right. And the the most important thing to
remember is the the, there's athere's like, blinking on the
term I just I just saw LinkedInposts about it. There's, there's
in like types of intelligenceslike the seven. I think it's
Howard Gardner like types ofintelligences. And then there's,
(15:31):
I think what what we grew upbeing being told was learning
styles, right? visual, auditory,or tactile, whatever. But what
the research has shown is thatlearning styles isn't a thing.
It's not, it's, the preferencesare real. But what matters for
effectiveness, and again, comingback to effectiveness is that
you're tying the right style tothe content. So it's kind of
(15:55):
like, are you trying to teachhow to identify birdsong? Or, I
guess, in our area? You know,visual inspection of something,
you know, something in QC. Butare you teaching it with
visuals, like you have to seevisually, you can't just
describe it? Are you you know,do you need to see the actual
thing with examples as opposedto pictures? Right. So just
(16:17):
being really clear with again,the most important thing is know
what you're trying toaccomplish, so that you can plan
to that, and it takes a lot oftime, it's hard work. That's
very, like heavy cognitive loadof, do I really know what I want
to get out of this program. Andthat's my argument against
buying off the shelf trainingprograms. Because it's very easy
(16:42):
to want to outsource that mentalload and say, Oh, no, someone
already made this thing. Someonealready made these elearning. So
I just have to give them to mypeople and make them watch these
videos, as opposed to well, whatdo a people specifically need in
order to be successful? Thesethe learnings could be a tool,
but what questions am I going toask about it? Or what's the
(17:05):
conversation? And how are wegoing to apply this new
information so that they can putit to use otherwise? You know,
just watch TED Talks? Or I don'tknow, you know, what's, what are
we? What are we doing? Yeah,
Dori Gonzalez-Acevedo (17:17):
and there's so much content out
there, too, is like so then whatdo you even choose? Right?
Because at that point, you'reyou it's just a commodity and
not something specific to yourorganization to the actual use
case that you need to the painpoints that you have to the
people that you have thecombination of full time
employees versus contractors,right.
Danielle Duran (17:38):
Oh, yeah, that's tricky with employment law in
some states. Yeah.
Dori Gonzalez-Acevedo (17:43):
It's a big, big deal. Right. So so and
we're in an industry that doesdo a lot of contracting. Right.
And so then, whoseresponsibility it is to train?
Right? Am I paying thosecontractors my time in order to
train them? Or should they becoming in trained as a some base
level, you know, understanding?
And so whose responsibility is,
Danielle Duran (18:04):
right? And how often are we evaluating? Like,
do we really know? Because theysay, you know, they're, they're
saying that they know thesethings, but do I? Do I check? Or
how do I check or what's, what'seffective to check?
Dori Gonzalez-Acevedo (18:18):
It's really hard. I mean, I struggle
even internally here. And as wewere building out our team here
at persona, like it was like toreally understand everyone's
needs, right? We're small enoughthat we can do that, right. But
yet, how to continuously monitorand engage with them in order
to, to have them grow. Right. Sowhat are the gaps? I am not a
(18:43):
learning specialist. Right.
Right. You know, and so, so, sowhen I approach it from a
scientific perspective, and I,you know, I try, it's trial and
error, right? It's, you know,put something in place, does it
work? Does it not work as aneffectiveness check? Right?
Yeah.
Danielle Duran (19:00):
But that's great that you're checking, did it
work? That's the most importantthing. That's the question I got
asked during an inspection withthe MHRA, you know, how do you
know that worked? How did youknow that aseptic technique
training worked? And, you know,having someone pass a quiz that
requires them to pass it at100%? That's, is that really
evidence that it did work? Andthis is, I was on a panel or I I
(19:25):
hosted a panel in June where Ihad a health authority, member
share, what are you reallylooking for when it comes to
effectiveness of training? Andhis his response, included a lot
of things like well, there'scertain things that people think
that we want, but that's all thefloor, like yes, you should have
strong training metrics ofcompleting training on time
(19:47):
quote, training, right SOPtraining. But that's that's just
a standard expectation that thatshould just happen. What really
matters is your your QMS whathow what are your deviations
looking like, what your cap islook like how effective are your
cap is. And the thing that hereally spent most time on was
(20:09):
collaboration, which I was notexpecting. And he talked a lot
about documentation ofprocedures? And is everybody in
the room that needs to be in theroom when designing that
procedure? Are the people whoare executing there to make sure
that it's going to work? Andthen do you test it before you
(20:31):
make it effective? Do you go andmake sure it works? And then is
there improvement over time? AndI that's not something that we
talk about ever when it comes totraining effects?
Dori Gonzalez-Acevedo (20:40):
Very interesting. So I don't know, 15
years ago, when when, when wedid peak user performance
qualifications, right. For CSP,when we were still kind of doing
that terminology for CSP, partof the methodology that I had
rolled out at that time wasactually executing an SOP.
(21:01):
Right, as evidence that thesystem was performing as
intended. Because if the SOPthat's written for that system,
is not able to execute and getthe result that you want, then
something's amiss. Right. Right.
I've seen that go away. Like,that doesn't happen anymore. And
I wonder about that, because Ithink that back to your earlier
(21:24):
point around how much time andeffort are we spending on these
activities? You know, we do alot of stuff around validation
and getting things to a state ofquote, unquote, you know,
validated. But the ongoingmaintenance, and the operation
is where all this stuff reallylives. Right. Right. Right. How
is it working? What what's notworking, even though their
(21:49):
systems and we programmaticallydesign them for some things.
Still, there's a huge amount ofinput human input that's
required, regardless of wherewe're talking about machine
learning or automation to writebecause
Danielle Duran (22:02):
there's no humans involved, which means
there's still going to be, youknow, room for error.
Dori Gonzalez-Acevedo (22:07):
Yeah, yeah. So what would your ideal?
I'm curious, what would yourideal kind of, you know, in a 40
hour week, what would you wantemployees to be spending their
time on? I'm curious to hearwhat you think,
Danielle Duran (22:24):
how people should spend time I think I if I
had a wish list, right? If ifmoney wasn't an object and
people can, I would love that.
From the what is it? There's thecommon ad 2010 model or
770 2010? Model have the most ofyour learning comes from doing
(22:47):
your job? And I had a manager incheese, when was it 2007? Maybe,
who? Two dozen eight, maybe whotold me the best professional
development you can do is yourjob well, and I thought, I think
I didn't want to hear that.
Because I think I wanted to doyou know, conferences and
(23:08):
courses and special projects,but she, I I'll never forget
that of her telling me you needto do your job? Well, and I
think about that a lot. Becausethere's a lot of parts of my job
that I'm I'm still, you know,things change so much. I have my
scope changes, things are new, Ihave to do a new type of system.
And if I could learn, I can keeplearning, even if I'm doing only
(23:33):
my job. Right. So I think, but Ithink the piece that's missing
is the reflection. So if if, youknow, people typically I think
they say, the effective, likeuse of people, it's you have 70%
of your time that you'reactually getting anything done.
The rest is who knows what. Soout of that 70%. If you're only
(23:56):
if you're doing your job job for70% of that time, then you still
need to build in I'd say atleast a half an hour, once or
twice a week for reflection,like what what am I doing that's
working? Who did I talk to thatsaid something really
insightful? worded? I readsomething that I want to you
know, apply. And then the 2010.
That's the, you know, coachingmentoring is 20. And then 10 is
(24:21):
the actual learning. I don'tknow, I honestly don't know if I
know anybody who actually spends10% of their time on learning
outside outside of work. I don'tI don't think I know anybody,
not even me. Yeah. And
Dori Gonzalez-Acevedo (24:42):
it's something we don't prioritize,
right. And part of what I alwayssay, though, is also but a
different approach to how you dowork. So one of the things I
just heard in what you weredescribing, Danielle, was that
how you approach work which isactually very similar to me and
maybe part of why we feel so?
Akin? Yeah? Is? It is learning?
(25:06):
Right again, doing it all in,right, like, fully consumed and
in the flow, if you will, right,like really asking all those
questions, you know, inside andout deconstructing
reconstructing is all part ofthe process of which I do my day
to day job.
Danielle Duran (25:26):
Right, right, and asking those questions and
figuring out, Well, who else doI need to talk to? Or who else
might know something about this?
Or who's doing this better? Or?
I don't know what went wrongthere. But I needed you know,
it's it's a lot of askingquestions to understand, right.
And I think that without havingreally a strong growth mindset,
(25:48):
that can be really hard, and itcan feel very threatening and
scary.
Dori Gonzalez-Acevedo (25:53):
And you also need an organization that
supports that. Yeah. Right.
Because if you don't have anorganization that supports
curiosity, and a growth mindset,what are you left with, you're
left with, you have to make sureyou get these 10 widgets done
before the end of the week,because that's your job. And
that's your task. And we reallydon't care about much anything
else other than the task. Right?
Danielle Duran (26:15):
Right. Right.
And well, there's the there'sthe aspect of the making the
widgets, where if something goeswrong with the widget, if
there's not a strong learningculture, then that deviation on
the widget isn't going to beescalated as quickly or as
thoroughly. Right. So havingthat learning mindset, having
that learning culture iscritical for a strong QMS.
(26:36):
Because you, you have to makepeople comfortable with
mistakes, and learning fromthose mistakes. And then the
last thing you want to do ishave five different people make
the same widget mistake, thenhow much waste is that right?
That's wasting time we've seenproduct, we've seen raw
materials, you know, putting indanger getting product to
patient, or whatever product towhoever uses a widget. And, and
(26:59):
that's where that bigger pictureLessons Learned comes in. And so
even if we don't have time tospend 20% of your time with a
coach or mentor, I mean, that'sreally what some of your manager
interactions should be like, areyou getting enough coaching from
your manager? are you presentingus information so that you can
be coached? And then are wereally preparing our managers to
(27:19):
be effective coaches? I don't,probably not enough, honestly,
not enough. But from I think,plenty of companies that are
successful have made statementsabout, you know, it's your
competitive advantages, howquickly you can learn, right,
Dori Gonzalez-Acevedo (27:41):
and adapt and pivot and all of that,
right. And in order to do that,a lot of the things that we're
talking about, right, thelearning mindset, the curiosity
is, you know, the constantquestioning, I also think that
questioning in some cultures isnot accepted, right. And so, so
to add to the complexity of thisentire conversation, right, is
(28:05):
to put in a diverse workforce,or hoping that most folks are
striving for a diverseworkforce. But that does require
even more question, moreunderstanding on everyone's
part.
Danielle Duran (28:17):
And sometimes you can't wait for, for people
to ask a question you need to beinviting, right? And invite the
how do you create a culture ofinviting that kind of feedback,
whether it's on behavior andperformance or on on a product
or on a process, right, like,what, and making sure that we
have lots of those diverseexperiences, so that we get the
(28:41):
best? And the end is possible?
Dori Gonzalez-Acevedo (28:44):
Yeah. So in your kind of exploration, I
know you've, you've done sometalking, as well as at some
major conferences. So you have alot of you hear a lot of things,
and what are what are some ofthe things that are pitfalls, or
the gotchas that folks couldquickly just nip in the bud and
(29:08):
stop doing in their learningprogram or, or do better?
Danielle Duran (29:13):
think some things I'll share one pet peeve,
which I think drives peoplecrazy, even if they don't
realize it, and that's gatheringdata that you don't use, or, or
having having data that that youcould use, but you don't do
anything. Like what like anexample could be a lot of people
(29:35):
might have quizzes, right orsome type of knowledge
assessment connected to readingan SOP or doing some reading a
PowerPoint, whatever it wouldbe, but how often are you
leveraging that data to improvean SOP like, jeez, this question
has a 30% pass rate on the firsttime? Maybe we should clarify it
(29:56):
if if you're using your SOPs astraining any materials, which
they're not usually written tobe a training material, then
then then you have to payattention to the performance of
it as a training material. thatwould that would be one thing.
Dori Gonzalez-Acevedo (30:14):
Well, that actually goes right back to
your whole the psychometricsright of a lot of stuff, right?
I mean, in a, in a, in a welldefined training program or
learning management program,right? All of those metrics are
part of the process of doing allof that, right. Which is
fascinating to me, because it'sthere's a lot of validating
(30:34):
talked about validation, there'sa lot of validation that goes
into a lot of that stuff.
Danielle Duran (30:38):
There is massive amounts, like massive, massive
projects with like, millions andmillions of people, right as
data points. That's years andyears of data over time. So I,
but I think the point that youjust made, is my maybe like,
that's the other thing that canbe a quick fix is, did you
clearly define your objectives?
Is your training program set upin a way that it's to accomplish
(31:01):
tasks? Or is it set up in a wayto build knowledge or skill? And
how clearly have you definedthose requirements for
qualification? So that you, youcan even track that? Because if
you haven't defined it, then youcan't gather data to show that
it's working or not working?
Dori Gonzalez-Acevedo (31:24):
Yeah, I think, again, given that we have
younger kids, right, what I'vebeen trying to instill is this,
you know, constant, learner,right, and learning how to learn
for themselves as they gothrough this process, and things
that I would never have thoughtI would say to them, like, I
(31:46):
really don't care about grades,I really don't care about this,
I don't really care about this,like things that I held very
true for me, right, I've thrownout the window, right? in
service of curiosity andlearning being the most
important skill set that theycan get. Now,
Danielle Duran (32:05):
yes, and the not being afraid to make mistakes I
had, I had no idea how much Iwould be battling this
perfection pressure that mydaughter puts on herself. And,
you know, just the, the, that'sthe whole idea of perfect is the
enemy of good or somethingperfect is the enemy of
(32:27):
complete, I don't know, one ofthose tech companies have
everything. But that's, it'shard. I think we all get kind of
caught up in that also. And it'ssome of that analysis paralysis
of well, I want to do this thingfor my training program, but
it's just really huge, bigthing. And so I just can't get
started. Because it's sooverwhelming, where, you know,
(32:49):
just pick one thing, pick oneroll, or pick one, pick one set
of SOPs on one critical, youknow, set of procedures that you
want to improve, pick one pieceof the QMS, pick one department,
you know, there's you got tojust start somewhere, and it'll
feel so good when you're done.
It will build some of thatmomentum to get through some of
(33:15):
the hardware things. Futurethinking?
Dori Gonzalez-Acevedo (33:24):
What do you think the impact of AI and
machine learning have onlearning programs?
Danielle Duran (33:35):
That's such a big question. And I have not
been like
Dori Gonzalez-Acevedo (33:39):
to ask you. But like, I think it's an
important, right, because it is.
Danielle Duran (33:45):
I think so far, what I know has been a trend. I
don't know about a trend. But Iknow people have gotten a lot
more involved with AR and VR. Sothe augmented reality, virtual
reality when it comes totraining, that's huge. And I'd
say I had pretty strong opinionsabout it when I was working to
(34:09):
support manufacturing, becausemy concern was that we were
treating people like robots,instead of leveraging the assets
that they are people instead ofrobots. So it's it's fine to do
mented reality where you know,you have some extra directions
and it helps them to do thetask. But it's very easy to get
(34:31):
cognitively lazy, honestly, isthe best word. Right? Like we
just want people to do this onething and want them to do it
perfectly every single time butthat's just not you know, that's
it might be more than it mightbe a higher expectation than is
reasonable. And, and if you wantto, you know, I just I had a
(34:57):
hard time really believing thatthe The ROI on investing in AR
VR was going to come back like,is it really worth the
investment? And what are youreally trying to accomplish? And
what are you really trying toautomate out because I'm sure
some of it is really good. But Ithink just like any tool that
comes into the training andlearning realm, especially on
(35:18):
the GXP side, people get veryexcited about tools, but not
necessarily building a programaround the tool, and really
having clearly definedobjectives and clearly defined
KPIs for effectiveness. So if,if you're implementing, and you
have all that in place, then I'msuper on board. But if you're
implementing it, because youthink it'll make it easier, it
(35:39):
won't. That's just not how itworks. But when it comes to AI,
I know a lot of like schoolsystems teachers are having to
really reevaluate what they'reexpecting. Yeah, just because
how people are leveraging it.
Dori Gonzalez-Acevedo (35:55):
Yeah, I think you're spot on around the
expectation piece, right? Ithink that there's. And I can't
even fathom, like, there's justso much to consume today. Right?
That's the other part, right?
Like that I worry about in justhow much is possible. And then
to your expectation point iswhat's necessary, right? And how
(36:19):
to leverage the technology inservice of doing better. And I
don't know, if we've, as anindustry, at least in the life
sciences, have, we're there yet?
Because we haven't really evenbeen able to do like, the basics
yet. Right? But I know it'swanted, and it's coming. Right?
(36:42):
Whether we like it or not, it'sit right. And so I I'm, I'm
pondering that sort of thing.
And not just in the learningspace, but also in the
manufacturing space. And becauseit's going to hit the
manufacturing floor faster thanever before. And in what are we
are we that far behind? And howdo we get up to speed?
Danielle Duran (37:08):
Yeah, I think, like anything else, like any of
these tools, the man even likeproject management tools, or,
you know, tracking, checklists,tools, if, if it's not
implemented mindfully with allthese other, you know,
objectives, and how we, youknow, it works and how are you
going to train people to use itand what's Okay, and what's not,
(37:31):
what are the guardrails? You'reyou're just adding more content,
you're adding more stuff forpeople to do. And there's only
so much that people can do, justperiod, you know, we maxed out.
So I remember reading somethingnot too long ago about advice
from someone saying this is,it's a great tool in order to
(37:53):
free you up to do the morecomplicated stuff like yeah,
let's let I'm 100%. Let'sautomate all the little things.
Thanks. Yeah.
Dori Gonzalez-Acevedo (38:02):
Well, and there's any automation tool,
right, I mean, that's, that's,that should be the expectation
and the first low hanging fruit,right to get all of those tasks
as automated as possible, sothat we can free up our brains
to take on some higher levelthinking. It's just a matter of,
what do you do? What's next? Andhow does that evolve when you
(38:24):
your basic programs are notreally hitting the marks? In
some some regards? Right? So wetalked a little bit about like,
you know, quality as an ROI, butlike, learning as an ROI is
probably even harder toquantify. I suspect, I haven't
tried have you to do metricsaround that. Tell me.
Danielle Duran (38:46):
So the the, I think one of the trickiest
parts, when, when working withyour cross functional partners
to define what those KPIs aregoing to look like, is not
getting caught up in Oh, it'stoo complex. We can't do it.
Because there's so many factors.
So there's the most common, I'dsay, way to measure ROI, ROI and
(39:09):
learning is the Kirkpatrickmodel, where it's four levels
level one is, did the peoplelike it? And that's where you
get smell sheets, or you get thesurvey at the end, you know, are
you satisfied? And then leveltwo is knowledge transfer. And
that's where the quizzes comein. So do you have evidence to
(39:31):
show that knowledge wastransferred, did did the
knowledge transfer part work?
And then level three isbehaviors. And that's this is
where it starts getting tricky,but and you have to define it on
the front end. So you define I'mgoing to do X training. The
behavior I want to change or thebehavior I want to build is x
(39:55):
and so how will I identify itlater? And that's hard is is
who's going to identify it? Howwill it be reported? How will
you record it? Right? And isthere different observers, and
you have to validate thedifferent observers. And that's
where in education, you'd userubrics, and then you train
people to rubrics and you norm.
But that's just not somethingI've ever really seen done here.
And I think, because it's notcommonly, then people want to
(40:20):
say, Well, you can't do it well,yet, you can. That's how you
grade AP essays, you know,that's how you evaluate
teachers, is you have a rubric,and you have people norm, and
you have standards at eachlevel, and then you all rate
until you all get the samescores. So behaviors is level
three, and then level four isbusiness results. So if you
like, let's say, you're workingon your deviation training,
(40:43):
program, deviations take acertain amount of time at the
you know, that's your baseline,you take a certain measure,
there's so many errors, and it'sthis many interactions or
whatever it's going to be, andthen you do your training
programs you test for did peoplelike it, you do a knowledge
check? Do they retain certainamounts of new things that they
have to do differently than thebehaviors? Do they change their
behaviors? are they submittingmore on time? Are they filling
(41:05):
these things out more correctly?
Are they you know, what are thebehaviors and then business
result well, before, before thetraining, deviation took, on
average, you know, 28 days, orhad this many errors, or to this
much time period, right. Andthen if you calculate that into
a cost, each deviation costs$10,000. But then after the
(41:27):
training, with the follow up,net, we're saving $3,000 per
deviation because of thistraining. So there's the
business result. But it takes alot of time. And it takes a lot
of data. And I just wish I hadmore lovely green belt black
belt people to work with. Tohelp with all that. And that's,
that's where I really wish therewas more collaboration between
(41:49):
the operational excellence orcontinuous improvement teams and
learning, because their work isso related. I think it's funny
some time, I remember, a boss Ihad a while ago, we were doing
worked in business operationsgroup. And we were doing an
exercise a marketing exercisefor our department. And one of
(42:10):
the questions was, who's ourgreatest competitor, like, we
offer a service learningcontinuous improvement, we offer
service, who's our greatestcompetitor, and it was our
customers, our greatestcompetitors, because they want
it, they want to do itthemselves, or they've always
done it themselves. They're notused to having support. And so
we're not involved in theseconversations. And so that's the
(42:30):
other advice I give to peopleis, you have experts out there
probably somewhere, you know, goto your learning person, and,
and partner. And, and you don'thave to, I don't know why I
don't know why people don't thatwould be a root cause I'd like
to explore is why when peopledon't reach out to their
learning departments, why not,
Dori Gonzalez-Acevedo (42:52):
but we see with quality in it too, or,
you know, quality in business,you know, like, it's reaching
across the aisle, it's havingshared conversations, it's
having the nuancedconversations, it's, you know,
it's really, for having the timeto do it, having the time to do
it, knowing who to do it with,right, who you know. And it
(43:14):
also, you know, really breakingdown the notion that your, your,
your lane is your lane, andwe're all connected, like we're
all connected, right, and reallyunderstanding that and one of
the things that I love aboutsome of the the good programs
that I've seen is really like anunderstanding from a quality
(43:36):
perspective, how we're allinvolved, right? Like, it's,
it's, it's so incrediblyimportant to have that that
connection, be illustrated bedemonstrated within a whole
organization. And part of thatis a whole Learning Foundation,
from beginning to end this justlike the quality like so,
(43:56):
quality and learning should notbe separate from its, you know,
it's all part of the same thing.
Yeah,
Danielle Duran (44:02):
we're all it's all it's all an organism. Right,
that's the system thinking. Soyou Peter and I'm not I don't
know how much again, this is tome, I would love to have a
system thinking program that youknow, that people understand
here's, you have to consider allthese other areas. And if you
(44:23):
touch it up here, thendownstream what happens just
like data
Dori Gonzalez-Acevedo (44:28):
moves, right so and it moves and shifts
so as you put into that system,you know, it's entropy extra
Pete right, like as you move inIT contracts and moves here and
then it moves and shifts again,and it has to be that continuous
look all the time, right ofwhere the needs and what is in
service of what and what are thein service of the need at the
(44:49):
moment because today isdifferent from six months. And
you know, the past six months.
Danielle Duran (44:56):
I mean, like weeks
Dori Gonzalez-Acevedo (44:59):
Yeah, So, all of those things. So, um,
this is awesome. And I think youand I also don't think this is
going to be our last podcastbecause we have lots to chat
about. I think this is a goodprime prime start here. And I
thank you for your time today. Ireally appreciate
Danielle Duran (45:23):
thank you for your thank you for the
conversation and I'd really loveto know what people want to talk
about more you know, what whatare people curious? Oh, what did
they wish they knew more about?
Dori Gonzalez-Acevedo (45:32):
That's great. So let will ask that
question when we post this out.
Have folks comments on and wemaybe we'll even do maybe a live
q&a next time and we'll get Oh,I love that. Awesome. Yeah.
Cool. All right, Danielle.
Thanks so much.
Danielle Duran (45:48):
Thanks, Tori.
Well, welcome to another episode of
(00:01):
software quality today. Hi,Danielle, how are you today? I'm
great. Thanks, Tori, for havingme. So I would I am so excited
about today, because we've metover the course of this last
year at a couple differentevents. But this last event in
December, we got to know eachother even more. And I'm so
excited about our newcollaboration and friendship and
(00:23):
all of that stuff. Love it. I'mso excited.
So before we get started, Iwould love for you to tell
everyone about your journey towhere you're at today as a
director of GXP learning.
Danielle Duran (00:37):
Sure, so I, I took a route through learning.
That's how that's how I gotwhere where I am right now. It
started my very first role outof college was teaching math and
science and medieval modernworld history to seventh graders
in South LA. And I was I wasfeeling pretty passionate, by
(01:00):
the end of my first couple ofyears about teaching math and
turning,you know, preteens into math
nerds, that was a mission that Ihad. But given how things worked
in large public school systemsat the time, I was bumped from
my position and was being pushedinto a role to teach more
history, which I was lesspassionate about. And that
(01:21):
caused me to reevaluate what Iwanted to be doing, I ended up
doing some recruitment for theteaching program that I was in
to make sure that my studentswere going to have really great
math teachers. But working inrecruitment taught me that I
wanted longer term relationshipswhere I could coach and support
(01:41):
the growth and success ofothers. And so I ended up in a
role where I was coaching andsupporting new teachers. And
that led me to a place ofmanaging sets of those coaches.
And then I moved into, then Imoved into other roles related
to teaching education for awhile, including some medication
(02:04):
technology, and then I ended upin philanthropy. So on the I've
done every single side ofeducation you can do. And in
that philanthropy role, Istarted doing a lot more
evaluation of effectiveness ofprograms and of education
programs. And, you know, whatare models of doing that and
working. Another role that I hadwas working with the state on
(02:29):
creating an assessment for theCalifornia Community College
system. And so I got to workwith psychometricians. And
that's when I first startedlearning about validation,
testing validation, and thatthat was very interested in
that. So we use little bits ofthings along the way. And I did
study science in college, Ispent a lot of time in labs. And
I think there's always a littlebit of something missing in my
(02:51):
heart for that science, heartmiss in me that, you know, kind
of loss for words for the rightright thing there. But a role
opened up at a biotech companycloser to where I lived. And I
was, I just jumped at theopportunity, it was in corporate
training. And in the compliancearea. So it was, it was great,
(03:16):
because I got to learneverything about GXP. From the
from the highest level, I got tounderstand everything about
compliance and what that means Igot to work with a lot of
auditors. And that gave me areally good view into what I was
doing. That department wasultimately eliminated. And I
moved into technical operationswhere I was overseeing a one of
(03:37):
our manufacturing sites and notsite one of the one of the
manufacturing groups within thelarger site, there were several.
So I worked in gene therapy fora little while supporting that
group, and then ultimately ledour site training team and was
(03:59):
working on global initiatives,and really wanted to just
continue having a larger globalimpact. So that's, and then I
found the role that I'm in rightnow where I get to oversee the
governance of of training Thetraining system, writ large for
the enterprise, and I amresponsible for all of the
(04:19):
regulated training.
Dori Gonzalez-Acevedo (04:23):
So part of your story is very common of
the some of the women that I'vealso interviewed this past year,
right, is that we've had thesekind of very divergent I don't
even mazes of careers, right.
Danielle Duran (04:40):
Oh, yeah. It was not a straight line.
Dori Gonzalez-Acevedo (04:43):
And your mom and all this other stuff are
all those.
Danielle Duran (04:46):
Yeah. And I had a baby during that time. Yep.
Dori Gonzalez-Acevedo (04:50):
So what about that having that diversity
of experience gives you theopportunity to do what You do
today.
Danielle Duran (05:03):
So I, in my experience so far I, when it
comes to my colleagues in, inthe regulated side of training,
I haven't really come acrossthat many that have a full
background in education. And,and really understanding at a
deep level how learning reallyworks within the neurobiology of
(05:25):
that is, and what, just the vasttools that are available for
effective learning. And a lot oftimes, I think folks in my type
of role will come from it orwill come from equality. And
then the folks in l&d that areon the commercial side, often
they're, they're reallysuccessful sales trainers, and
(05:47):
it's a, you know, a step on theway to management. But I think
the advantage that I have isreally understanding what
effective learning is, and howand how it connects to culture,
and how important the culturepart is when it comes to
learning. And then neuro, theneuro stuff matters, because
(06:09):
it's how, how available, yourmind even is in order to learn.
And that's not something that wethink about or talk about a
whole lot.
Dori Gonzalez-Aceved (06:18):
Absolutely 100% agree. And one of the
things in what you're saying,strikes me as a real issue over
the years that I've seen is thatthe effectiveness piece, right,
whether we're talking abouteffectiveness of, of Kappa,
implementation or effectivenessof learning, there has been a
huge gap in within the industry.
And one of the things often hasbeen said, oh, we'll just do if
(06:40):
we didn't learn and understand,right. And I know that that bugs
you out, like really, reallybad, and it bugs me out to you.
But so can you say more aboutthat why this effectiveness
piece is really, reallycritical?
Danielle Duran (06:54):
I think that Well, I also think that this is
something that the healthauthorities are paying attention
to even more. So it's even moreimportant that we have the
conversation. I think in thepast, it was acceptable that a
training activity beingcomplete, meant that it was
(07:15):
effective, but especially withinwhat you're referring to where
training, the word training isequated to an attestation of
understanding, without anymeasure of oh, what's our
evidence, and I find that sointeresting for this field where
everything is evidence based,right? You're not going to take
(07:37):
a product from, you know, acrossthe phases without really strong
evidence that it's got reallyhigh efficacy. So you know,
what, why, how come we're notpracticing the same thing when
it comes to our learning, andit's expensive, conducting
training activities, it takes alot of time, even if it's just
(07:58):
reading SOPs, which is nottraining, just reading. But
that's, you know, it takes time,and people who only have so much
bandwidth, so much mental RAM,you can say, in order to process
new information, and then ifthey're not going to put it to
you. So if you don't have anactual training activity in
(08:18):
there putting it to use applyingit, they're really not going to
learn it effectively. I mean,yeah,
Dori Gonzalez-Acevedo (08:24):
yeah, it's a I think it's a big issue
that we seen in qualitymanagement programs, or are not
such great quality managementprograms. Right. I think we've
also, historically have been ledto think that if we do the
checklist, and that we providethe checklist to the auditors or
the regulators, then then thatif as long as that checklist is
(08:46):
done, and all those signaturesare there as that adaptation is
there, then then we're good. Andwe can walk away, and we've done
our job. Right. And and that'snot necessarily the case, right?
I think there's plenty ofexamples out there where, you
know, we've put as a cap item,right, that it's due to
(09:09):
training, right? Human error,human error is retraining and
retrain. And, and how many timesdo we do that right over and
over again, and simplify it inthat sort of way, but yet, get
to an end result where somethingis reproducible, and effective,
and learned from and added valueto write all of those things.
(09:34):
And then the other part of thatis walking the walk, like saying
saying, Oh, go do this training,but actually invest in the
dollars associated to that. So Iknow you have some thoughts
around how how, you know, whereis learning and in the scale of
organizational hierarchy?
Danielle Duran (09:54):
Yeah. Well, I think I think the most the most
critical thing that any One cando and a lot of a lot of what I
recommend is balsley. Like,quote free in that it's, it's
really just how are youleveraging the resources you
already have primarily peopleand their time. And I remember I
(10:16):
found a quote some time ago thatI wish I knew who said it, maybe
someone one of the listenerswill know. But it's something
like, If you don't have time todo it right. Now, when will you
have time to fix it? And I know,they're stuck somewhere that
I've read that, you know, talkabout the cost of quality and
the cost of preventing an error,as opposed to fixing an error.
(10:39):
There's something I talked to mycentral daughter about
literally, like two days ago, Iwas like, okay, but is it? Is it
easier if we prevent this giantmess from happening? Or is it
you know? And what is what?
What's the investment? How am Igoing to, you know, what, how
much
Dori Gonzalez-Acevedo (10:53):
time I think so you're talking to your
seven year old around that Ithink most folks know that
organically at some level thatthat is the right thing to do.
Danielle Duran (11:02):
But we don't do it. So this so this? So I'll
start even with this idea ofhuman error and kappa. So
there's a human error? To whatextent are we holding ourselves
accountable to spending the timeto get to a really good,
specific root cause? And if thatroot cause really is why I
didn't understand what I wassupposed to do? That's different
(11:25):
than I didn't follow theirdirections. Right? Like, those
are two very differentconversations. I could see
retrain, sure, if you justdidn't do it, then like, Okay,
we need to have a little heavierhand. But if I didn't know what
to do, that's very different.
But if you're if your kappa isretrained the exact same way you
did the first time, then what'syour confidence that it's going
(11:45):
to work the second time, so itdidn't work the first time. And
so it's, I think a lot of timesin our field, things are so
fast, and no one wants to slowdown, because we're all under so
much pressure. Right.
Dori Gonzalez-Acevedo (12:00):
I would argue that a lot of many
verticals. It's not just likesciences. Right. I mean, I think
that's also kind of a the theawakening out of COVID. Right?
Right. I think now we canglobally and universally say, we
don't have enough brain mentalload to go around?
Danielle Duran (12:23):
Oh, yeah. Now for all the things that were
demanding of everybody. So andthat's, that's where? So using
that as the little kernelexample. But thinking bigger of
if, if someone if a manager saysmy people it, what does it look
like to be qualified? What doesit mean? And then take that time
(12:45):
to really define? What does itmean to be qualified that I feel
really confident in my peoplebeing successful later? And then
if they think that someone needssome additional training, then
then how is the manager going toknow that that training work?
How do they know that it was agood investment in time? What
should they be looking forlater, to see? Oh, that that
training did have an impact? AndI think that's where I really
(13:11):
wish one of the things I wish Isaw more of was managers being
super engaged in what's in mypeople's curriculum? What what
do they need to do, whether it'sSOPs, or on the job trainings,
or, you know, some other type ofelearning? Is that really the
thing that's gonna get me mypeople to where they need to go?
Dori Gonzalez-Acevedo (13:31):
Well, so let's, let's pause there for a
moment. Because do we think thatthe managers at those levels
actually have the skills thatthey need in order even to make
those assessments?
Danielle Duran (13:40):
Right? Well, that's a very good question. And
I'm not sure they're, I don't Idon't know. I think a lot of
times, not necessarily, but Idon't know if they're ever
taught what that looks like.
Right? We oftentimes, especiallythe mid level managers, they're
not, that's not somethingthey've ever even seen. So how
are they supposed to know how todo this? That's right. And, and
then there's the, you know, verycommon leadership training,
(14:03):
right that people might get, butthat's not teaching you this
very technical skill of how am Idefining qualification for my
people? How will I know what agap is in their performance and
how to match that with theright, I think I've just seen
some really, like heartbreakingexamples of, quote, performance
management when someone isn'tperforming, but the manager
doesn't know how to coach theminto higher performance. Because
(14:25):
it's it's not it's notstraightforward. And it's not
it's not simple. And sometimesit takes a lot of that
Dori Gonzalez-Acevedo (14:33):
so complicated. I mean, yeah, I
know, even with my teenager,right? Like, there's so many
complexities to learning styles,my own myself, just like sick,
and it is not something that Ireally understood until much
(14:53):
much later on in my life. Nowthat I know that I know some
things that I can do in order tohelp mitigate that. Right? But
there's there's a whole spectrumout here for for how one can
consume information in general.
Danielle Duran (15:10):
Right. And the the most important thing to
remember is the the, there's athere's like, blinking on the
term I just I just saw LinkedInposts about it. There's, there's
in like types of intelligenceslike the seven. I think it's
Howard Gardner like types ofintelligences. And then there's,
(15:31):
I think what what we grew upbeing being told was learning
styles, right? visual, auditory,or tactile, whatever. But what
the research has shown is thatlearning styles isn't a thing.
It's not, it's, the preferencesare real. But what matters for
effectiveness, and again, comingback to effectiveness is that
you're tying the right style tothe content. So it's kind of
(15:55):
like, are you trying to teachhow to identify birdsong? Or, I
guess, in our area? You know,visual inspection of something,
you know, something in QC. Butare you teaching it with
visuals, like you have to seevisually, you can't just
describe it? Are you you know,do you need to see the actual
thing with examples as opposedto pictures? Right. So just
(16:17):
being really clear with again,the most important thing is know
what you're trying toaccomplish, so that you can plan
to that, and it takes a lot oftime, it's hard work. That's
very, like heavy cognitive loadof, do I really know what I want
to get out of this program. Andthat's my argument against
buying off the shelf trainingprograms. Because it's very easy
(16:42):
to want to outsource that mentalload and say, Oh, no, someone
already made this thing. Someonealready made these elearning. So
I just have to give them to mypeople and make them watch these
videos, as opposed to well, whatdo a people specifically need in
order to be successful? Thesethe learnings could be a tool,
but what questions am I going toask about it? Or what's the
(17:05):
conversation? And how are wegoing to apply this new
information so that they can putit to use otherwise? You know,
just watch TED Talks? Or I don'tknow, you know, what's, what are
we? What are we doing? Yeah,
Dori Gonzalez-Acevedo (17:17):
and there's so much content out
there, too, is like so then whatdo you even choose? Right?
Because at that point, you'reyou it's just a commodity and
not something specific to yourorganization to the actual use
case that you need to the painpoints that you have to the
people that you have thecombination of full time
employees versus contractors,right.
Danielle Duran (17:38):
Oh, yeah, that's tricky with employment law in
some states. Yeah.
Dori Gonzalez-Acevedo (17:43):
It's a big, big deal. Right. So so and
we're in an industry that doesdo a lot of contracting. Right.
And so then, whoseresponsibility it is to train?
Right? Am I paying thosecontractors my time in order to
train them? Or should they becoming in trained as a some base
level, you know, understanding?
And so whose responsibility is,
Danielle Duran (18:04):
right? And how often are we evaluating? Like,
do we really know? Because theysay, you know, they're, they're
saying that they know thesethings, but do I? Do I check? Or
how do I check or what's, what'seffective to check?
Dori Gonzalez-Acevedo (18:18):
It's really hard. I mean, I struggle
even internally here. And as wewere building out our team here
at persona, like it was like toreally understand everyone's
needs, right? We're small enoughthat we can do that, right. But
yet, how to continuously monitorand engage with them in order
to, to have them grow. Right. Sowhat are the gaps? I am not a
(18:43):
learning specialist. Right.
Right. You know, and so, so, sowhen I approach it from a
scientific perspective, and I,you know, I try, it's trial and
error, right? It's, you know,put something in place, does it
work? Does it not work as aneffectiveness check? Right?
Yeah.
Danielle Duran (19:00):
But that's great that you're checking, did it
work? That's the most importantthing. That's the question I got
asked during an inspection withthe MHRA, you know, how do you
know that worked? How did youknow that aseptic technique
training worked? And, you know,having someone pass a quiz that
requires them to pass it at100%? That's, is that really
evidence that it did work? Andthis is, I was on a panel or I I
(19:25):
hosted a panel in June where Ihad a health authority, member
share, what are you reallylooking for when it comes to
effectiveness of training? Andhis his response, included a lot
of things like well, there'scertain things that people think
that we want, but that's all thefloor, like yes, you should have
strong training metrics ofcompleting training on time
(19:47):
quote, training, right SOPtraining. But that's that's just
a standard expectation that thatshould just happen. What really
matters is your your QMS whathow what are your deviations
looking like, what your cap islook like how effective are your
cap is. And the thing that hereally spent most time on was
(20:09):
collaboration, which I was notexpecting. And he talked a lot
about documentation ofprocedures? And is everybody in
the room that needs to be in theroom when designing that
procedure? Are the people whoare executing there to make sure
that it's going to work? Andthen do you test it before you
(20:31):
make it effective? Do you go andmake sure it works? And then is
there improvement over time? AndI that's not something that we
talk about ever when it comes totraining effects?
Dori Gonzalez-Acevedo (20:40):
Very interesting. So I don't know, 15
years ago, when when, when wedid peak user performance
qualifications, right. For CSP,when we were still kind of doing
that terminology for CSP, partof the methodology that I had
rolled out at that time wasactually executing an SOP.
(21:01):
Right, as evidence that thesystem was performing as
intended. Because if the SOPthat's written for that system,
is not able to execute and getthe result that you want, then
something's amiss. Right. Right.
I've seen that go away. Like,that doesn't happen anymore. And
I wonder about that, because Ithink that back to your earlier
(21:24):
point around how much time andeffort are we spending on these
activities? You know, we do alot of stuff around validation
and getting things to a state ofquote, unquote, you know,
validated. But the ongoingmaintenance, and the operation
is where all this stuff reallylives. Right. Right. Right. How
is it working? What what's notworking, even though their
(21:49):
systems and we programmaticallydesign them for some things.
Still, there's a huge amount ofinput human input that's
required, regardless of wherewe're talking about machine
learning or automation to writebecause
Danielle Duran (22:02):
there's no humans involved, which means
there's still going to be, youknow, room for error.
Dori Gonzalez-Acevedo (22:07):
Yeah, yeah. So what would your ideal?
I'm curious, what would yourideal kind of, you know, in a 40
hour week, what would you wantemployees to be spending their
time on? I'm curious to hearwhat you think,
Danielle Duran (22:24):
how people should spend time I think I if I
had a wish list, right? If ifmoney wasn't an object and
people can, I would love that.
From the what is it? There's thecommon ad 2010 model or
770 2010? Model have the most ofyour learning comes from doing
(22:47):
your job? And I had a manager incheese, when was it 2007? Maybe,
who? Two dozen eight, maybe whotold me the best professional
development you can do is yourjob well, and I thought, I think
I didn't want to hear that.
Because I think I wanted to doyou know, conferences and
(23:08):
courses and special projects,but she, I I'll never forget
that of her telling me you needto do your job? Well, and I
think about that a lot. Becausethere's a lot of parts of my job
that I'm I'm still, you know,things change so much. I have my
scope changes, things are new, Ihave to do a new type of system.
And if I could learn, I can keeplearning, even if I'm doing only
(23:33):
my job. Right. So I think, but Ithink the piece that's missing
is the reflection. So if if, youknow, people typically I think
they say, the effective, likeuse of people, it's you have 70%
of your time that you'reactually getting anything done.
The rest is who knows what. Soout of that 70%. If you're only
(23:56):
if you're doing your job job for70% of that time, then you still
need to build in I'd say atleast a half an hour, once or
twice a week for reflection,like what what am I doing that's
working? Who did I talk to thatsaid something really
insightful? worded? I readsomething that I want to you
know, apply. And then the 2010.
That's the, you know, coachingmentoring is 20. And then 10 is
(24:21):
the actual learning. I don'tknow, I honestly don't know if I
know anybody who actually spends10% of their time on learning
outside outside of work. I don'tI don't think I know anybody,
not even me. Yeah. And
Dori Gonzalez-Acevedo (24:42):
it's something we don't prioritize,
right. And part of what I alwayssay, though, is also but a
different approach to how you dowork. So one of the things I
just heard in what you weredescribing, Danielle, was that
how you approach work which isactually very similar to me and
maybe part of why we feel so?
Akin? Yeah? Is? It is learning?
(25:06):
Right again, doing it all in,right, like, fully consumed and
in the flow, if you will, right,like really asking all those
questions, you know, inside andout deconstructing
reconstructing is all part ofthe process of which I do my day
to day job.
Danielle Duran (25:26):
Right, right, and asking those questions and
figuring out, Well, who else doI need to talk to? Or who else
might know something about this?
Or who's doing this better? Or?
I don't know what went wrongthere. But I needed you know,
it's it's a lot of askingquestions to understand, right.
And I think that without havingreally a strong growth mindset,
(25:48):
that can be really hard, and itcan feel very threatening and
scary.
Dori Gonzalez-Acevedo (25:53):
And you also need an organization that
supports that. Yeah. Right.
Because if you don't have anorganization that supports
curiosity, and a growth mindset,what are you left with, you're
left with, you have to make sureyou get these 10 widgets done
before the end of the week,because that's your job. And
that's your task. And we reallydon't care about much anything
else other than the task. Right?
Danielle Duran (26:15):
Right. Right.
And well, there's the there'sthe aspect of the making the
widgets, where if something goeswrong with the widget, if
there's not a strong learningculture, then that deviation on
the widget isn't going to beescalated as quickly or as
thoroughly. Right. So havingthat learning mindset, having
that learning culture iscritical for a strong QMS.
(26:36):
Because you, you have to makepeople comfortable with
mistakes, and learning fromthose mistakes. And then the
last thing you want to do ishave five different people make
the same widget mistake, thenhow much waste is that right?
That's wasting time we've seenproduct, we've seen raw
materials, you know, putting indanger getting product to
patient, or whatever product towhoever uses a widget. And, and
(26:59):
that's where that bigger pictureLessons Learned comes in. And so
even if we don't have time tospend 20% of your time with a
coach or mentor, I mean, that'sreally what some of your manager
interactions should be like, areyou getting enough coaching from
your manager? are you presentingus information so that you can
be coached? And then are wereally preparing our managers to
(27:19):
be effective coaches? I don't,probably not enough, honestly,
not enough. But from I think,plenty of companies that are
successful have made statementsabout, you know, it's your
competitive advantages, howquickly you can learn, right,
Dori Gonzalez-Acevedo (27:41):
and adapt and pivot and all of that,
right. And in order to do that,a lot of the things that we're
talking about, right, thelearning mindset, the curiosity
is, you know, the constantquestioning, I also think that
questioning in some cultures isnot accepted, right. And so, so
to add to the complexity of thisentire conversation, right, is
(28:05):
to put in a diverse workforce,or hoping that most folks are
striving for a diverseworkforce. But that does require
even more question, moreunderstanding on everyone's
part.
Danielle Duran (28:17):
And sometimes you can't wait for, for people
to ask a question you need to beinviting, right? And invite the
how do you create a culture ofinviting that kind of feedback,
whether it's on behavior andperformance or on on a product
or on a process, right, like,what, and making sure that we
have lots of those diverseexperiences, so that we get the
(28:41):
best? And the end is possible?
Dori Gonzalez-Acevedo (28:44):
Yeah. So in your kind of exploration, I
know you've, you've done sometalking, as well as at some
major conferences. So you have alot of you hear a lot of things,
and what are what are some ofthe things that are pitfalls, or
the gotchas that folks couldquickly just nip in the bud and
(29:08):
stop doing in their learningprogram or, or do better?
Danielle Duran (29:13):
think some things I'll share one pet peeve,
which I think drives peoplecrazy, even if they don't
realize it, and that's gatheringdata that you don't use, or, or
having having data that that youcould use, but you don't do
anything. Like what like anexample could be a lot of people
(29:35):
might have quizzes, right orsome type of knowledge
assessment connected to readingan SOP or doing some reading a
PowerPoint, whatever it wouldbe, but how often are you
leveraging that data to improvean SOP like, jeez, this question
has a 30% pass rate on the firsttime? Maybe we should clarify it
(29:56):
if if you're using your SOPs astraining any materials, which
they're not usually written tobe a training material, then
then then you have to payattention to the performance of
it as a training material. thatwould that would be one thing.
Dori Gonzalez-Acevedo (30:14):
Well, that actually goes right back to
your whole the psychometricsright of a lot of stuff, right?
I mean, in a, in a, in a welldefined training program or
learning management program,right? All of those metrics are
part of the process of doing allof that, right. Which is
fascinating to me, because it'sthere's a lot of validating
(30:34):
talked about validation, there'sa lot of validation that goes
into a lot of that stuff.
Danielle Duran (30:38):
There is massive amounts, like massive, massive
projects with like, millions andmillions of people, right as
data points. That's years andyears of data over time. So I,
but I think the point that youjust made, is my maybe like,
that's the other thing that canbe a quick fix is, did you
clearly define your objectives?
Is your training program set upin a way that it's to accomplish
(31:01):
tasks? Or is it set up in a wayto build knowledge or skill? And
how clearly have you definedthose requirements for
qualification? So that you, youcan even track that? Because if
you haven't defined it, then youcan't gather data to show that
it's working or not working?
Dori Gonzalez-Acevedo (31:24):
Yeah, I think, again, given that we have
younger kids, right, what I'vebeen trying to instill is this,
you know, constant, learner,right, and learning how to learn
for themselves as they gothrough this process, and things
that I would never have thoughtI would say to them, like, I
(31:46):
really don't care about grades,I really don't care about this,
I don't really care about this,like things that I held very
true for me, right, I've thrownout the window, right? in
service of curiosity andlearning being the most
important skill set that theycan get. Now,
Danielle Duran (32:05):
yes, and the not being afraid to make mistakes I
had, I had no idea how much Iwould be battling this
perfection pressure that mydaughter puts on herself. And,
you know, just the, the, that'sthe whole idea of perfect is the
enemy of good or somethingperfect is the enemy of
(32:27):
complete, I don't know, one ofthose tech companies have
everything. But that's, it'shard. I think we all get kind of
caught up in that also. And it'ssome of that analysis paralysis
of well, I want to do this thingfor my training program, but
it's just really huge, bigthing. And so I just can't get
started. Because it's sooverwhelming, where, you know,
(32:49):
just pick one thing, pick oneroll, or pick one, pick one set
of SOPs on one critical, youknow, set of procedures that you
want to improve, pick one pieceof the QMS, pick one department,
you know, there's you got tojust start somewhere, and it'll
feel so good when you're done.
It will build some of thatmomentum to get through some of
(33:15):
the hardware things. Futurethinking?
Dori Gonzalez-Acevedo (33:24):
What do you think the impact of AI and
machine learning have onlearning programs?
Danielle Duran (33:35):
That's such a big question. And I have not
been like
Dori Gonzalez-Acevedo (33:39):
to ask you. But like, I think it's an
important, right, because it is.
Danielle Duran (33:45):
I think so far, what I know has been a trend. I
don't know about a trend. But Iknow people have gotten a lot
more involved with AR and VR. Sothe augmented reality, virtual
reality when it comes totraining, that's huge. And I'd
say I had pretty strong opinionsabout it when I was working to
(34:09):
support manufacturing, becausemy concern was that we were
treating people like robots,instead of leveraging the assets
that they are people instead ofrobots. So it's it's fine to do
mented reality where you know,you have some extra directions
and it helps them to do thetask. But it's very easy to get
(34:31):
cognitively lazy, honestly, isthe best word. Right? Like we
just want people to do this onething and want them to do it
perfectly every single time butthat's just not you know, that's
it might be more than it mightbe a higher expectation than is
reasonable. And, and if you wantto, you know, I just I had a
(34:57):
hard time really believing thatthe The ROI on investing in AR
VR was going to come back like,is it really worth the
investment? And what are youreally trying to accomplish? And
what are you really trying toautomate out because I'm sure
some of it is really good. But Ithink just like any tool that
comes into the training andlearning realm, especially on
(35:18):
the GXP side, people get veryexcited about tools, but not
necessarily building a programaround the tool, and really
having clearly definedobjectives and clearly defined
KPIs for effectiveness. So if,if you're implementing, and you
have all that in place, then I'msuper on board. But if you're
implementing it, because youthink it'll make it easier, it
(35:39):
won't. That's just not how itworks. But when it comes to AI,
I know a lot of like schoolsystems teachers are having to
really reevaluate what they'reexpecting. Yeah, just because
how people are leveraging it.
Dori Gonzalez-Acevedo (35:55):
Yeah, I think you're spot on around the
expectation piece, right? Ithink that there's. And I can't
even fathom, like, there's justso much to consume today. Right?
That's the other part, right?
Like that I worry about in justhow much is possible. And then
to your expectation point iswhat's necessary, right? And how
(36:19):
to leverage the technology inservice of doing better. And I
don't know, if we've, as anindustry, at least in the life
sciences, have, we're there yet?
Because we haven't really evenbeen able to do like, the basics
yet. Right? But I know it'swanted, and it's coming. Right?
(36:42):
Whether we like it or not, it'sit right. And so I I'm, I'm
pondering that sort of thing.
And not just in the learningspace, but also in the
manufacturing space. And becauseit's going to hit the
manufacturing floor faster thanever before. And in what are we
are we that far behind? And howdo we get up to speed?
Danielle Duran (37:08):
Yeah, I think, like anything else, like any of
these tools, the man even likeproject management tools, or,
you know, tracking, checklists,tools, if, if it's not
implemented mindfully with allthese other, you know,
objectives, and how we, youknow, it works and how are you
going to train people to use itand what's Okay, and what's not,
(37:31):
what are the guardrails? You'reyou're just adding more content,
you're adding more stuff forpeople to do. And there's only
so much that people can do, justperiod, you know, we maxed out.
So I remember reading somethingnot too long ago about advice
from someone saying this is,it's a great tool in order to
(37:53):
free you up to do the morecomplicated stuff like yeah,
let's let I'm 100%. Let'sautomate all the little things.
Thanks. Yeah.
Dori Gonzalez-Acevedo (38:02):
Well, and there's any automation tool,
right, I mean, that's, that's,that should be the expectation
and the first low hanging fruit,right to get all of those tasks
as automated as possible, sothat we can free up our brains
to take on some higher levelthinking. It's just a matter of,
what do you do? What's next? Andhow does that evolve when you
(38:24):
your basic programs are notreally hitting the marks? In
some some regards? Right? So wetalked a little bit about like,
you know, quality as an ROI, butlike, learning as an ROI is
probably even harder toquantify. I suspect, I haven't
tried have you to do metricsaround that. Tell me.
Danielle Duran (38:46):
So the the, I think one of the trickiest
parts, when, when working withyour cross functional partners
to define what those KPIs aregoing to look like, is not
getting caught up in Oh, it'stoo complex. We can't do it.
Because there's so many factors.
So there's the most common, I'dsay, way to measure ROI, ROI and
(39:09):
learning is the Kirkpatrickmodel, where it's four levels
level one is, did the peoplelike it? And that's where you
get smell sheets, or you get thesurvey at the end, you know, are
you satisfied? And then leveltwo is knowledge transfer. And
that's where the quizzes comein. So do you have evidence to
(39:31):
show that knowledge wastransferred, did did the
knowledge transfer part work?
And then level three isbehaviors. And that's this is
where it starts getting tricky,but and you have to define it on
the front end. So you define I'mgoing to do X training. The
behavior I want to change or thebehavior I want to build is x
(39:55):
and so how will I identify itlater? And that's hard is is
who's going to identify it? Howwill it be reported? How will
you record it? Right? And isthere different observers, and
you have to validate thedifferent observers. And that's
where in education, you'd userubrics, and then you train
people to rubrics and you norm.
But that's just not somethingI've ever really seen done here.
And I think, because it's notcommonly, then people want to
(40:20):
say, Well, you can't do it well,yet, you can. That's how you
grade AP essays, you know,that's how you evaluate
teachers, is you have a rubric,and you have people norm, and
you have standards at eachlevel, and then you all rate
until you all get the samescores. So behaviors is level
three, and then level four isbusiness results. So if you
like, let's say, you're workingon your deviation training,
(40:43):
program, deviations take acertain amount of time at the
you know, that's your baseline,you take a certain measure,
there's so many errors, and it'sthis many interactions or
whatever it's going to be, andthen you do your training
programs you test for did peoplelike it, you do a knowledge
check? Do they retain certainamounts of new things that they
have to do differently than thebehaviors? Do they change their
behaviors? are they submittingmore on time? Are they filling
(41:05):
these things out more correctly?
Are they you know, what are thebehaviors and then business
result well, before, before thetraining, deviation took, on
average, you know, 28 days, orhad this many errors, or to this
much time period, right. Andthen if you calculate that into
a cost, each deviation costs$10,000. But then after the
(41:27):
training, with the follow up,net, we're saving $3,000 per
deviation because of thistraining. So there's the
business result. But it takes alot of time. And it takes a lot
of data. And I just wish I hadmore lovely green belt black
belt people to work with. Tohelp with all that. And that's,
that's where I really wish therewas more collaboration between
(41:49):
the operational excellence orcontinuous improvement teams and
learning, because their work isso related. I think it's funny
some time, I remember, a boss Ihad a while ago, we were doing
worked in business operationsgroup. And we were doing an
exercise a marketing exercisefor our department. And one of
(42:10):
the questions was, who's ourgreatest competitor, like, we
offer a service learningcontinuous improvement, we offer
service, who's our greatestcompetitor, and it was our
customers, our greatestcompetitors, because they want
it, they want to do itthemselves, or they've always
done it themselves. They're notused to having support. And so
we're not involved in theseconversations. And so that's the
(42:30):
other advice I give to peopleis, you have experts out there
probably somewhere, you know, goto your learning person, and,
and partner. And, and you don'thave to, I don't know why I
don't know why people don't thatwould be a root cause I'd like
to explore is why when peopledon't reach out to their
learning departments, why not,
Dori Gonzalez-Acevedo (42:52):
but we see with quality in it too, or,
you know, quality in business,you know, like, it's reaching
across the aisle, it's havingshared conversations, it's
having the nuancedconversations, it's, you know,
it's really, for having the timeto do it, having the time to do
it, knowing who to do it with,right, who you know. And it
(43:14):
also, you know, really breakingdown the notion that your, your,
your lane is your lane, andwe're all connected, like we're
all connected, right, and reallyunderstanding that and one of
the things that I love aboutsome of the the good programs
that I've seen is really like anunderstanding from a quality
(43:36):
perspective, how we're allinvolved, right? Like, it's,
it's, it's so incrediblyimportant to have that that
connection, be illustrated bedemonstrated within a whole
organization. And part of thatis a whole Learning Foundation,
from beginning to end this justlike the quality like so,
(43:56):
quality and learning should notbe separate from its, you know,
it's all part of the same thing.
Yeah,
Danielle Duran (44:02):
we're all it's all it's all an organism. Right,
that's the system thinking. Soyou Peter and I'm not I don't
know how much again, this is tome, I would love to have a
system thinking program that youknow, that people understand
here's, you have to consider allthese other areas. And if you
(44:23):
touch it up here, thendownstream what happens just
like data
Dori Gonzalez-Acevedo (44:28):
moves, right so and it moves and shifts
so as you put into that system,you know, it's entropy extra
Pete right, like as you move inIT contracts and moves here and
then it moves and shifts again,and it has to be that continuous
look all the time, right ofwhere the needs and what is in
service of what and what are thein service of the need at the
(44:49):
moment because today isdifferent from six months. And
you know, the past six months.
Danielle Duran (44:56):
I mean, like weeks
Dori Gonzalez-Acevedo (44:59):
Yeah, So, all of those things. So, um,
this is awesome. And I think youand I also don't think this is
going to be our last podcastbecause we have lots to chat
about. I think this is a goodprime prime start here. And I
thank you for your time today. Ireally appreciate
Danielle Duran (45:23):
thank you for your thank you for the
conversation and I'd really loveto know what people want to talk
about more you know, what whatare people curious? Oh, what did
they wish they knew more about?
Dori Gonzalez-Acevedo (45:32):
That's great. So let will ask that
question when we post this out.
Have folks comments on and wemaybe we'll even do maybe a live
q&a next time and we'll get Oh,I love that. Awesome. Yeah.
Cool. All right, Danielle.
Thanks so much.
Danielle Duran (45:48):
Thanks, Tori.
Popular Podcasts
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.