April 2, 2025 • 23 mins
Join us in this engaging episode in which Dr. Adriana Cimetta and Dr. Rebecca Friesen of the Center for Evaluation Assessment Research & Evaluation (CEARE) highlight the journey of three doctoral students from the University of Arizona, who recently clinched the top prize at the 2024 American Evaluation Association's United States Student Evaluation Case Competition. Hannah Douglas, Emilia Mbone Roland-Umasabor, and Yamini Bukhaya share their experiences and insights into the world of evaluation, shedding light on how the discipline serves as a formidable catalyst for change.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Jeffrey Anthony (00:01):
Welcome to the Inquiry Oasis, the University of
Arizona College of Education'spodcast.
Here, in the heart of theSonoran Desert, we bring you
conversations with our esteemedfaculty members and staff whose
research impacts lives fromsouthern Arizona to the far
reaches of the globe.
We explore the transformativepower of education in this
(00:21):
border town where diversecultures and ideas converge,
weaving a tapestry of innovationwith compassion and a sense of
wonder.
So join us as we journeythrough the sands of curiosity,
unearthing insights that enrichand inspire.
Sit back and relax as we inviteyou to dive into the inquiry
oasis.
Dr. Adriana Cimetta (00:44):
Hello, we are excited to talk to you today
.
I'm Adriana Cimetta.
I'm an associate researchprofessor in the Department of
Educational Psychology and thedirector for the Center for
Evaluation, Assessment, Researchand Evaluation, which is a very
long name that we haveshortened to CEARE.
The centers and my researchfocus on evaluating the impact
(01:06):
of educational practices,supports and experiences in
creating equitable educationalopportunities for diverse
populations of students,including inclusive and
culturally responsive evaluationpractices.
Today, we would like to discussand recognize the wonderful
achievements of a team of threedoctoral students from the
Center of Educational Assessment, research and Evaluation Hannah
(01:26):
Douglas, Emilia Roland-Umasaborand Yamini Bhukya.
They are all PhD students ineducational psychology and they
won the 2024 American EvaluationAssociation's United States
Student Evaluation CaseCompetition.
Great job, congratulations.
(01:47):
These students, known as theevaluators ensemble for the
competition, had the top-ratedproposal out of 12 teams that
participated in this nationalcompetition.
I, along with Dr RebeccaFriesen, had the honor of
serving as the team coaches tosupport Emilia, Hannah and
Yamini in preparing for thecompetition.
To give a brief overview of thecompetition, pre-read materials
(02:11):
were distributed to all theteams with guidelines for the
competition.
Then the teams received thecase materials, which contained
details about the organizationand the program that needed to
be evaluated.
The student teams then hadseven hours to read through the
organization's case materialsand develop a comprehensive
evaluation plan.
The students were not allowedto have any assistance from the
(02:32):
coaches on the day of thecompetition, and I just have to
say how proud I am of Hannah,Emilia and Yamini's application
of their evaluation knowledgeand skills to thoughtfully
develop an inclusive andculturally winning responsive
evaluation plan.
Before we talk to the stars oftoday's podcast, I wanted to
give Dr Rebecca Friesen anopportunity to reflect on her
(02:53):
experience supporting andcoaching the students'
development for thiscompetition, Dr.
Friesen.
Dr. Rebecca Friesen (02:59):
It has been an absolute pleasure working
with these people.
They have been so.
They are so thoughtful andhardworking and have worked on a
variety of different evaluationprojects and I am just so, very
proud of them.
I want to say I did not do verymuch at all with coaching them
(03:20):
before the competition.
This is all them.
This is their hard work, theirideas, their knowledge that they
applied and clearly workedreally well together.
It's a very fun, very fun,hardworking, thoughtful group.
So I'm just very, very proud ofthem.
Dr. Adriana Cimetta (03:37):
Thanks, Rebecca.
Okay, so now we're going toswitch to the students, and I
have a couple of questions.
We're going to talk about theirexperience and what evaluation
means to them.
So, first I'm wondering if eachof you could tell us what
inspired you to compete in theAmerican Evaluations US Student
(03:57):
Evaluation Case Competition, andcan you discuss what it means
for you to win, to have won thecompetition?
We'll start with you, Hannah.
Hannah Douglas (04:05):
Absolutely.
I had previously competed theyear before for the inaugural
competition and for me that wasone of the first experiences
working collaboratively with ateam and it was one of my first
experiences writingcollaboratively and I had such a
great experience I couldn'twait to do it again the
following year.
And this year's AEA theconference theme was all about
(04:26):
amplifying voices and for methat was something that I was
really drawn to and I know thatthis particular theme would
provide an opportunity to reallyglean into some of the skills
and resources that I've workedfrom other projects for.
So it was really exciting to beable to kind of pair that with
(04:47):
this year's conference theme.
Dr. Adriana Cimetta (04:49):
Great and Yamini.
Yamini Bhukya (04:51):
I think I was really inspired to compete in
this competition because it's anopportunity for me to apply my
academic learning to the realworld settings and I feel like
evaluation is not just a theorybut it's all about making impact
in the society, and I thinkthis year's competition theme
really, really resonated with meand it was really inspiring to
(05:12):
work with a program which wasfocusing on youth, and I think
it felt like an opportunity forus to grow as evaluators and
also to collaborate with theteam and work on such a
wonderful themed competition.
Dr. Adriana Cimetta (05:24):
Thank you, Emilia.
Emilia Roland-Umasabor (05:26):
Yeah, for me it was just the right
time because I was ready at thattime to get involved in
evaluation.
And that project came in justat the right time when that made
up my mind to say, okay, nowyou're done with classwork, it's
time to get out there and putall what you've learned, all the
theory you've learned, intopractice.
And when I got that email to bepart of the team, I jumped on
(05:50):
it and that was the bestdecision I made because it gave
me an opportunity to workcollaboratively.
For someone who's not techsavvy to be able to do that with
a team of other my teammates itwas really amazing for me.
Dr. Adriana Cimetta (06:06):
Well, it sounds like a great experience
for all of you.
Could you tell us a little bitabout your case competition
project, how you prepared forthe competition, what it was
like?
I know you've all talked aboutthe collaboration, but what it
was like developing yourevaluation plan for this
community organization in a day,receiving the materials and
producing the plan.
Hannah Douglas (06:27):
And I think part of what made it really exciting
to do it was because it wasdone in a day and that it was a
real life case that we wereworking on.
The day of the competition wereceived that case summary with
the full details of the programitself, and so this was Caldera
Youth, arts, education nonprofit, based in Portland, Oregon,
where our conference will takeplace.
(06:48):
And I think that was what wasexciting too is that we were
working on a real life case andthe evaluation proposal, the
pre-reading, everything as faras the preparation for it, they
really outlined whatrequirements needed to be in
there and so as a team we kindof looked at what our strengths
were and kind of decided kind ofhow to divvy up those tasks and
(07:09):
what we could individually workon, what we needed to work on
together and then how toproofread it all.
So I think preparing prior tothe day was looking at just
prior examples from previousyears competitions, looking at
other logic models, focusedevaluation plans and different
examples that we could pull fromthat.
Yamini Bhukya (07:32):
I think, adding to what Hannah said, our project
, our competition about theCaldera program, youth, which
focused mostly on theenvironmental, education, arts
and mentorship with the middleschool and high school students,
so it just gave us about sevenhours on the day of the
competition.
So it was mostly aboutimmersing ourselves, trying to
understand about the projectthat we were working on and it
(07:54):
was mostly about brainstorming,research and teamwork and we
tried to understand theproject's goals, objectives and
its stances on the equity andsocial justice and everything.
So we try to understand how towork our evaluation plan around
the subject and for us.
I think we tried our best tobalance the creativity and the
(08:19):
practicality to make itinteresting and innovative but
also, at the same time,manageable, to create a good
evaluation plan for the youth.
Excellent.
Emilia Roland-Umasabor (08:29):
Hannah and Yamini have kind of said it
all.
But what we did, the previouscases that we looked at and all
the pre-reads that were sent tous were really helpful.
But for me, not knowing exactlywhat you were going to get was
actually kind of overwhelming ina way, because you're like what
(08:51):
do I read, what do I focus on?
I have no idea what I'm goingto get.
What do I focus on?
I have no idea what I'm goingto get.
So what I did at that point,when we did the pre-reading and
determined what we wanted to do,I went back to previous class
5-8, and I looked at my course,the text we had, and I looked at
(09:13):
some of the materials, somechapters that we were given, and
I went through some of them tokind of refresh my memory for
some of the things that we'regoing to encounter when we
actually open the case, becauseonce you've opened that case you
cannot get help from anyone.
You have to start working on it.
Our time starts then.
So I needed to be prepared.
So when we come together, weall bring in our skills and our
(09:34):
resources to make sure we geteverything done in time.
Dr. Adriana Cimetta (09:41):
Thank you.
Well, it clearly makes me happyto hear that your EDP 582
educational evaluation coursehelped prepare you for this
competition.
That is the goal of the courseto be able to go out and do
successful evaluations so I'mglad that was helpful.
Could you also speak to whatelements of the evaluation were
important to the project and whyyou thought they were important
to include in your team'sevaluation plan?
(10:04):
So you all have had thisevaluation course.
Some of you have worked onevaluation projects, so what
components of evaluation andwhat parts of evaluation were
really important to you toincorporate into your plan for
the Caldera Arts Program?
Hannah Douglas (10:20):
Especially knowing the theme of the
conference was on amplifyingvoices in evaluation.
I think it was really importantthat we incorporated partner
input and prioritizedinclusivity throughout the
evaluation plan.
So one of the things that wereally emphasized we were trying
to figure out how to create thelogic model and our evaluation
(10:41):
matrix was how do we include thepartner voices and how do we
make sure that everyone had aequitable representation in the
evaluation plan, and so ourpartner analysis really
prioritized making sure that allthe relevant partners were not
only considered but reallyengaged in the process
throughout and so that we hadmore of a holistic and
(11:04):
actionable evaluation plan.
I think we also looked attaking more of a data justice
approach to integrate that intoour evaluation plan to make sure
that we were transparent, thatwe were very community driven,
because this was a youth artsprogram within the community in
Portland, so we wanted to makesure that we were representative
of the community then that wetreated the evaluation findings
(11:27):
in a way that would like promotethat well.
Yamini Bhukya (11:33):
I think, in developing our evaluation plan,
inclusivity and culturalresponsiveness were our top
priorities, because Calderaworks with the youth from
diverse underserved backgrounds,including many BIPOC students
and those from low-income ruralareas, so we felt that it was
crucial to reflect the diversityin our evaluation plan.
So we tried to mainly focus onthe participatory evaluation and
(11:58):
, just like Hannah said, wefocus most on data justice and
we try to integrate bothqualitative and quantitative
methods into it, such as surveys, focus groups and creative
mechanism creative feedbackmechanisms in order to create a
holistic evaluation plan.
Dr. Adriana Cimetta (12:13):
Thanks, just in case people on the
podcast or who are listeningdon't know what BIPOC means
Black Indigenous People of Color.
Thanks, just in case people onthe podcast or who are listening
don't know what BIPOC meansBlack Indigenous People of Color
.
Thanks, I just wanted to makesure everybody was aware of that
term, Emilia.
Emilia Roland-Umasabor (12:25):
Yeah, like my teammates have said from
the pre-read, the majorinformation that they wanted us
to focus on was equity and datajustice, and that was the main
thing that we looked into.
We made sure that we coveredevery aspect of that and not to
leave out any relevant party.
(12:48):
So we looked at all the data,made sure every aspect of it,
all the parties involved in it,the youths, the BIPOC, all of
them, we made sure.
We made sure they were allincluded in the evaluation.
Dr. Adriana Cimetta (13:03):
So thank you.
Well, it sounds like a greatexperience and a successful
experience, shifting away fromthe competition and exploring
what interests you aboutevaluation.
Can each of you discuss how youenvision evaluation or how you
hope to use evaluation to impactcommunities, to improve
people's lives?
Hannah Douglas (13:24):
I see evaluation as a tool really for
empowerment.
I think evaluation really helpscommunities and individuals
gain insight into the program'seffectiveness and I think
evaluation has the impact todrive meaningful change.
And the thing that I love beingpart of evaluation is it's not
discipline specific.
(13:44):
I like being able to see allthe different programs and what
the missions and the values thatare driven by the program or
the service that's beingevaluated, and so I think
evaluation really does impactcommunities.
It does improve people's livesby the results.
People are doing meaningful,creative things and evaluation
(14:05):
just brings that to light andbrings to light the
effectiveness of that program.
I think it also illuminatesthose inequities at times as
well and really supports changeif it needs to be brought about.
I think it is a problem-solvingdriven approach to be able to
say like what you're doing isvery well, this is small ways
you can improve, and so I thinkevaluation is kind of ties into
(14:28):
both positive and negativebenefits for programs, but
ultimately it does drive aboutmeaningful change.
Dr. Adriana Cimetta (14:35):
Thanks, Yamini.
Yamini Bhukya (14:37):
I think evaluation, when done right, has
a real, real power to create ameaningful and a strong change
in a societal impact.
And we see it not only as atool of success but also
empowering the individuals.
And especially when we havemeaningful data, we understand
what's working, what is notworking, how to do like make
improvements or suggestions orany kind of changes.
(14:58):
And I think evaluation alsotalks about accountability of
any particular organization,helping the organization to
improve but also holding on toits mission.
And I think in the case that wewere working as the Caldera
Youth Program, I think thatevaluation plan, which focuses
(15:19):
mostly on the arts and thementorship and the environmental
education, I think that wouldreally help foster success and
the personal growth and buildconfidence of the youth.
So I think it's reallyimportant.
I see evaluation as a catalystfor empowerment, just like
Hannah, because you'reempowering the organization or
(15:42):
the people who are intended toreceive this benefit from
whatever the results of yourevaluation are, you're
empowering them.
I'm going to borrow from atheme.
It's a way of amplifying theirvoices through you including
them in the evaluation process.
You're amplifying their voicesthrough you including them in
the evaluation process.
You're amplifying their voices.
(16:03):
You're giving them thatopportunity to be part of
something.
I also see evaluation as a toolfor data-driven decisions,
because when an evaluation isdone, it generates insights and
that insight is used to generatepolicies and improve those
programs.
So that's how I can seeevaluation.
Dr. Adriana Cimetta (16:25):
Excellent.
I think you all touched uponthe empowerment, the inclusivity
, the social justice and thoseare all really important things
and that really can impact acommunity and having the
information to make thesedata-driven decisions.
Can you talk a little bit moreor tell me why or how evaluation
(16:46):
work is meaningful to you onyour educational journey and
your future academic and careerplans?
What role does evaluationpotentially play in that?
Hannah Douglas (16:57):
I know for me I love evaluation, mainly for me
because it is not isolated to aspecific discipline For me as a
graduate student and I stilldon't fully know what I want to
be when I grow up, and part ofwhat draws me to evaluation is
that I'm not forced to decide,I'm not pigeonholed into one
specific discipline, andevaluation is exciting because
(17:19):
you get to see differentprograms and services, from
policy to education to researchto service.
You get to get insight intoscience courses to community
projects in schools and out ofschools, and so for me, I love
evaluation because I getexposure to so many different
people, different disciplines.
(17:40):
I find that really enrichingand exciting and meaningful.
You get to just see what otherpeople are doing and get to play
a small part in that byproviding evaluation efforts to
showcase the meaningful workthat they're already doing.
Thank you.
Dr. Adriana Cimetta (17:54):
Yamini, how about you?
Yamini Bhukya (17:56):
I think for me, evaluation is really, really
meaningful because it helped meunderstand my academic knowledge
and learning and put it intothe real world settings, and I
think this competition was apivotal experience for me right
now and also for my futureacademic career as well, because
, looking ahead, I really wantto work with social policy,
(18:18):
public policy, educationalpolicy and the nonprofit
organization, so I thinkevaluation would really play a
critical role in those areas andI feel like it's all about
working with the data and makinginformed decisions.
Emilia Roland-Umasabor (18:33):
Yeah, for me, coming from a school
psychology background,evaluation is really meaningful
to me because there's thattransparency and you have the
opportunity to make a meaningfulimpact.
You're able to make a changebecause, once that evaluation is
done, you're able to see, theworld is open.
(18:55):
You're able to see a lot ofthings, you're able to
collaborate with people, you'reable to network, you're able to
make informed decisions withothers.
So that's the way I seeevaluation.
I see it as the what isescaping me right now, but it is
an amazing thing and it'ssomething that everyone should
have an opportunity toexperience, because you're able
(19:19):
to put your skills into practice.
You're able to figure outthings and walk through things
you know through step by step.
Dr. Adriana Cimetta (19:29):
Great.
Well, we've spent a lot of timetoday talking about evaluation,
the competition, what it meansto you, and I would love to
share some resources with ourlisteners.
And I would love to share someresources with our listeners, so
I'd love to hear from each ofyou about a book or a paper or a
resource that has significantlyinfluenced your development as
an evaluator, and could youshare one of these with the
(19:51):
audience, our listeners, andexplain why it's impactful to
you?
Hannah Douglas (19:56):
One of the most powerful ones that I've
encountered.
I had a hard time narrowing itdown to just one, but for me,
the Urban Institute has a Do NoHarm project.
They have a lot of resourcesfocused on data reporting, data
visualization, and they provideda lot of ways for applying
equity, for data visualization,accessibility and how to manage
(20:20):
your data in an equitable way.
The Urban Institute's Do NoHarm Project is probably one of
my favorites that I go to.
Yamini Bhukya (20:29):
For me.
It's a book by Michael Quinn.
It's about utilization-focusedevaluation.
I came across this book duringmy undergrad in one of my
courses, and I never knew Iwould be a part of an evaluation
team during my undergrad in oneof my courses and I never knew
I would be a part of anevaluation team during my PhD.
So this book mainly emphasizesthat evaluations should be
designed and implemented keepingin mind the end users.
It talks about how the findingsare only useful when they're
(20:53):
actionable and meaningful.
Also, again, being a part ofthe CR evaluation lab is also
very critical to me, so yeah,Well, that's good to hear and,
yes, michael Kempatton's work isvery useful.
Dr. Adriana Cimetta (21:06):
Right, the utilization focused evaluation
and Emilia, how about you?
Emilia Roland-Umasabor (21:10):
Yeah, I mentioned this before.
The main book that really is itfor me is the evaluation text.
It's titled Evaluation (21:18):
A Systematic Approach.
It's by [Peter H.
] ossi.
We had that book EDP 582, andthat book was it for me, for a
novice.
That's a book you need to, atext you need to read.
It simplifies everything andtakes you through steps of
evaluation, what every stepmeans and how to go about it.
(21:41):
And then there were some othermaterials from that class that I
still refer to, like theKellogg Logic Model Guide.
That was another one that isreally really, really good.
So if you're into that, checkit out.
Dr. Adriana Cimetta (21:57):
Excellent, that check it out, Excellent.
Well, Hannah, Emilia and Yamini, thank you for your very
insightful reflection on yourexperience participating and
winning the United StatesStudent Evaluation Case
Competition and sharing with usa little bit about your
development as evaluators.
And I wanted to let ourlisteners know that Hannah,
(22:18):
Emilia and Yamini will beformally recognized at the
annual American EvaluationAssociation's conference and
they will be competing as theUnited States representative in
the World Student EvaluationCase Competition.
So I'm very excited for them tohave that experience and
represent the US.
(22:39):
So good luck as you prepare torepresent the US in the World
Student Evaluation CaseCompetition.
Thank you, thank you, thank you.
Welcome to the Inquiry Oasis, the University of
Arizona College of Education'spodcast.
Here, in the heart of theSonoran Desert, we bring you
conversations with our esteemedfaculty members and staff whose
research impacts lives fromsouthern Arizona to the far
reaches of the globe.
We explore the transformativepower of education in this
(00:21):
border town where diversecultures and ideas converge,
weaving a tapestry of innovationwith compassion and a sense of
wonder.
So join us as we journeythrough the sands of curiosity,
unearthing insights that enrichand inspire.
Sit back and relax as we inviteyou to dive into the inquiry
oasis.
Dr. Adriana Cimetta (00:44):
Hello, we are excited to talk to you today
.
I'm Adriana Cimetta.
I'm an associate researchprofessor in the Department of
Educational Psychology and thedirector for the Center for
Evaluation, Assessment, Researchand Evaluation, which is a very
long name that we haveshortened to CEARE.
The centers and my researchfocus on evaluating the impact
(01:06):
of educational practices,supports and experiences in
creating equitable educationalopportunities for diverse
populations of students,including inclusive and
culturally responsive evaluationpractices.
Today, we would like to discussand recognize the wonderful
achievements of a team of threedoctoral students from the
Center of Educational Assessment, research and Evaluation Hannah
(01:26):
Douglas, Emilia Roland-Umasaborand Yamini Bhukya.
They are all PhD students ineducational psychology and they
won the 2024 American EvaluationAssociation's United States
Student Evaluation CaseCompetition.
Great job, congratulations.
(01:47):
These students, known as theevaluators ensemble for the
competition, had the top-ratedproposal out of 12 teams that
participated in this nationalcompetition.
I, along with Dr RebeccaFriesen, had the honor of
serving as the team coaches tosupport Emilia, Hannah and
Yamini in preparing for thecompetition.
To give a brief overview of thecompetition, pre-read materials
(02:11):
were distributed to all theteams with guidelines for the
competition.
Then the teams received thecase materials, which contained
details about the organizationand the program that needed to
be evaluated.
The student teams then hadseven hours to read through the
organization's case materialsand develop a comprehensive
evaluation plan.
The students were not allowedto have any assistance from the
(02:32):
coaches on the day of thecompetition, and I just have to
say how proud I am of Hannah,Emilia and Yamini's application
of their evaluation knowledgeand skills to thoughtfully
develop an inclusive andculturally winning responsive
evaluation plan.
Before we talk to the stars oftoday's podcast, I wanted to
give Dr Rebecca Friesen anopportunity to reflect on her
(02:53):
experience supporting andcoaching the students'
development for thiscompetition, Dr.
Friesen.
Dr. Rebecca Friesen (02:59):
It has been an absolute pleasure working
with these people.
They have been so.
They are so thoughtful andhardworking and have worked on a
variety of different evaluationprojects and I am just so, very
proud of them.
I want to say I did not do verymuch at all with coaching them
(03:20):
before the competition.
This is all them.
This is their hard work, theirideas, their knowledge that they
applied and clearly workedreally well together.
It's a very fun, very fun,hardworking, thoughtful group.
So I'm just very, very proud ofthem.
Dr. Adriana Cimetta (03:37):
Thanks, Rebecca.
Okay, so now we're going toswitch to the students, and I
have a couple of questions.
We're going to talk about theirexperience and what evaluation
means to them.
So, first I'm wondering if eachof you could tell us what
inspired you to compete in theAmerican Evaluations US Student
(03:57):
Evaluation Case Competition, andcan you discuss what it means
for you to win, to have won thecompetition?
We'll start with you, Hannah.
Hannah Douglas (04:05):
Absolutely.
I had previously competed theyear before for the inaugural
competition and for me that wasone of the first experiences
working collaboratively with ateam and it was one of my first
experiences writingcollaboratively and I had such a
great experience I couldn'twait to do it again the
following year.
And this year's AEA theconference theme was all about
(04:26):
amplifying voices and for methat was something that I was
really drawn to and I know thatthis particular theme would
provide an opportunity to reallyglean into some of the skills
and resources that I've workedfrom other projects for.
So it was really exciting to beable to kind of pair that with
(04:47):
this year's conference theme.
Dr. Adriana Cimetta (04:49):
Great and Yamini.
Yamini Bhukya (04:51):
I think I was really inspired to compete in
this competition because it's anopportunity for me to apply my
academic learning to the realworld settings and I feel like
evaluation is not just a theorybut it's all about making impact
in the society, and I thinkthis year's competition theme
really, really resonated with meand it was really inspiring to
(05:12):
work with a program which wasfocusing on youth, and I think
it felt like an opportunity forus to grow as evaluators and
also to collaborate with theteam and work on such a
wonderful themed competition.
Dr. Adriana Cimetta (05:24):
Thank you, Emilia.
Emilia Roland-Umasabor (05:26):
Yeah, for me it was just the right
time because I was ready at thattime to get involved in
evaluation.
And that project came in justat the right time when that made
up my mind to say, okay, nowyou're done with classwork, it's
time to get out there and putall what you've learned, all the
theory you've learned, intopractice.
And when I got that email to bepart of the team, I jumped on
(05:50):
it and that was the bestdecision I made because it gave
me an opportunity to workcollaboratively.
For someone who's not techsavvy to be able to do that with
a team of other my teammates itwas really amazing for me.
Dr. Adriana Cimetta (06:06):
Well, it sounds like a great experience
for all of you.
Could you tell us a little bitabout your case competition
project, how you prepared forthe competition, what it was
like?
I know you've all talked aboutthe collaboration, but what it
was like developing yourevaluation plan for this
community organization in a day,receiving the materials and
producing the plan.
Hannah Douglas (06:27):
And I think part of what made it really exciting
to do it was because it wasdone in a day and that it was a
real life case that we wereworking on.
The day of the competition wereceived that case summary with
the full details of the programitself, and so this was Caldera
Youth, arts, education nonprofit, based in Portland, Oregon,
where our conference will takeplace.
(06:48):
And I think that was what wasexciting too is that we were
working on a real life case andthe evaluation proposal, the
pre-reading, everything as faras the preparation for it, they
really outlined whatrequirements needed to be in
there and so as a team we kindof looked at what our strengths
were and kind of decided kind ofhow to divvy up those tasks and
(07:09):
what we could individually workon, what we needed to work on
together and then how toproofread it all.
So I think preparing prior tothe day was looking at just
prior examples from previousyears competitions, looking at
other logic models, focusedevaluation plans and different
examples that we could pull fromthat.
Yamini Bhukya (07:32):
I think, adding to what Hannah said, our project
, our competition about theCaldera program, youth, which
focused mostly on theenvironmental, education, arts
and mentorship with the middleschool and high school students,
so it just gave us about sevenhours on the day of the
competition.
So it was mostly aboutimmersing ourselves, trying to
understand about the projectthat we were working on and it
(07:54):
was mostly about brainstorming,research and teamwork and we
tried to understand theproject's goals, objectives and
its stances on the equity andsocial justice and everything.
So we try to understand how towork our evaluation plan around
the subject and for us.
I think we tried our best tobalance the creativity and the
(08:19):
practicality to make itinteresting and innovative but
also, at the same time,manageable, to create a good
evaluation plan for the youth.
Excellent.
Emilia Roland-Umasabor (08:29):
Hannah and Yamini have kind of said it
all.
But what we did, the previouscases that we looked at and all
the pre-reads that were sent tous were really helpful.
But for me, not knowing exactlywhat you were going to get was
actually kind of overwhelming ina way, because you're like what
(08:51):
do I read, what do I focus on?
I have no idea what I'm goingto get.
What do I focus on?
I have no idea what I'm goingto get.
So what I did at that point,when we did the pre-reading and
determined what we wanted to do,I went back to previous class
5-8, and I looked at my course,the text we had, and I looked at
(09:13):
some of the materials, somechapters that we were given, and
I went through some of them tokind of refresh my memory for
some of the things that we'regoing to encounter when we
actually open the case, becauseonce you've opened that case you
cannot get help from anyone.
You have to start working on it.
Our time starts then.
So I needed to be prepared.
So when we come together, weall bring in our skills and our
(09:34):
resources to make sure we geteverything done in time.
Dr. Adriana Cimetta (09:41):
Thank you.
Well, it clearly makes me happyto hear that your EDP 582
educational evaluation coursehelped prepare you for this
competition.
That is the goal of the courseto be able to go out and do
successful evaluations so I'mglad that was helpful.
Could you also speak to whatelements of the evaluation were
important to the project and whyyou thought they were important
to include in your team'sevaluation plan?
(10:04):
So you all have had thisevaluation course.
Some of you have worked onevaluation projects, so what
components of evaluation andwhat parts of evaluation were
really important to you toincorporate into your plan for
the Caldera Arts Program?
Hannah Douglas (10:20):
Especially knowing the theme of the
conference was on amplifyingvoices in evaluation.
I think it was really importantthat we incorporated partner
input and prioritizedinclusivity throughout the
evaluation plan.
So one of the things that wereally emphasized we were trying
to figure out how to create thelogic model and our evaluation
(10:41):
matrix was how do we include thepartner voices and how do we
make sure that everyone had aequitable representation in the
evaluation plan, and so ourpartner analysis really
prioritized making sure that allthe relevant partners were not
only considered but reallyengaged in the process
throughout and so that we hadmore of a holistic and
(11:04):
actionable evaluation plan.
I think we also looked attaking more of a data justice
approach to integrate that intoour evaluation plan to make sure
that we were transparent, thatwe were very community driven,
because this was a youth artsprogram within the community in
Portland, so we wanted to makesure that we were representative
of the community then that wetreated the evaluation findings
(11:27):
in a way that would like promotethat well.
Yamini Bhukya (11:33):
I think, in developing our evaluation plan,
inclusivity and culturalresponsiveness were our top
priorities, because Calderaworks with the youth from
diverse underserved backgrounds,including many BIPOC students
and those from low-income ruralareas, so we felt that it was
crucial to reflect the diversityin our evaluation plan.
So we tried to mainly focus onthe participatory evaluation and
(11:58):
, just like Hannah said, wefocus most on data justice and
we try to integrate bothqualitative and quantitative
methods into it, such as surveys, focus groups and creative
mechanism creative feedbackmechanisms in order to create a
holistic evaluation plan.
Dr. Adriana Cimetta (12:13):
Thanks, just in case people on the
podcast or who are listeningdon't know what BIPOC means
Black Indigenous People of Color.
Thanks, just in case people onthe podcast or who are listening
don't know what BIPOC meansBlack Indigenous People of Color
.
Thanks, I just wanted to makesure everybody was aware of that
term, Emilia.
Emilia Roland-Umasabor (12:25):
Yeah, like my teammates have said from
the pre-read, the majorinformation that they wanted us
to focus on was equity and datajustice, and that was the main
thing that we looked into.
We made sure that we coveredevery aspect of that and not to
leave out any relevant party.
(12:48):
So we looked at all the data,made sure every aspect of it,
all the parties involved in it,the youths, the BIPOC, all of
them, we made sure.
We made sure they were allincluded in the evaluation.
Dr. Adriana Cimetta (13:03):
So thank you.
Well, it sounds like a greatexperience and a successful
experience, shifting away fromthe competition and exploring
what interests you aboutevaluation.
Can each of you discuss how youenvision evaluation or how you
hope to use evaluation to impactcommunities, to improve
people's lives?
Hannah Douglas (13:24):
I see evaluation as a tool really for
empowerment.
I think evaluation really helpscommunities and individuals
gain insight into the program'seffectiveness and I think
evaluation has the impact todrive meaningful change.
And the thing that I love beingpart of evaluation is it's not
discipline specific.
(13:44):
I like being able to see allthe different programs and what
the missions and the values thatare driven by the program or
the service that's beingevaluated, and so I think
evaluation really does impactcommunities.
It does improve people's livesby the results.
People are doing meaningful,creative things and evaluation
(14:05):
just brings that to light andbrings to light the
effectiveness of that program.
I think it also illuminatesthose inequities at times as
well and really supports changeif it needs to be brought about.
I think it is a problem-solvingdriven approach to be able to
say like what you're doing isvery well, this is small ways
you can improve, and so I thinkevaluation is kind of ties into
(14:28):
both positive and negativebenefits for programs, but
ultimately it does drive aboutmeaningful change.
Dr. Adriana Cimetta (14:35):
Thanks, Yamini.
Yamini Bhukya (14:37):
I think evaluation, when done right, has
a real, real power to create ameaningful and a strong change
in a societal impact.
And we see it not only as atool of success but also
empowering the individuals.
And especially when we havemeaningful data, we understand
what's working, what is notworking, how to do like make
improvements or suggestions orany kind of changes.
(14:58):
And I think evaluation alsotalks about accountability of
any particular organization,helping the organization to
improve but also holding on toits mission.
And I think in the case that wewere working as the Caldera
Youth Program, I think thatevaluation plan, which focuses
(15:19):
mostly on the arts and thementorship and the environmental
education, I think that wouldreally help foster success and
the personal growth and buildconfidence of the youth.
So I think it's reallyimportant.
I see evaluation as a catalystfor empowerment, just like
Hannah, because you'reempowering the organization or
(15:42):
the people who are intended toreceive this benefit from
whatever the results of yourevaluation are, you're
empowering them.
I'm going to borrow from atheme.
It's a way of amplifying theirvoices through you including
them in the evaluation process.
You're amplifying their voicesthrough you including them in
the evaluation process.
You're amplifying their voices.
(16:03):
You're giving them thatopportunity to be part of
something.
I also see evaluation as a toolfor data-driven decisions,
because when an evaluation isdone, it generates insights and
that insight is used to generatepolicies and improve those
programs.
So that's how I can seeevaluation.
Dr. Adriana Cimetta (16:25):
Excellent.
I think you all touched uponthe empowerment, the inclusivity
, the social justice and thoseare all really important things
and that really can impact acommunity and having the
information to make thesedata-driven decisions.
Can you talk a little bit moreor tell me why or how evaluation
(16:46):
work is meaningful to you onyour educational journey and
your future academic and careerplans?
What role does evaluationpotentially play in that?
Hannah Douglas (16:57):
I know for me I love evaluation, mainly for me
because it is not isolated to aspecific discipline For me as a
graduate student and I stilldon't fully know what I want to
be when I grow up, and part ofwhat draws me to evaluation is
that I'm not forced to decide,I'm not pigeonholed into one
specific discipline, andevaluation is exciting because
(17:19):
you get to see differentprograms and services, from
policy to education to researchto service.
You get to get insight intoscience courses to community
projects in schools and out ofschools, and so for me, I love
evaluation because I getexposure to so many different
people, different disciplines.
(17:40):
I find that really enrichingand exciting and meaningful.
You get to just see what otherpeople are doing and get to play
a small part in that byproviding evaluation efforts to
showcase the meaningful workthat they're already doing.
Thank you.
Dr. Adriana Cimetta (17:54):
Yamini, how about you?
Yamini Bhukya (17:56):
I think for me, evaluation is really, really
meaningful because it helped meunderstand my academic knowledge
and learning and put it intothe real world settings, and I
think this competition was apivotal experience for me right
now and also for my futureacademic career as well, because
, looking ahead, I really wantto work with social policy,
(18:18):
public policy, educationalpolicy and the nonprofit
organization, so I thinkevaluation would really play a
critical role in those areas andI feel like it's all about
working with the data and makinginformed decisions.
Emilia Roland-Umasabor (18:33):
Yeah, for me, coming from a school
psychology background,evaluation is really meaningful
to me because there's thattransparency and you have the
opportunity to make a meaningfulimpact.
You're able to make a changebecause, once that evaluation is
done, you're able to see, theworld is open.
(18:55):
You're able to see a lot ofthings, you're able to
collaborate with people, you'reable to network, you're able to
make informed decisions withothers.
So that's the way I seeevaluation.
I see it as the what isescaping me right now, but it is
an amazing thing and it'ssomething that everyone should
have an opportunity toexperience, because you're able
(19:19):
to put your skills into practice.
You're able to figure outthings and walk through things
you know through step by step.
Dr. Adriana Cimetta (19:29):
Great.
Well, we've spent a lot of timetoday talking about evaluation,
the competition, what it meansto you, and I would love to
share some resources with ourlisteners.
And I would love to share someresources with our listeners, so
I'd love to hear from each ofyou about a book or a paper or a
resource that has significantlyinfluenced your development as
an evaluator, and could youshare one of these with the
(19:51):
audience, our listeners, andexplain why it's impactful to
you?
Hannah Douglas (19:56):
One of the most powerful ones that I've
encountered.
I had a hard time narrowing itdown to just one, but for me,
the Urban Institute has a Do NoHarm project.
They have a lot of resourcesfocused on data reporting, data
visualization, and they provideda lot of ways for applying
equity, for data visualization,accessibility and how to manage
(20:20):
your data in an equitable way.
The Urban Institute's Do NoHarm Project is probably one of
my favorites that I go to.
Yamini Bhukya (20:29):
For me.
It's a book by Michael Quinn.
It's about utilization-focusedevaluation.
I came across this book duringmy undergrad in one of my
courses, and I never knew Iwould be a part of an evaluation
team during my undergrad in oneof my courses and I never knew
I would be a part of anevaluation team during my PhD.
So this book mainly emphasizesthat evaluations should be
designed and implemented keepingin mind the end users.
It talks about how the findingsare only useful when they're
(20:53):
actionable and meaningful.
Also, again, being a part ofthe CR evaluation lab is also
very critical to me, so yeah,Well, that's good to hear and,
yes, michael Kempatton's work isvery useful.
Dr. Adriana Cimetta (21:06):
Right, the utilization focused evaluation
and Emilia, how about you?
Emilia Roland-Umasabor (21:10):
Yeah, I mentioned this before.
The main book that really is itfor me is the evaluation text.
It's titled Evaluation (21:18):
A Systematic Approach.
It's by [Peter H.
] ossi.
We had that book EDP 582, andthat book was it for me, for a
novice.
That's a book you need to, atext you need to read.
It simplifies everything andtakes you through steps of
evaluation, what every stepmeans and how to go about it.
(21:41):
And then there were some othermaterials from that class that I
still refer to, like theKellogg Logic Model Guide.
That was another one that isreally really, really good.
So if you're into that, checkit out.
Dr. Adriana Cimetta (21:57):
Excellent, that check it out, Excellent.
Well, Hannah, Emilia and Yamini, thank you for your very
insightful reflection on yourexperience participating and
winning the United StatesStudent Evaluation Case
Competition and sharing with usa little bit about your
development as evaluators.
And I wanted to let ourlisteners know that Hannah,
(22:18):
Emilia and Yamini will beformally recognized at the
annual American EvaluationAssociation's conference and
they will be competing as theUnited States representative in
the World Student EvaluationCase Competition.
So I'm very excited for them tohave that experience and
represent the US.
(22:39):
So good luck as you prepare torepresent the US in the World
Student Evaluation CaseCompetition.
Thank you, thank you, thank you.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com