January 29, 2025 • 37 mins
In April, we hosted a webinar about the recently released IDEAS Impact Framework Toolkit—a free online resource designed to help innovators in the field of early childhood build improved programs and products that are positioned to achieve greater impact in their communities. During the webinar, we provided an overview of the site and had the opportunity to hear from two organizations in the field about how they leveraged the toolkit and its resources to shape their work: Valley Settlement and Raising a Reader. This episode of the Brain Architects podcast features highlights from the webinar. If you’re interested in hearing a full walk through of the toolkit by the Director of our Pediatric Innovation Initiative, Dr. Melanie Berry, please head over to our YouTube channel to view the full webinar recording.
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
Welcome to The Brain Architects, a podcast from the Center on the Developing Child at
(00:07):
Harvard University.
I'm Corrie Zimmerman, the Center's Chief Program Officer.
Our Center believes that advances in the science of child development provide a powerful source
of new ideas that can improve outcomes for children and their caregivers.
By sharing the latest science from the field, we hope to help you make that science actionable
and apply it in your work in ways that can increase your impact.
(00:29):
With that goal in mind, the Center recently released the Ideas Impact Framework Toolkit.
It's a free, online resource designed to help innovators in the field of early childhood
build improved programs and products that are positioned to achieve greater impact in
their communities.
This toolkit is self-guided, self-paced, and provides a structured and flexible approach
that facilitates program development, evaluation, and fast-cycle iteration.
(00:53):
It includes resources to help teams develop and investigate a clear and precise theory
of change.
In April, we hosted a webinar about the toolkit, where we provided an overview of the site
and had the opportunity to hear from teams at several organizations in the field about
how they leverage the toolkit and its resources to shape their work.
We're excited to share those discussions with you here on this episode of the Brain
(01:14):
Architects podcast.
If you're interested in hearing a full walkthrough of the toolkit by the director of our Pediatric
Innovation Initiative, Dr. Melanie Berry, please head over to our YouTube channel to view the
full webinar recording.
You'll also hear from Dr. Melanie Berry during the Q&A portion.
The full ideas toolkit we'll be talking about today can be found at ideas.developingchild.harvard.edu.
(01:38):
And now, without further ado, here's Dr. Eshna Badruzman, the center's senior project manager
for instructional design and the moderator for our panel discussion.
Hello, everyone.
Welcome.
My name is Dr. Eshna Badruzman.
I am a senior project manager for instructional design at the Center on the Developing Child
(01:59):
at Harvard University, or HCDC.
And I'm part of the development team of the Ideas Impact Framework Toolkit.
And today, I'll be your host.
So you may hear me come off mute and help guide presenters.
And I'll be facilitating our question and answer period.
So we are so pleased to be talking to you today about this resource.
(02:22):
The Ideas Impact Framework was born out of more than a decade of the Frontiers of Innovation
Initiative, or FOI.
And some of you may have been partners in that effort.
So while our team no longer offers live training on the framework, we are so excited to be
introducing it to you as a free open access resource.
(02:43):
And we really hope that this format is going to help make ideas accessible to innovators
in the field of early childhood development moving forward.
The framework was developed in partnership with the University of Washington College
of Education and the University of Oregon Center for Translational Neuroscience with
support from the Gates Foundation, the LEGO Foundation, Horticus, and the Hemera Foundation.
(03:08):
I encourage you to check out our history and acknowledgments page of the toolkit for more
information about our various collaborations and supporters throughout time as well.
Now I'll go ahead and introduce our first set of speakers from folks at Valley Settlement.
(03:29):
We have with us Carla Reyes, who is the program manager of the El Paso Cito Mobile Preschool
Program at Valley Settlement, which is a nonprofit that works to create opportunities for the
Latino community in the Aspen to Parishute region of Colorado. Carla joined Valley Settlement
in March 2015 as a preschool teacher for El Paso Cito until June 2021 when she took on
(03:53):
a leadership role. And we also have Sally Boatten, who is the director of development
and communications at Valley Settlement, a nonprofit again serving the rural Aspen to
Parishute region of Colorado with six two-generation programs designed by and for local Latino
immigrant families. And Sally has been with Valley Settlement for over five years and
(04:17):
began managing the organization's evaluation function in 2021. Thank you so much Carla
and Sally, look forward to hearing from you.
Good afternoon everyone. Thank you for inviting us to share our work with you all and how
we have used the framework. I'm going to talk a little bit about the El Paso Cito program
(04:41):
and how it kind of started. The idea of El Paso Cito began in 2011. We had two bilingual
and bicultural community organizers who met one-on-one with about 300 families from the
Aspen to Parishute region of Colorado and they learned about their lives and the barriers
(05:04):
that they faced within our community. One of the findings from the initial listening
tour was that only 1% of Latino children in our community were enrolled in preschool.
We also learned that three of the biggest barriers for families to participate in preschool
programs were language, cost, and transportation resulting in lack of access. Now we have all
(05:28):
this information and we started thinking creatively of different ways that we could bring more
access to preschool education to our community. El Paso Cito was one of the first two-generation
programs that we launched in Valley Settlement to address the needs of preschool education.
And throughout the years, Valley Settlement has continued to learn, evolve, and co-design
(05:53):
programming to respond to community needs. Now our program has four mobile preschool
buses that have been retrofitted into small preschool classrooms. We have two teachers
on the bus and we serve eight children at a time. We provide families with about five
to ten hours of free preschool education. We have about 96 children that we serve annually,
(06:20):
between 40 to 50 children graduating at the end of the school year and moving on to kindergarten.
And currently right now we serve five different neighborhoods within our community and we
strive to build close relationships with families. So our program really is designed to meet
(06:41):
families at where they are at and start breaking down those barriers. We host family nights,
we have home visits with our families, we have parent-teacher conferences, we have different
ways that families can volunteer within our program. We provide a lot of materials for
(07:01):
families to use at home so that they can do home activities and homework packets with
their students and we really try to engage with the families. So each one of our teachers
speaks Spanish, is bilingual and bicultural, so this really allows that bond and that relationship
to build with each one of our families. I'm going to hand it off to Sally Bowen and she's
(07:24):
going to talk a little bit more about how we've used the framework.
Thanks Carla. So several years ago we started working with the team at Frontiers of Innovation
to refine and evolve our evaluation practices. This work included creating theories of change
for each of our programs, researching and recommending observational assessments to
(07:45):
measure participant progress towards our program targets and outcomes, and creating implementation
guides for our programs to detail the kind of critical components of our work and ensure
that future staff can implement programs with fidelity while still continuing to listen
to and evolve alongside families. Since the early days of our programming, Valley Settlement
(08:08):
has invested in evaluation to measure, understand, and strengthen the changes that children and
families create in their lives through our programming. Working with the team at FOI
really brought this to the next level. Over the last few years, we've been working to
be more inclusive and participatory in our evaluation process. So now our entire staff
(08:28):
gathers for three days every summer in what I call an evaluation retreat where we review
our annual program data as a team and then try to answer those questions. Que, por que,
y ahora que? Or as the toolkit outlines, what? What does the data say? Why? Why might the
data say this? Or what does it mean? And finally, now what? What do we do to tweak or change
(08:53):
in our evaluation approach or in our programming based on what we see in the data?
Our teams then create action plans to outline those changes that they want to make. We're
usually tweaking one or more program components for the upcoming year. On day two of the retreat,
teams then go in and refine and evolve their theories of change. So we really see the theory
(09:17):
of change as a living document that breathes and grows alongside our programming. They
identify what targets and outcomes they're interested in measuring for the coming program
year. And then after that evaluation retreat, we work together with an evaluation consultant
to refine our measurement tools. And then I always try to call out and highlight that
(09:40):
I am in the minority at Valley Settlements. Our staff are largely of the community that
we work with. Most of our staff are immigrants or children of immigrants. Many have grown
up in this community or immigrated to the community as adults with young children. And
so they have those shared lived experiences with the participants in our programs. And
(10:01):
many of our staff have actually been former participants in our programs. Having the entire
team participate in this process is incredibly valuable. It really places the experts in
the work, our staff in the evaluator seat. And we gain so much more by having that inclusive
participatory process. And we're really so grateful to have our work shared in the online
(10:26):
toolkit because, you know, I am not an expert in the ideas framework by any means. And that's
kind of the whole point is that it's very usable. You can go in, you can click through
this toolkit, you can see how it all is structured and works. And it just makes for a really
kind of manageable, useful process that you can engage in. Thanks so much.
(10:51):
Thank you, Carla and Sally. Really appreciate you taking the time to share your experiences
with us. Now we'll hear from folks from Raising a Reader. So Raising a Reader supports families
to build, practice and grow reading routines at home. Their award-winning evidence-based
program helps caring adults set their children up for success by creating shared reading
(11:14):
routines, fostering social-emotional learning, healthy family relationships and learning
skills needed to thrive in school and beyond. And first, we'll hear from Michelle Siosenheiman,
who's Senior Vice President, Program and Partnerships. And in her role, Michelle is responsible for
overseeing program development, growth and impact. And then we'll hear from Andres Garcia
(11:36):
Lopez, who's a Senior Project Manager at the Center on the Developing Child. And in his
role, he's coached many early childhood development entrepreneurs, including Raising a Reader,
in developing strategies to maintain their science-based impact while scaling their ventures.
Welcome, Michelle and Andres. Thank you so much for having me. I'll start
(11:59):
with a brief overview of Raising a Reader and how we've used the framework. And then
Andres and I will engage in some conversation. So Raising a Reader is a national family engagement
and early literacy organization through our network of affiliates and partners across
34 states in both rural and urban communities. We engage and support parents and the other
caring adults in children's lives, help strengthen the bonds with their children while building
(12:24):
critical early reading and social-emotional skills. So along with our award-winning multicultural
and multilingual book collection, we provide easy-to-use materials and guides that are
really designed to make the most of that shared reading time in the home. So our work really
does begin, though, with partnering with local agencies who become members of our affiliate
(12:44):
network, a community of practice in which we can share best practices and build connections.
And we provide professional development, technical assistance, and capacity-building support
to this network of affiliates and partners who really work across the intersection of
systems, supporting children and families at the various points throughout their educational
(13:04):
and developmental journey. So that's, you know, in ECE, K-12, Health and Human Services.
And we're really able to meet families in the spaces and places where they are involved.
And how the framework has really impacted our work was that we were introduced to Andres
and the framework at a really critical inflection point in our history. So we're over 23 years
(13:28):
old, and Raising Reader had 39 independent evaluations that proved the success of our
classic Red Book Bag program and its impact on improving and sustaining home literacy
environments. But one thing that we realized through our work with Andres was that there
were critical aspects to our work that we weren't capturing in our theory of change.
Andres, maybe I'll stop there and then we can chat. Does that work?
(13:53):
Sounds good. That works, Michelle. Thanks so much for that overview. And I'm so excited
to be part of this panel. And it's an honor to share it with you, Michelle, and with the
Valley Settlement team. So I'll just add a few things. I was working with Michelle as
part of a fellowship that the Center partnered with the Promise Venture Studio. And as it
(14:21):
was mentioned before, the theory of change on the ideas framework really helps you think
about what are the key ingredients that my organization, my program, works on or provides
to families and or maybe to partners that get to the targets that move the needle towards
(14:41):
my outcomes. One thing that was different about Raising Reader was that they work with
partners. So I wanted to mention that sometimes the ideas framework can be and the theory
of change could be flexible and adaptable to meet your needs. Originally, there are
three columns in the theory of change. But we're working with Michelle, we thought we
(15:02):
should have an extra column because they wanted to look at how working with partners and affiliate
organizations, what the strategies that we're doing raising a reader was getting to the
targets in partners and affiliates and how that was getting to the outcomes with families.
And that was a key component on identifying precisely the actions that get to the targets
(15:25):
and to the outcomes. And I'll pause there so that Michelle can share more about the
specifics of what some of those strategies were and how that helped the organization.
Thanks, Andres. Yeah, so one of the things, including that additional column that Andres
was talking about, it really helped us think through how are we really building that educator
(15:46):
capacity and how are we really providing professional development around early childhood development.
And another aspect to it is that we knew we did it all the time and we had stories about
how we did it all the time. But using the theory of change, the framework to really
make it much more precise is really helping us think through how are we doing it. And
so it also helped us think about how our programming is impacting early relational health through
(16:12):
strengthening healthy family bonds. And so it really has made us to be better poised
to effectively test and evaluate how we are doing this work and what is and what is not
working. Thank you, Michelle. And one comment that I
add as we have about 90% of the participants that are now in the webinar are new to the
(16:32):
framework. Sometimes you may use the framework as a program developer or somebody who is
implementing a program like Michelle, but you could also use it to help other organizations
like the way I have used it as part of the Center on the Developing Child or as Promise
Venture Studio has also used it with social entrepreneurs in their organization. The framework
(16:56):
is really helpful in helping you think through your impact strategy. I'll mention a brief
comment if you're familiar with other frameworks that innovators use like the Lean Startup
or Business Model Canvas or other ones. These helps you think through in a very clear way,
in a simplified way, what are your strategies and how am I getting to the outcomes. And
(17:22):
because it's simple and it can fit in one page, it also facilitates communication, communicating
internally and externally with the families you work with, with the potential funders
and with potential partners. But I'll pause there.
One more thing I just wanted to add about the framework is how it really helped us think
(17:44):
about our innovations and new programming too into our theory of change and help facilitate
that fast cycle iteration because over the last few years, we've developed two new programs
and explored how we were successful in exploring how we can integrate technology into our programming.
And we didn't have that in our previous theory of change. And so being able to build that
(18:05):
into using the framework to build that into our theory of change, thinking about the evaluation,
how do we get that feedback loop? It was really beneficial and helpful for us as we're continuing
to innovate and develop new programs too.
Thank you, Michelle.
Thank you so much for sharing your experiences. And we're going to go ahead and pull some
(18:29):
questions off chat. And I thought maybe we'll start with kind of a somewhat broad question
and that came from Karen Soto is that are there any prerequisites for an organization
to implement the ideas framework? And I know Melanie might want to speak to this.
Sure. I would say there aren't necessarily any standard prerequisites, but I do think
(18:55):
having worked with a lot of different organizations around this framework, there are some conditions
that set you up to be more or less successful or effective using the framework. One thing
I would say is it's important to have all the right people at the table. So I mentioned
that one of the principles of the framework is co-creation and this idea that bringing
(19:18):
together a group of people who have multiple perspectives on the program can be really
valuable. And that might include leadership, people who are responsible for developing
or implementing the program, people who will lead on the evaluation or research efforts.
But equally importantly, you might invite people to contribute who have a role in actually
(19:43):
delivering the service, working directly with kids, families and caregivers, and better
yet invite a representative from the community that you serve to be part of this process.
So that's the first piece is just having the right people at the table. And then the second
thing I would say is timing can be important. So this framework is really designed to help
(20:04):
you prepare for a fast cycle iteration process. So to prepare for a round of collecting data,
reviewing that data, interpreting and analyzing it, making sense of it, and putting what you
learn into practice. So the timing there can be important. Are you set up and prepared
to actually put this plan into action? Do you have the resources you need as everyone
(20:27):
bought in, et cetera? Yeah. And having the authority to put what you learn into practice.
So if you're implementing a program that was developed by someone else, do you have sort
of the leeway to make changes to how you're implementing that program based on what you
learned? Or are there kind of more strict parameters around how you implement that program?
(20:51):
Thanks, Melanie. The next question we have came from Nicholas, and it says, this is a
question from Ms. Reyes from Albucieto. Were there any outcomes or benefits that happened
unexpectedly from developing this program, i.e. unintended consequences that happened
which you did not expect? Yes. So there, it's definitely been a learning curve. We've definitely
(21:17):
had to modify and just evolve the program. One of the biggest changes that we've made
just recently is changing the program from a five-hour week model where children receive
two and a half hours of preschool twice a week to offering five hours of preschool twice
(21:38):
a week. So in total, they're receiving 10 hours a week. And this really came from listening
and taking the time to listen to parents and hear what their needs were. For years, our
parents have been asking for more time on the bus. We're really trying to make an impact
on how many children we served. Like I said, we have a capacity to serve 96 children in
(22:04):
our valley. So that's 96 children that otherwise wouldn't be receiving preschool, you know,
in a traditional preschool setting. And we've noticed recently we've had a decrease in our
enrollment. So it's been a little bit harder to enroll children into our shorter classes.
And I think that has now impacted our school district and our other centers that have grown
(22:27):
their capacity in their centers, which was the ultimate goal to get more children into
preschool and enrolled. So we've now looked at how we can, because we're mobile, we can
now take our program and start serving communities that don't have that access. So it's been
(22:47):
playing out lately that we've noticed these trends.
Thank you. And actually, I just realized that this question is the way it got segmented
in the question answer section. I didn't realize that the Nicholas who asked the question has
the same question for folks at Raising a Reader. So were there any outcomes or benefits that
(23:08):
happened unexpectedly from developing the program?
Sure. So I'll say that we have had 39 independent evaluations that showed, and we knew that
Raising a Reader helped improve home literacy environments, and which is like increased
shared reading time, increased duration, and frequency, improved the number of books in
(23:31):
the home. But one thing that we were hearing from folks was, oh, well, it's helping me
build confidence in supporting my child's early learning in the home. It's really providing
a sense of comfort and support for our families, this daily reading routine. And so creating
a new theory of change with coaching support from Andres to really make much more precise.
(23:53):
We were able to build those kinds of things into our theory of change, which then led
us to improving our measurement tools. So asking specific questions so that we could
actually get some more data around, well, the stories are great. It's great. It's also
helpful to have our surveys also reflect some of that more quantitatively as well.
(24:14):
Thank you, Michelle. And we have one question here that asks if someone could speak to how
this theory of change framework can inform logic model use and development, ensuring
that the information is complementary and not duplicative for programs who choose to
create both types of resources.
(24:37):
I'm happy to feel that, get asked that question a lot. From what I understand, those two terms,
theory of change and logic model are often used actually interchangeably in the field.
And there really isn't solid consensus on how the two are similar and different. The
(24:58):
best guidance on that that I've found is that logic models tend to be more standardized.
They often include inputs, activities, outputs, and then short term and long term outcomes
or variations on that theme. And they're really, they're descriptive. Theories of change are
(25:21):
a bit less standardized. So if you Google the term theory of change, you'll find many,
many, many different approaches to theory of change. But in general, they're intended
to be causal models that really explain how and why the expected changes come about. That's
(25:43):
one way of thinking about that. Then logic models are more descriptive and theories of
change are really intended to be explanatory causal models. Honestly, in practice, I find
that they're, you know, when someone asks you for a logic model or a theory of change,
you really have to follow up to ask what they mean specifically, what they're looking for
there because I think expectations vary widely. Our approach to theory of change is really,
(26:08):
it zooms in on the point of service delivery or the point of contact with kids, families,
caregivers, and follows that through to the ultimate outcomes that you're hoping to see,
which are typically child level in the field of early childhood. And it can be really helpful
to set you up to make a plan for gathering data to better understand whether you're having
(26:32):
the impact that you're hoping to achieve. I hope that's helpful.
Sally, do you want to add to that?
Yeah, so I actually have a real life example of how we've used both at the same time. So
we recently worked with the team at Mathematica to create a two-gen logic model. They did
a project with different two-gen organizations across the country. And so we have been using
(26:57):
theories of change for years in our programs, in our six different programs, to really identify,
you know, what is the, what are our strategies, what are the targets, what are the outcomes
we're trying to have in each of our programs? So it's kind of granular. And then we worked
with the team at Mathematica to create this, like, what does our whole organization do?
(27:20):
And what is the whole change we're trying to make in the community and in children and
families? And so having that overarching logic model that our theories of change then kind
of feed up into, you can see how they interact, how they're incorporated in that larger logic
model has been just really interesting. But we're not duplicating. So we don't have like
(27:44):
a logic model and a theory of change for each of our programs.
Does anybody else want to speak to that before I move to the next question? Okay. So we have
a question here from Eric Marlowe and it asks, in your experience, I think this could be
open to anyone here, how long are the typical iteration cycles? How long do you recommend
(28:07):
evaluating and adapting elements of a given program so that changes are made neither too
soon nor too late?
I can take a first pass, but then I'd be curious to hear from our colleagues at Raising a Reader
and Valley Settlement. If I understood correctly, Eshna, was the question like how often or
how long does it take or maybe a little bit of both?
(28:27):
My interpretation was a little bit of both.
Okay. I think so the way this question gets asked to us often is how fast is fast cycle?
Like, are we talking something you can do in days, weeks, months, years? And the answer
there I think is, unfortunately, it depends. It really depends on the nature of the program
(28:49):
or service or product that you're looking to evaluate and improve. If it's, let's say,
a 10 week parenting group, then a single cycle could take, you know, you'd want a couple
of months to plan and prepare to identify, to develop your theory of change, to identify
(29:09):
your questions, to figure out your study design, figure out the tools you want to use, prepare
for data collection. Then obviously you need the 10 weeks to go by where you're actually
delivering the program to kids and families. And then you'll need some time afterwards
to analyze, interpret, and make sense of that data. But that timeframe is obviously going
(29:33):
to be really different if it's a program that's implemented on a school year calendar, for
instance, or if you've developed something like an app or a website that families can
engage with as they choose. And maybe dosage and engagement looks really different from
parent to parent. So there's no right answer for how long a cycle can take. You really
(29:57):
just need to be thoughtful about what you're hoping to learn. And then in terms of how
often, I think that really varies again from organization to organization and what the
appetite and bandwidth is to engage in this kind of iterative learning. I know that Valley
Settlement, for instance, has really built this into your kind of culture and your routines
(30:18):
as an organization. And it seems like you've developed a really nice kind of annual rhythm.
So maybe I could pass the baton to Sally and Carla to talk more about that.
Yeah. So as you say, Melanie, we really do our evaluation on an annual basis. So most
of our programs happen during the school year. We do pre and post surveys. We also do pre,
(30:40):
mid and post TS Gold assessments for our Boosecito preschool. And then we really, we do the bulk
of data analysis in June and July. And then every July we gather together and do that
evaluation retreat with our entire team. And then in August, we're kind of refining our
evaluation and planning the next evaluation cycle. But what I will say is that when we're
(31:04):
piloting new initiatives, we are trying to be a little more like eyes on as the initiative
is happening. So for example, we worked a few years ago to implement a child development
associate course for family, friends and neighbors providers, and also for high school students
who are Spanish speaking. And we were, we were doing little pre and post assessments
(31:28):
throughout at the beginning and end of kind of each section of the course or module, just
to understand like, was our approach working? How could we pivot and adjust? So as we're
piloting new things, we do try to be a little more rapid, if you will.
I would just echo the same thing that we do have an annual, like an annual cycle, I guess,
(31:49):
where we do an annual evaluation, annual check in with our affiliates is what we call it.
And then we do have a couple of pilots right now where we are calling them mini learning
phases. After each learning phase, then we'll take a look at the feedback and then see what
tweaks or what modifications we need to do to improve the program. And then we have our
second learning phase and things like that. A quick follow up from Nicholas to this question
(32:15):
was, so does that mean that the theory of change is different with each iteration?
So we definitely like, we can evolve our theory of change every year, we go in as a team and
look at the strategies and say like, you know, are you still doing these things? How are
you doing them? You know, we definitely change and evolve in our programming. We're not doing
(32:35):
the same thing every year because programs, neighborhoods, communities change and evolve
and we learn new things every year. So we do go in and tweak our theory of change every
year.
We just revised ours, so we haven't changed it. But one thing I think it is helping us
(32:55):
think about though, as we are developing our program, how to stay focused on what it is
that we really want to do, because there are so many needs and we serve so many different
communities that have different needs. How do we stay true to this, the theory of change
that for our mission and things like that. So it really helps us identify our clear lanes,
(33:17):
like where is our most unique impact and helps us stay there and not kind of stray just because
there might be a funding source over there or something like that.
I'll just add that it's a great question and it gives us a chance to underscore that absolutely
a theory of change can and should evolve over time as you learn. So we call them living
(33:40):
documents. It's not a one-time exercise that you do and you make a PDF of it and it's done.
It's a living document that you come back to after every round of learning and you say,
what did we learn? How can we refine our theory of change? How can we refine our actual program
or product or service? And how could we refine how we're gathering data and learning going
(34:04):
forward? Thank you. So now we have a question from
Megan Crystal asking if we have any examples, if we know of any state level policy or programs
that have used this framework. I have a couple that come to mind. So a while
(34:26):
back as part of the Frontiers of Innovation initiative that Ashna mentioned, there was
a project team who implemented a video coaching program to support childcare and early learning
professionals and that project was done in partnership with the State Department of Early
(34:47):
Learning and this framework was used to sort of articulate theory of change for that approach
and to actually work with partners at the University of Washington to craft the evaluation
plan. That's one example. And then another more recent example that
comes to mind is our center worked with partners at the Massachusetts Department of Early Education
(35:13):
and Care to create an initiative wide theory of change, actually similarly for an effort
underway to bring early care and support organizations to build the capacity of actually childcare
directors across the state. So we worked with them to create an initiative
(35:33):
wide theory of change and then each organization who is providing that service used that initiative
wide theory of change as a template and kind of tailored it for their particular approach.
And I think that's still being used right now as the initiative continues to kind of
facilitate learning and improvement over time. There might be other examples, but those are
(35:55):
two that come to mind. Thanks, Melanie. Does anybody else?
Oh, can I squeeze in one more just in case folks are looking for examples? We worked
with an organization called TOPS, which is based in the Netherlands and they provide,
if I'm remembering correctly, services to families with newborn children. And I think
(36:17):
that's actually like a nationwide program that's used ideas to drive towards greater
impact. And there are resources or references about the TOPS program in the resources section
of the toolkit. I think there's a research article there that talks about their experience.
Great. Thanks, everyone. I was hoping we might have time for one more question, but I actually
(36:40):
see that we're pretty close to the end there. So I just wanted to thank you all for joining
us. Thank our panelists for sharing your experiences and your learning. We really appreciate it.
And thank you all so much for joining. We really, really hope that this resource is
useful for you all. And we wish you the best in your continued work supporting kids and
(37:03):
families.
The Brain Architects is a product of the Center on the Developing Child at Harvard University.
You can find us at developingchild.harvard.edu, where we'll post any resources that were
discussed in this episode. We're also on Twitter at Harvard Center, Facebook at Center on the
Developing Child, and Instagram at Developing Child Harvard.
Welcome to The Brain Architects, a podcast from the Center on the Developing Child at
(00:07):
Harvard University.
I'm Corrie Zimmerman, the Center's Chief Program Officer.
Our Center believes that advances in the science of child development provide a powerful source
of new ideas that can improve outcomes for children and their caregivers.
By sharing the latest science from the field, we hope to help you make that science actionable
and apply it in your work in ways that can increase your impact.
(00:29):
With that goal in mind, the Center recently released the Ideas Impact Framework Toolkit.
It's a free, online resource designed to help innovators in the field of early childhood
build improved programs and products that are positioned to achieve greater impact in
their communities.
This toolkit is self-guided, self-paced, and provides a structured and flexible approach
that facilitates program development, evaluation, and fast-cycle iteration.
(00:53):
It includes resources to help teams develop and investigate a clear and precise theory
of change.
In April, we hosted a webinar about the toolkit, where we provided an overview of the site
and had the opportunity to hear from teams at several organizations in the field about
how they leverage the toolkit and its resources to shape their work.
We're excited to share those discussions with you here on this episode of the Brain
(01:14):
Architects podcast.
If you're interested in hearing a full walkthrough of the toolkit by the director of our Pediatric
Innovation Initiative, Dr. Melanie Berry, please head over to our YouTube channel to view the
full webinar recording.
You'll also hear from Dr. Melanie Berry during the Q&A portion.
The full ideas toolkit we'll be talking about today can be found at ideas.developingchild.harvard.edu.
(01:38):
And now, without further ado, here's Dr. Eshna Badruzman, the center's senior project manager
for instructional design and the moderator for our panel discussion.
Hello, everyone.
Welcome.
My name is Dr. Eshna Badruzman.
I am a senior project manager for instructional design at the Center on the Developing Child
(01:59):
at Harvard University, or HCDC.
And I'm part of the development team of the Ideas Impact Framework Toolkit.
And today, I'll be your host.
So you may hear me come off mute and help guide presenters.
And I'll be facilitating our question and answer period.
So we are so pleased to be talking to you today about this resource.
(02:22):
The Ideas Impact Framework was born out of more than a decade of the Frontiers of Innovation
Initiative, or FOI.
And some of you may have been partners in that effort.
So while our team no longer offers live training on the framework, we are so excited to be
introducing it to you as a free open access resource.
(02:43):
And we really hope that this format is going to help make ideas accessible to innovators
in the field of early childhood development moving forward.
The framework was developed in partnership with the University of Washington College
of Education and the University of Oregon Center for Translational Neuroscience with
support from the Gates Foundation, the LEGO Foundation, Horticus, and the Hemera Foundation.
(03:08):
I encourage you to check out our history and acknowledgments page of the toolkit for more
information about our various collaborations and supporters throughout time as well.
Now I'll go ahead and introduce our first set of speakers from folks at Valley Settlement.
(03:29):
We have with us Carla Reyes, who is the program manager of the El Paso Cito Mobile Preschool
Program at Valley Settlement, which is a nonprofit that works to create opportunities for the
Latino community in the Aspen to Parishute region of Colorado. Carla joined Valley Settlement
in March 2015 as a preschool teacher for El Paso Cito until June 2021 when she took on
(03:53):
a leadership role. And we also have Sally Boatten, who is the director of development
and communications at Valley Settlement, a nonprofit again serving the rural Aspen to
Parishute region of Colorado with six two-generation programs designed by and for local Latino
immigrant families. And Sally has been with Valley Settlement for over five years and
(04:17):
began managing the organization's evaluation function in 2021. Thank you so much Carla
and Sally, look forward to hearing from you.
Good afternoon everyone. Thank you for inviting us to share our work with you all and how
we have used the framework. I'm going to talk a little bit about the El Paso Cito program
(04:41):
and how it kind of started. The idea of El Paso Cito began in 2011. We had two bilingual
and bicultural community organizers who met one-on-one with about 300 families from the
Aspen to Parishute region of Colorado and they learned about their lives and the barriers
(05:04):
that they faced within our community. One of the findings from the initial listening
tour was that only 1% of Latino children in our community were enrolled in preschool.
We also learned that three of the biggest barriers for families to participate in preschool
programs were language, cost, and transportation resulting in lack of access. Now we have all
(05:28):
this information and we started thinking creatively of different ways that we could bring more
access to preschool education to our community. El Paso Cito was one of the first two-generation
programs that we launched in Valley Settlement to address the needs of preschool education.
And throughout the years, Valley Settlement has continued to learn, evolve, and co-design
(05:53):
programming to respond to community needs. Now our program has four mobile preschool
buses that have been retrofitted into small preschool classrooms. We have two teachers
on the bus and we serve eight children at a time. We provide families with about five
to ten hours of free preschool education. We have about 96 children that we serve annually,
(06:20):
between 40 to 50 children graduating at the end of the school year and moving on to kindergarten.
And currently right now we serve five different neighborhoods within our community and we
strive to build close relationships with families. So our program really is designed to meet
(06:41):
families at where they are at and start breaking down those barriers. We host family nights,
we have home visits with our families, we have parent-teacher conferences, we have different
ways that families can volunteer within our program. We provide a lot of materials for
(07:01):
families to use at home so that they can do home activities and homework packets with
their students and we really try to engage with the families. So each one of our teachers
speaks Spanish, is bilingual and bicultural, so this really allows that bond and that relationship
to build with each one of our families. I'm going to hand it off to Sally Bowen and she's
(07:24):
going to talk a little bit more about how we've used the framework.
Thanks Carla. So several years ago we started working with the team at Frontiers of Innovation
to refine and evolve our evaluation practices. This work included creating theories of change
for each of our programs, researching and recommending observational assessments to
(07:45):
measure participant progress towards our program targets and outcomes, and creating implementation
guides for our programs to detail the kind of critical components of our work and ensure
that future staff can implement programs with fidelity while still continuing to listen
to and evolve alongside families. Since the early days of our programming, Valley Settlement
(08:08):
has invested in evaluation to measure, understand, and strengthen the changes that children and
families create in their lives through our programming. Working with the team at FOI
really brought this to the next level. Over the last few years, we've been working to
be more inclusive and participatory in our evaluation process. So now our entire staff
(08:28):
gathers for three days every summer in what I call an evaluation retreat where we review
our annual program data as a team and then try to answer those questions. Que, por que,
y ahora que? Or as the toolkit outlines, what? What does the data say? Why? Why might the
data say this? Or what does it mean? And finally, now what? What do we do to tweak or change
(08:53):
in our evaluation approach or in our programming based on what we see in the data?
Our teams then create action plans to outline those changes that they want to make. We're
usually tweaking one or more program components for the upcoming year. On day two of the retreat,
teams then go in and refine and evolve their theories of change. So we really see the theory
(09:17):
of change as a living document that breathes and grows alongside our programming. They
identify what targets and outcomes they're interested in measuring for the coming program
year. And then after that evaluation retreat, we work together with an evaluation consultant
to refine our measurement tools. And then I always try to call out and highlight that
(09:40):
I am in the minority at Valley Settlements. Our staff are largely of the community that
we work with. Most of our staff are immigrants or children of immigrants. Many have grown
up in this community or immigrated to the community as adults with young children. And
so they have those shared lived experiences with the participants in our programs. And
(10:01):
many of our staff have actually been former participants in our programs. Having the entire
team participate in this process is incredibly valuable. It really places the experts in
the work, our staff in the evaluator seat. And we gain so much more by having that inclusive
participatory process. And we're really so grateful to have our work shared in the online
(10:26):
toolkit because, you know, I am not an expert in the ideas framework by any means. And that's
kind of the whole point is that it's very usable. You can go in, you can click through
this toolkit, you can see how it all is structured and works. And it just makes for a really
kind of manageable, useful process that you can engage in. Thanks so much.
(10:51):
Thank you, Carla and Sally. Really appreciate you taking the time to share your experiences
with us. Now we'll hear from folks from Raising a Reader. So Raising a Reader supports families
to build, practice and grow reading routines at home. Their award-winning evidence-based
program helps caring adults set their children up for success by creating shared reading
(11:14):
routines, fostering social-emotional learning, healthy family relationships and learning
skills needed to thrive in school and beyond. And first, we'll hear from Michelle Siosenheiman,
who's Senior Vice President, Program and Partnerships. And in her role, Michelle is responsible for
overseeing program development, growth and impact. And then we'll hear from Andres Garcia
(11:36):
Lopez, who's a Senior Project Manager at the Center on the Developing Child. And in his
role, he's coached many early childhood development entrepreneurs, including Raising a Reader,
in developing strategies to maintain their science-based impact while scaling their ventures.
Welcome, Michelle and Andres. Thank you so much for having me. I'll start
(11:59):
with a brief overview of Raising a Reader and how we've used the framework. And then
Andres and I will engage in some conversation. So Raising a Reader is a national family engagement
and early literacy organization through our network of affiliates and partners across
34 states in both rural and urban communities. We engage and support parents and the other
caring adults in children's lives, help strengthen the bonds with their children while building
(12:24):
critical early reading and social-emotional skills. So along with our award-winning multicultural
and multilingual book collection, we provide easy-to-use materials and guides that are
really designed to make the most of that shared reading time in the home. So our work really
does begin, though, with partnering with local agencies who become members of our affiliate
(12:44):
network, a community of practice in which we can share best practices and build connections.
And we provide professional development, technical assistance, and capacity-building support
to this network of affiliates and partners who really work across the intersection of
systems, supporting children and families at the various points throughout their educational
(13:04):
and developmental journey. So that's, you know, in ECE, K-12, Health and Human Services.
And we're really able to meet families in the spaces and places where they are involved.
And how the framework has really impacted our work was that we were introduced to Andres
and the framework at a really critical inflection point in our history. So we're over 23 years
(13:28):
old, and Raising Reader had 39 independent evaluations that proved the success of our
classic Red Book Bag program and its impact on improving and sustaining home literacy
environments. But one thing that we realized through our work with Andres was that there
were critical aspects to our work that we weren't capturing in our theory of change.
Andres, maybe I'll stop there and then we can chat. Does that work?
(13:53):
Sounds good. That works, Michelle. Thanks so much for that overview. And I'm so excited
to be part of this panel. And it's an honor to share it with you, Michelle, and with the
Valley Settlement team. So I'll just add a few things. I was working with Michelle as
part of a fellowship that the Center partnered with the Promise Venture Studio. And as it
(14:21):
was mentioned before, the theory of change on the ideas framework really helps you think
about what are the key ingredients that my organization, my program, works on or provides
to families and or maybe to partners that get to the targets that move the needle towards
(14:41):
my outcomes. One thing that was different about Raising Reader was that they work with
partners. So I wanted to mention that sometimes the ideas framework can be and the theory
of change could be flexible and adaptable to meet your needs. Originally, there are
three columns in the theory of change. But we're working with Michelle, we thought we
(15:02):
should have an extra column because they wanted to look at how working with partners and affiliate
organizations, what the strategies that we're doing raising a reader was getting to the
targets in partners and affiliates and how that was getting to the outcomes with families.
And that was a key component on identifying precisely the actions that get to the targets
(15:25):
and to the outcomes. And I'll pause there so that Michelle can share more about the
specifics of what some of those strategies were and how that helped the organization.
Thanks, Andres. Yeah, so one of the things, including that additional column that Andres
was talking about, it really helped us think through how are we really building that educator
(15:46):
capacity and how are we really providing professional development around early childhood development.
And another aspect to it is that we knew we did it all the time and we had stories about
how we did it all the time. But using the theory of change, the framework to really
make it much more precise is really helping us think through how are we doing it. And
so it also helped us think about how our programming is impacting early relational health through
(16:12):
strengthening healthy family bonds. And so it really has made us to be better poised
to effectively test and evaluate how we are doing this work and what is and what is not
working. Thank you, Michelle. And one comment that I
add as we have about 90% of the participants that are now in the webinar are new to the
(16:32):
framework. Sometimes you may use the framework as a program developer or somebody who is
implementing a program like Michelle, but you could also use it to help other organizations
like the way I have used it as part of the Center on the Developing Child or as Promise
Venture Studio has also used it with social entrepreneurs in their organization. The framework
(16:56):
is really helpful in helping you think through your impact strategy. I'll mention a brief
comment if you're familiar with other frameworks that innovators use like the Lean Startup
or Business Model Canvas or other ones. These helps you think through in a very clear way,
in a simplified way, what are your strategies and how am I getting to the outcomes. And
(17:22):
because it's simple and it can fit in one page, it also facilitates communication, communicating
internally and externally with the families you work with, with the potential funders
and with potential partners. But I'll pause there.
One more thing I just wanted to add about the framework is how it really helped us think
(17:44):
about our innovations and new programming too into our theory of change and help facilitate
that fast cycle iteration because over the last few years, we've developed two new programs
and explored how we were successful in exploring how we can integrate technology into our programming.
And we didn't have that in our previous theory of change. And so being able to build that
(18:05):
into using the framework to build that into our theory of change, thinking about the evaluation,
how do we get that feedback loop? It was really beneficial and helpful for us as we're continuing
to innovate and develop new programs too.
Thank you, Michelle.
Thank you so much for sharing your experiences. And we're going to go ahead and pull some
(18:29):
questions off chat. And I thought maybe we'll start with kind of a somewhat broad question
and that came from Karen Soto is that are there any prerequisites for an organization
to implement the ideas framework? And I know Melanie might want to speak to this.
Sure. I would say there aren't necessarily any standard prerequisites, but I do think
(18:55):
having worked with a lot of different organizations around this framework, there are some conditions
that set you up to be more or less successful or effective using the framework. One thing
I would say is it's important to have all the right people at the table. So I mentioned
that one of the principles of the framework is co-creation and this idea that bringing
(19:18):
together a group of people who have multiple perspectives on the program can be really
valuable. And that might include leadership, people who are responsible for developing
or implementing the program, people who will lead on the evaluation or research efforts.
But equally importantly, you might invite people to contribute who have a role in actually
(19:43):
delivering the service, working directly with kids, families and caregivers, and better
yet invite a representative from the community that you serve to be part of this process.
So that's the first piece is just having the right people at the table. And then the second
thing I would say is timing can be important. So this framework is really designed to help
(20:04):
you prepare for a fast cycle iteration process. So to prepare for a round of collecting data,
reviewing that data, interpreting and analyzing it, making sense of it, and putting what you
learn into practice. So the timing there can be important. Are you set up and prepared
to actually put this plan into action? Do you have the resources you need as everyone
(20:27):
bought in, et cetera? Yeah. And having the authority to put what you learn into practice.
So if you're implementing a program that was developed by someone else, do you have sort
of the leeway to make changes to how you're implementing that program based on what you
learned? Or are there kind of more strict parameters around how you implement that program?
(20:51):
Thanks, Melanie. The next question we have came from Nicholas, and it says, this is a
question from Ms. Reyes from Albucieto. Were there any outcomes or benefits that happened
unexpectedly from developing this program, i.e. unintended consequences that happened
which you did not expect? Yes. So there, it's definitely been a learning curve. We've definitely
(21:17):
had to modify and just evolve the program. One of the biggest changes that we've made
just recently is changing the program from a five-hour week model where children receive
two and a half hours of preschool twice a week to offering five hours of preschool twice
(21:38):
a week. So in total, they're receiving 10 hours a week. And this really came from listening
and taking the time to listen to parents and hear what their needs were. For years, our
parents have been asking for more time on the bus. We're really trying to make an impact
on how many children we served. Like I said, we have a capacity to serve 96 children in
(22:04):
our valley. So that's 96 children that otherwise wouldn't be receiving preschool, you know,
in a traditional preschool setting. And we've noticed recently we've had a decrease in our
enrollment. So it's been a little bit harder to enroll children into our shorter classes.
And I think that has now impacted our school district and our other centers that have grown
(22:27):
their capacity in their centers, which was the ultimate goal to get more children into
preschool and enrolled. So we've now looked at how we can, because we're mobile, we can
now take our program and start serving communities that don't have that access. So it's been
(22:47):
playing out lately that we've noticed these trends.
Thank you. And actually, I just realized that this question is the way it got segmented
in the question answer section. I didn't realize that the Nicholas who asked the question has
the same question for folks at Raising a Reader. So were there any outcomes or benefits that
(23:08):
happened unexpectedly from developing the program?
Sure. So I'll say that we have had 39 independent evaluations that showed, and we knew that
Raising a Reader helped improve home literacy environments, and which is like increased
shared reading time, increased duration, and frequency, improved the number of books in
(23:31):
the home. But one thing that we were hearing from folks was, oh, well, it's helping me
build confidence in supporting my child's early learning in the home. It's really providing
a sense of comfort and support for our families, this daily reading routine. And so creating
a new theory of change with coaching support from Andres to really make much more precise.
(23:53):
We were able to build those kinds of things into our theory of change, which then led
us to improving our measurement tools. So asking specific questions so that we could
actually get some more data around, well, the stories are great. It's great. It's also
helpful to have our surveys also reflect some of that more quantitatively as well.
(24:14):
Thank you, Michelle. And we have one question here that asks if someone could speak to how
this theory of change framework can inform logic model use and development, ensuring
that the information is complementary and not duplicative for programs who choose to
create both types of resources.
(24:37):
I'm happy to feel that, get asked that question a lot. From what I understand, those two terms,
theory of change and logic model are often used actually interchangeably in the field.
And there really isn't solid consensus on how the two are similar and different. The
(24:58):
best guidance on that that I've found is that logic models tend to be more standardized.
They often include inputs, activities, outputs, and then short term and long term outcomes
or variations on that theme. And they're really, they're descriptive. Theories of change are
(25:21):
a bit less standardized. So if you Google the term theory of change, you'll find many,
many, many different approaches to theory of change. But in general, they're intended
to be causal models that really explain how and why the expected changes come about. That's
(25:43):
one way of thinking about that. Then logic models are more descriptive and theories of
change are really intended to be explanatory causal models. Honestly, in practice, I find
that they're, you know, when someone asks you for a logic model or a theory of change,
you really have to follow up to ask what they mean specifically, what they're looking for
there because I think expectations vary widely. Our approach to theory of change is really,
(26:08):
it zooms in on the point of service delivery or the point of contact with kids, families,
caregivers, and follows that through to the ultimate outcomes that you're hoping to see,
which are typically child level in the field of early childhood. And it can be really helpful
to set you up to make a plan for gathering data to better understand whether you're having
(26:32):
the impact that you're hoping to achieve. I hope that's helpful.
Sally, do you want to add to that?
Yeah, so I actually have a real life example of how we've used both at the same time. So
we recently worked with the team at Mathematica to create a two-gen logic model. They did
a project with different two-gen organizations across the country. And so we have been using
(26:57):
theories of change for years in our programs, in our six different programs, to really identify,
you know, what is the, what are our strategies, what are the targets, what are the outcomes
we're trying to have in each of our programs? So it's kind of granular. And then we worked
with the team at Mathematica to create this, like, what does our whole organization do?
(27:20):
And what is the whole change we're trying to make in the community and in children and
families? And so having that overarching logic model that our theories of change then kind
of feed up into, you can see how they interact, how they're incorporated in that larger logic
model has been just really interesting. But we're not duplicating. So we don't have like
(27:44):
a logic model and a theory of change for each of our programs.
Does anybody else want to speak to that before I move to the next question? Okay. So we have
a question here from Eric Marlowe and it asks, in your experience, I think this could be
open to anyone here, how long are the typical iteration cycles? How long do you recommend
(28:07):
evaluating and adapting elements of a given program so that changes are made neither too
soon nor too late?
I can take a first pass, but then I'd be curious to hear from our colleagues at Raising a Reader
and Valley Settlement. If I understood correctly, Eshna, was the question like how often or
how long does it take or maybe a little bit of both?
(28:27):
My interpretation was a little bit of both.
Okay. I think so the way this question gets asked to us often is how fast is fast cycle?
Like, are we talking something you can do in days, weeks, months, years? And the answer
there I think is, unfortunately, it depends. It really depends on the nature of the program
(28:49):
or service or product that you're looking to evaluate and improve. If it's, let's say,
a 10 week parenting group, then a single cycle could take, you know, you'd want a couple
of months to plan and prepare to identify, to develop your theory of change, to identify
(29:09):
your questions, to figure out your study design, figure out the tools you want to use, prepare
for data collection. Then obviously you need the 10 weeks to go by where you're actually
delivering the program to kids and families. And then you'll need some time afterwards
to analyze, interpret, and make sense of that data. But that timeframe is obviously going
(29:33):
to be really different if it's a program that's implemented on a school year calendar, for
instance, or if you've developed something like an app or a website that families can
engage with as they choose. And maybe dosage and engagement looks really different from
parent to parent. So there's no right answer for how long a cycle can take. You really
(29:57):
just need to be thoughtful about what you're hoping to learn. And then in terms of how
often, I think that really varies again from organization to organization and what the
appetite and bandwidth is to engage in this kind of iterative learning. I know that Valley
Settlement, for instance, has really built this into your kind of culture and your routines
(30:18):
as an organization. And it seems like you've developed a really nice kind of annual rhythm.
So maybe I could pass the baton to Sally and Carla to talk more about that.
Yeah. So as you say, Melanie, we really do our evaluation on an annual basis. So most
of our programs happen during the school year. We do pre and post surveys. We also do pre,
(30:40):
mid and post TS Gold assessments for our Boosecito preschool. And then we really, we do the bulk
of data analysis in June and July. And then every July we gather together and do that
evaluation retreat with our entire team. And then in August, we're kind of refining our
evaluation and planning the next evaluation cycle. But what I will say is that when we're
(31:04):
piloting new initiatives, we are trying to be a little more like eyes on as the initiative
is happening. So for example, we worked a few years ago to implement a child development
associate course for family, friends and neighbors providers, and also for high school students
who are Spanish speaking. And we were, we were doing little pre and post assessments
(31:28):
throughout at the beginning and end of kind of each section of the course or module, just
to understand like, was our approach working? How could we pivot and adjust? So as we're
piloting new things, we do try to be a little more rapid, if you will.
I would just echo the same thing that we do have an annual, like an annual cycle, I guess,
(31:49):
where we do an annual evaluation, annual check in with our affiliates is what we call it.
And then we do have a couple of pilots right now where we are calling them mini learning
phases. After each learning phase, then we'll take a look at the feedback and then see what
tweaks or what modifications we need to do to improve the program. And then we have our
second learning phase and things like that. A quick follow up from Nicholas to this question
(32:15):
was, so does that mean that the theory of change is different with each iteration?
So we definitely like, we can evolve our theory of change every year, we go in as a team and
look at the strategies and say like, you know, are you still doing these things? How are
you doing them? You know, we definitely change and evolve in our programming. We're not doing
(32:35):
the same thing every year because programs, neighborhoods, communities change and evolve
and we learn new things every year. So we do go in and tweak our theory of change every
year.
We just revised ours, so we haven't changed it. But one thing I think it is helping us
(32:55):
think about though, as we are developing our program, how to stay focused on what it is
that we really want to do, because there are so many needs and we serve so many different
communities that have different needs. How do we stay true to this, the theory of change
that for our mission and things like that. So it really helps us identify our clear lanes,
(33:17):
like where is our most unique impact and helps us stay there and not kind of stray just because
there might be a funding source over there or something like that.
I'll just add that it's a great question and it gives us a chance to underscore that absolutely
a theory of change can and should evolve over time as you learn. So we call them living
(33:40):
documents. It's not a one-time exercise that you do and you make a PDF of it and it's done.
It's a living document that you come back to after every round of learning and you say,
what did we learn? How can we refine our theory of change? How can we refine our actual program
or product or service? And how could we refine how we're gathering data and learning going
(34:04):
forward? Thank you. So now we have a question from
Megan Crystal asking if we have any examples, if we know of any state level policy or programs
that have used this framework. I have a couple that come to mind. So a while
(34:26):
back as part of the Frontiers of Innovation initiative that Ashna mentioned, there was
a project team who implemented a video coaching program to support childcare and early learning
professionals and that project was done in partnership with the State Department of Early
(34:47):
Learning and this framework was used to sort of articulate theory of change for that approach
and to actually work with partners at the University of Washington to craft the evaluation
plan. That's one example. And then another more recent example that
comes to mind is our center worked with partners at the Massachusetts Department of Early Education
(35:13):
and Care to create an initiative wide theory of change, actually similarly for an effort
underway to bring early care and support organizations to build the capacity of actually childcare
directors across the state. So we worked with them to create an initiative
(35:33):
wide theory of change and then each organization who is providing that service used that initiative
wide theory of change as a template and kind of tailored it for their particular approach.
And I think that's still being used right now as the initiative continues to kind of
facilitate learning and improvement over time. There might be other examples, but those are
(35:55):
two that come to mind. Thanks, Melanie. Does anybody else?
Oh, can I squeeze in one more just in case folks are looking for examples? We worked
with an organization called TOPS, which is based in the Netherlands and they provide,
if I'm remembering correctly, services to families with newborn children. And I think
(36:17):
that's actually like a nationwide program that's used ideas to drive towards greater
impact. And there are resources or references about the TOPS program in the resources section
of the toolkit. I think there's a research article there that talks about their experience.
Great. Thanks, everyone. I was hoping we might have time for one more question, but I actually
(36:40):
see that we're pretty close to the end there. So I just wanted to thank you all for joining
us. Thank our panelists for sharing your experiences and your learning. We really appreciate it.
And thank you all so much for joining. We really, really hope that this resource is
useful for you all. And we wish you the best in your continued work supporting kids and
(37:03):
families.
The Brain Architects is a product of the Center on the Developing Child at Harvard University.
You can find us at developingchild.harvard.edu, where we'll post any resources that were
discussed in this episode. We're also on Twitter at Harvard Center, Facebook at Center on the
Developing Child, and Instagram at Developing Child Harvard.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
My Favorite Murder with Karen Kilgariff and Georgia Hardstark
My Favorite Murder is a true crime comedy podcast hosted by Karen Kilgariff and Georgia Hardstark. Each week, Karen and Georgia share compelling true crimes and hometown stories from friends and listeners. Since MFM launched in January of 2016, Karen and Georgia have shared their lifelong interest in true crime and have covered stories of infamous serial killers like the Night Stalker, mysterious cold cases, captivating cults, incredible survivor stories and important events from history like the Tulsa race massacre of 1921. My Favorite Murder is part of the Exactly Right podcast network that provides a platform for bold, creative voices to bring to life provocative, entertaining and relatable stories for audiences everywhere. The Exactly Right roster of podcasts covers a variety of topics including historic true crime, comedic interviews and news, science, pop culture and more. Podcasts on the network include Buried Bones with Kate Winkler Dawson and Paul Holes, That's Messed Up: An SVU Podcast, This Podcast Will Kill You, Bananas and more.
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.