More on Evaluation for Learning

This post is a response to "How Might We Celebrate Learning through Evaluation?" Learn more about the conversation here. I...

This post is a response to "How Might We Celebrate Learning through Evaluation?" Learn more about the conversation here.

I appreciated Sally's post on the topic of active learning and the questions she has posed are similar ones we challenge ourselves with at the Packard Foundation. The comments of Sonal Shah (I wasn't there at SoCap, but Beth Kanter was) resonate as well with our own work and approach to balancing the rigor of evaluation along with simultaneous commitment to continuous improvement.

At the Packard Foundation our commitment to effectiveness directly plays out in our evaluation and learning culture and practices. Our approach to evaluation is guided by three main principles:
  1. Success depends on a willingness to solicit feedback and take corrective action when necessary
  2. \n
  3. Improvement should be continuous, and we should learn from our mistakes
  4. \n
  5. Evaluation should be conducted in partnership with those who are doing the work in order to maximize learning and minimize the burden on grantees
  6. \n

Through these principles we acknowledge that no effort can be successful without feedback on a continuous basis and without good data. This has to be done in the spirit of improvement and collaboration and with staff and grantees being allowed to make mistakes, to go in a wrong direction, learn from previous work, and take corrective action. It has to be done with the acknowledgment that it is the grantees that are doing the work and that the grantees have to be involved in that conversation. As a Foundation, we need to be aware that the burden of evaluation can be large and that we work to minimize the burden and maximize the value of evaluation for our grantees.

Over the past four years we have been shifting from evaluation for proof or accountability ("Did the program work?") to evaluation for program improvement ("What did we learn that can help us make the program better?"). The latter reflects an approach we refer to as "real-time" evaluation. For us, real-time means balancing monitoring and evaluation to effectively support learning and continuous improvement as our grant-making strategies are implemented. In practice, this extends further than evaluation, and represents our overall approach to an appropriate monitoring, evaluation, and learning system for each programmatic area. Real-time monitoring and evaluation are integrated to regularly facilitate opportunities for learning to occur and bring timely evaluation data-in accessible formats-to the table for reflection and use in decision making. Rather than focus just on evaluation, we have been encouraging a culture that "thinks evaluatively" throughout the grant-making lifecycle of planning, implementation, monitoring, assessment, and course-correction.

We do not have a one-size-fits-all approach to monitoring and evaluation. Rather, we ask our program staff to consider the following factors when they are formulating their monitoring, evaluation, and learning agendas: What questions do we seek to answer? Who is the audience for this information? What level of rigor do they require to be convinced? How complex are the strategies? What is the timeframe for needing information? Finally, what are the overall program resources being invested? In response to these questions, the evaluation approach selected may range from retrospective to real-time evaluation or a combination of both, using qualitative and quantitative data, and loosely aligned or highly rigorous methods. We also encourage staff to consider their monitoring, evaluation, and learning needs at the beginning of a subprogram. We have found that doing so is more likely to lead to logic models, outcomes, indicators, and dashboards that are useful to informing decision making and program improvement rather than being requirements that are imposed on the subprogram with no connection to programmatic work.

Making the shifts from evaluation for proof to evaluation for program improvement was greatly aided by practices that were already underway within the Foundation. Since 2004 our Preschool subprogram has been engaged with the Harvard Family Research Project in a real-time evaluation. The HFRP's approach represented a new way of doing evaluation at the Foundation. The evaluation has in many ways been a strategic partner by serving as a mechanism for the timely flow of strategic information to facilitate the Preschool subprogram's development. From the start, its emphasis has been on continuous (or real-time) feedback and learning. Because the strategy relied on advocacy and policy change, for which there were practically no established evaluation methods, the evaluation also required methodological creativity. Traditional evaluation approaches in which the evaluator develops an evaluation design and then reports back when the data are all collected and analyzed, or in which the evaluator assesses impact after the strategy has been implemented, would have been less useful here.

The importance of these practices may sound self-evident, i.e., everyone should do these, but they are much harder to actually carry out. Not everything has worked as planned in these evaluations, but both program staff and the evaluators have become skilled at adaptation. The evaluation field has room for growth if more evaluation is going to be real-time. Evaluators themselves are learning how to be both rigorous and faster. I like what IDEO is doing in this space. I think it is just the kind of integration of evaluation rigor, rapid cycles, and continuous feedback that many of us are looking for to build into our practice.

Much has been made of the distinction between evaluations designed for accountability (determining whether a program did what it said it would do) and evaluations designed for learning (supporting ongoing decision making and continuous improvement). In truth, evaluations rarely are either one or the other. Typically they must be both (e.g., a program officer may be more interested in learning while a board member may be more interested in accountability), and the evaluation must find a way to ensure that both users' needs are met.

Balance and feedback are key here. How can we get the highest quality information we need, when we need it, to make the best program strategy decisions without undue burden of reporting among those we rely upon for data? And while we might have these good intentions within the Packard Foundation, they will not mean much if they don't translate into real value, that is, impact. At the forefront, our focus is still on the key questions: What are our goals or outcomes we want to achieve? Are we meeting them? It is the real-time aspect that allows us to better know if we are making progress towards those outcomes, what is contributing to our progress (or lack of it), and therefore how to make better strategy decisions along the way.

Guest blogger Gale Berkowitz directs evaluation at the David and Lucile Packard Foundation.

Julian Meehan

Young leaders from around the world are gathering at the United Nations Headquarters in New York Saturday to address arguably the most urgent issue of our time. The Youth Climate Summit comes on the heels of an international strike spearheaded by Greta Thunberg, the 16-year-old climate activist from Sweden, who arrived in New York via emissions-free sailboat earlier this month.

Translated from Swedish, "berg" means "mountain," so it may feel fated that a young woman with Viking blood in her veins and summit in her name would be at the helm. But let's go out on a limb and presume Thunberg, in keeping with most activists, would chafe at the notion of pre-ordained "destiny," and rightly so. Destiny is passive — it happens to you. It's also egomaniacal. Change, on the other hand, is active; you have to fight. And it is humble. "We need to get angry and understand what is at stake," Thunberg declared. "And then we need to transform that anger into action."

This new generation of activists' most pernicious enemy is denial. The people in charge — complacent politicians and corporation heads who grossly benefit from maintaining the status quo — are buffered from real-life consequences of climate change. But millions of people don't share that privilege. For them, climate change isn't an abstract concept, but a daily state of emergency, whether it comes in the form of "prolonged drought in sub-Saharan Africa…devastating tropical storms sweeping across Southeast Asia, the Caribbean and the Pacific…[or] heatwaves and wildfires," as Amnesty International reportsare all too real problems people are facing on a regular basis.

RELATED: Greta Thunberg urges people to turn to nature to combat climate change

Keep Reading Show less
The Planet

Millions of people in over 150 countries across the globe marched for lawmakers and corporations to take action to help stop climate change on Friday, September 20.

The Climate Strikes were organized by children around the world as an extension of the of the "Fridays for Future" campaign. Students have been walking out of classrooms on Fridays to speak out about political inaction surrounding the climate crisis.

"We need to act right now to stop burning fossil fuels and ensure a rapid energy revolution with equity, reparations and climate justice at its heart," organizers say.

There's no doubt the visual images from the marches send a powerful message to those on the ground but especially those watching from around the world. GOOD's own Gabriel Reilich was on the scene for the largest of the Climate Strikes. Here are 18 of the best signs from the Climate Strike march in New York City.

Keep Reading Show less

September 20th marks the beginning of a pivotal push for the future of our planet. The Global Climate Strike will set the stage for the United Nations Climate Action Summit, where more than 60 nations are expected to build upon their commitment to 2015's Paris Agreement for combating climate change.

Millions of people are expected to take part in an estimated 4,000 events across 130 countries.

Keep Reading Show less
The Planet
via Apple

When the iPhone 11 debuted on September 10, it was met with less enthusiasm than the usual iPhone release. A lot of techies are holding off purchasing the latest gadget until Apple releases a phone with 5G technology.

Major US phone carriers have yet to build out the infrastructure necessary to provide a consistent 5G experience, so Apple didn't feel it necessary to integrate the technology into its latest iPhone.

A dramatic new feature on the iPhone 11 Pro is its three camera lenses. The three lenses give users the the original wide, plus ultrawide and telephoto options.

Keep Reading Show less
via I love butter / Flickr

We often dismiss our dreams as nonsensical dispatches from the mind while we're deep asleep. But recent research proves that our dreams can definitely affect our waking lives.

People often dream about their significant others and studies show it actually affects how we behave towads them the next day.

"A lot of people don't pay attention to their dreams and are unaware of the impact they have on their state of mind," said Dylan Selterman, psychology lecturer at the University of Maryland, says according to The Huffington Post. "Now we have evidence that there is this association."

Keep Reading Show less