How Might We Celebrate Learning through Evaluation?
- Posted by: Sally Madsen
- on September 2, 2009 at 5:45 am
Why do we evaluate? Sometimes it’s for reflective validation: qualifying the success of a program after it is complete. Other times it’s for active learning: seeing what is working well and what could be improved, and using this insight to change things for the better.
Evaluation for validation has an important role in comparing different approaches: Which approach has the most impact? Which gives the best value for money? How can this affect strategy moving forward? The downside of this type of evaluation is that it often doesn’t produce conclusions until months or years after the actual project has ended—when the opportunity to change course or affect the project outcome is gone. Evaluation for active learning, on the other hand, allows you to take action as soon as a problem is identified. In design and innovation, evaluation for learning is a natural and essential part of the process.
In its most basic form, evaluation for learning is the intuitive thought process of trial and error that occurs within a designer’s mind. But in a complex project, it’s often helpful to put structure around evaluation since it would be unmanageable to have too many pieces of the design under consideration at once. Structure can come in the form of a specific set of hypotheses or prototypes to evaluate, and a schedule governing the times for evaluation and the times for incorporating learnings.
One way we use evaluation at IDEO is to test design ideas at a very early stage before we invest in further development. In a project about how nurses exchange information when they change shifts, an IDEO and Kaiser team tested rough prototypes in a hospital nursing unit. Three times a day, at each shift change, we gathered feedback and iterated on the concepts. By the end of the prototyping and field testing phase, we had a concept that felt right to the nurses; the early stage evaluation had led to a product that was highly intuitive to use. It has now spread to all hospitals in the network, and has been declared a best practice by the Institute for Healthcare Improvement.
Another way we use evaluation is to guide programs at a larger scale. Ripple Effect is a collaboration between IDEO, Acumen Fund, and local water providers in India and Kenya to improve access to drinking water for the poor. (Click here for a short slide presentation on evaluation and the Ripple Effect project.) The project started in India, where we visited organizations in the field, gathered them together for a collaborative design workshop, and provided design business support as they developed new concepts in areas such as water distribution, health awareness, and safe water storage. Throughout the process we were evaluating: collecting stories from the field to inform development, seeking direct feedback from the project’s users on ideas, and later evaluating the success of the pilots themselves. The evaluation not only fed into the innovations, but also informed the process as a whole. We are now conducting the second phase of work in Kenya, which has been redesigned significantly to incorporate learning from India—for example, we are increasing the amount of time we spend in the field in preparation for the pilots, and we are emphasizing the parts of the design process that the Indian organizations found most helpful.
I’d love to pose these questions to our readers:
- When are the best times to evaluate and incorporate learnings?
- How do you ensure that your evaluation findings shape the way interventions or programs are being implemented?
- What are ways to evaluate a project that provide both immediate learning and longer-term validation of results?
Guest blogger Sally Madsen is a designer and project leader at IDEO. On Friday, look for a response on this topic from Lakshmi Karan of the Skoll Foundation.
DISCUSSION: 7 Comments
Hi Sally,Excellent post! Much of what you touched on resonates in my own work and approach here in Ghana. And I’m continually impressed by IDEO’s results and people.I feel like you have touched on important elements of the learning approach to innovation:1) Action learning, which I understand to imply a cycle of reflection -> theorizing/hypothesizing -> planning tests -> executing, and back to reflection;2) Simple questions of “what works?” and “what doesn’t work?” that inspire deeper probing questions through a trail of “why does it (not) work?”;3) Structured hypothesis testing with set frequencies for evaluation and integrating learning, supported by rigorous tracking of indicators;4) Testing before investing.But while you have touched on these critical elements, I feel like you’ve (purposefully?) left out the most critical aspect of the “how”. So I’m also very curious about responses to your questions.A question I’m particularly interested in is how to plan prototyping design and iterations in the real world where there are many factors influencing a pilot’s success. Is an adapted “design of experiments”-type approach for each hypothesis used? How do you know one hospital unit is diverse enough to permit scale with minimal loss of outcome effectiveness? How do you know how many iterations?
Hi Wayne,
Sorry for omitting the “how”– not purposeful. Take a look at IDEO’s Human Centered Design Toolkit, which describes many of our design and evaluation methods in detail and is absolutely open source. These are the fundamentals.
The number of iterations we’ll do depends on many factors– is the goal to create a final market-ready design, or a set of concepts that a client will continue to develop? what’s the budget? what are the time constraints? We’re balancing the desire for as many iterations and learning experience as possible with the practical constraints of timing and resources.
The types of feedback we seek also varies. Early on in the process, we’re looking for inspiration– speaking to “extreme users”, looking for natural work-arounds and creative solutions that people have come up with, seeking inspiration from analogous contexts or industries, and trying to getting a range of perspectives. Later in the process we are still seeking a wide range of stakeholders to speak with, but the focus shifts to direct feedback on a concept or design to inform further iterations. Even later in the process might be statistically significant market research.
Great stuff! Good luck with your work in Ghana.
Great post, Sally. These are really important questions. One point I’d like to add to the mix is about celebrating failure. If you are going to celebrate learning, you also have to be capable of celebrating negative learning. This is understandably tough for many organizations. Yet without the ability to absorb and learn from things that make us uncomfortable or disappointed, how can we iterate until we are able to succeed? Much of this is cultural to particular organizations. For designers, it is quite OK to keep iterating, through both positive and negative results, until we get to success. For others, it is not OK. I’d suggest that we all try to celebrate failures every chance we get, by extracting the learning that comes from failure and celebrating that. Give praise to those who have not gotten what they expected, and encouragement for iterating again. This can help. Considering most everything we do as a “prototype”, much as Google keeps its software in “beta” for many years, is a mental shift that can help as it removes too much importance of each individual iteration and puts the focus back onto the ultimate results. What do you think?
Absolutely. Thanks for adding this, Doug!
(Please excuse the use of capitolization. I am unable to use italisized letters, which would be the appropriate application for stressing an idea in this forum. I am certainly not angry
) I realize this mundane definition of evaluation taught in all education programs may be boring but it does encapsule the phenomena of any relevant evaluation. In order for evaluation to occur a very SPECIFIC OBJECTIVE must be in place to evaluate. What is the foundation of evaluation if there is no basis for which to refer to? There must be a structure of thought ( the prime goal of the project) to use as the tool of evaluation. The larger project (the prime objective) needs to be broken down into smaller units of objectives, the number of which would depend on the complexity of the prime objective. This “breaking down” of objectives into “sub”objectives continues until one is at the starting base of what already exists. What is done with evaluation, which would be the conclusion of whether or not each objective was met, can be further studied to whatever degree, depending on what is necessary for correction. The depth of the study would be determined, as previously sugggested, the amount of resources the reality of the project and its resouces provide. This breaking down of already successsful projects of a similar nature would provide a skeletal structure of the present objectives and sub objectives of the current project and its desired success. The objectives would be in place before the project was started. It places a more scientific or exact plan of original action and allows for a greater degree of success in any endeavor. Success being the achievement of a specific objective.
This is a great point. In the innovation process, we have objectives in place at the beginning of the project. But since the process of innovation entails coming up with new and unexpected ideas, we also spend time re-evaluating and adjusting objectives midway through the project. We may have the overall goal fixed (e.g. increase accessibility to safe water for 500,000 people earning <$2/day, over 5 years) while the sub-objectives (e.g. goals related to specific ideas or interventions) become clear as we move through the process.
Sally, I really enjoyed reading about how you’ve ensured a
commitment to active learning within the evaluation process. There is always an energy that comes
through when people talk about their work in this way and I will now be sure to
keep up with your progress on the Ripple Effect. In terms of the types of evaluation I would argue that
there ought not to be such a distinct separation between evaluation for
validation and active learning, and rather that they are inextricably linked.
I have worked with a number of non-profit/charitable
organisations and social enterprises (in the UK) who are carrying out fantastic
work within local communities, constantly moving forward and creatively building
on their work. However when
it comes to their formal evaluation e.g. end of year 1, year 5 etc, they leave the
majority of these progressions out and their evaluation becomes a largely lifeless, often tenuous in meaning, set of numbers and case bites.
Evaluation for validation does not need to be carried out in
this way – it can be an accumulation of active learning over a time period/s –
a summary of the critical elements learnt (both the challenging and rewarding
ones – which will inevitably include some quantitative data but critically a
depth of qualitative data). This
requires however a different expectation of what validates a project – is it a
depth of knowledge and meaningfulness or an acceptance of easy to calculate
numbers and sound bites.
A key to the collation of active learning as an evaluation
for validation is recording. People
are often less good at recording their insight as it can seriously stifle the
creative process. However by
recording a summary of key lessons learnt and what this has meant to the
projects development (in a relevant format to the work being carried out to
reduce the stifling) will take less time than doing a separate, from scratch
(or near enough), validation evaluation later – which unfortunately most
projects have to do to satisfy funders.
It requires a redefining of what evaluation information is
meaningful and it places an emphasis on learning in the short, medium and long
term as key to success. Funders in
particular need to encourage this in their projects if they truly want
innovative work – otherwise a significant amount of resource will be spent on quantitative
and limited qualitative information that is not actually that relevant or
helpful to their project. It also
requires a flexibility in targets and a willingness to challenge assumptions on
what is success – and consequently what is the most important information for
decision-making.