How Might We Put People at the Center of Evaluation?

My mother recently traveled to Argentina where, as part of an occupational therapy program, she and a group of students finished a multiple year project for a rural, community sustained health clinic.  While there, a number of women came into the clinic with back and neck pain.  In order to develop a better understanding of the cause of these ailments, they “shadowed” the women throughout their day.  This  qualitative study resulted in small changes that could help reduce chronic pain.  An example of a solution found through observation was raising the rocks upon which the women washed their clothes, thus reducing the degree the women had to bend and stress to their back and neck.  Interestingly and understandably, the women were less likely to listen or believe a small group of foreign academic practitioners.  What was discovered (and similar to what many marketers in the States call the “nagging factor”) was that educating teenagers who in turn would advise their parents ended up being the best course for preventative care.

Thanks for your comments. While surveys are appropriate in some instances, we find they are generally not as effective for gaining feedback on prototypes. Especially from a group like young children, it is tough to get the same depth of response from a survey as it is from observations and interviews.

The shadowing activity that Beth describes sounds really powerful and is a tool we frequently use to inform design. For other techniques for conducting human-centered observations and prototyping, check out the Human Centered Design Toolkit http://www.ideo.com/work/featured/human-centered-design-toolkit/.

What are other ways readers have tried putting people at the center?

If development is indeed about supporting people in their own development rather than just doing it for them, then people must be at the centre of both how success is defined and how it is evaluated. They need to be generators and users of the information, and they are best able to say how an intervention is impacting on their lives.

When we intervene into existing systems that are complex and unpredictable we become part of the system, and the better we learn to work with and alongside the energies and actors in the system, the more we are likely to achieve shared goals. Qualitative sensing is at least as important as measurement in helping us plan effectively and assess our impact. We need to balance our perspectives with the perspectives of others. These are the things that enable success and enable us to explain why change happens in the way it does and how different actors contribute to achieving and sustaining success.

But while personal stories and perspectives are an essential part of understanding the context and the impact an intervention is having, they need to be collected and used with the same rigour as any other form of evidence.

RCTs are not the only form of rigorous impact assessment, and as the previous writers have commented, while they might have precision, they achieve this by eliminating complexity and end up telling us very little. Contemporary social science has many approaches to rigour. One that everyone understands is the rigour of a court room in which evidence of many different kinds from many different sources are gathered to make a plausible case or argument. This approach may lack the precision of the laboratory, but we do not work in laboratories.

Keystone and its partners is developing an integrated Impact Planning, Assessment and Learning (IPAL) method that offers a rigorous frame and practice for learning in complex social change processes. It integrates a number of separate but closely connected elements that help funders, implementing organisations and the primary constituents plan for and assess their respective impacts.

First we help them develop a comprehensive theory of change that provides a comprehensive and flexible learning framework within which organizations and other constituents can map pathways to outcomes, plan a range of flexible strategies, identify qualitative behavioural as well as quantitative indicators of success and recognize and document evidence of their impact – whether it is positive or negative, planned or unplanned.

Then we propose a system of learning with constituents that involves gathering feedback and other evidence of impact and analyzing this in dialogue with constituents. As they engage in dialogue with constituents, organizations test their theories of change and action against the perspectives of other constituents and adapt them accordingly.

Observing, watching and listening for evidence of change as they go about their daily work. Staff and others can record their observations in ‘change journals’ and after a while it becomes second nature.

Asking for feedback can take the form of an informal conversation or a formal activity like a survey or interviews. Formal feedback techniques such as surveys can generate detailed feedback from large numbers of people on many specific aspects of the organization’s behaviour and performance. Anonymous surveys can be a safe and non-threatening way to bring issues to the surface (or onto the table) that would otherwise remain hidden. Survey results can stimulate deeper learning dialogues that build understanding, trust and more effective action. Qualitative feedback can be quantified and compared offering opportunities for benchmarking and progress measurement.

Discussing…Can be informal or formal using structured dialogue techniques such as focus groups, world café, appreciative enquiry, the most significant change technique, and others. Dialogue is generative when it explores different perspectives and experience on the evidence and what it means.

Documenting can take the form of a simple ‘impact ledger’ spreadsheet or notebook as well as visual representations.

Reflecting – it is very important that organizations build regular times into the rhythms of their organizations in which they can review the evidence and consider what they are learning from it.

Validated impact reporting is a new approach to reporting as a process in which the organisation’s understanding is first reported back to constituents and their feedback is incorporated into public reports that go out to the world.

Andre Proctor is director of programmes at Keystone Accountability.

The focus on grade point averages in higher education creates a problematic dichotomy between leading and following that incentivizes students to hone their skills for memorization above true learning.In order to maintain an acceptable grade point average, it is rarely necessary that a student has actually learned much of anything. Instead, it is far more likely that he or she has just done what has been done before, completed the homework, and knows enough facts to do well on the exam.While this certainly builds knowledge in some respects, the problem stems from the environmental constraints that prevent thinking, discovery, and exploration beyond what everyone else in the room is doing. When students are incentivized to follow a syllabus, learn what they’re told to learn, and do nothing more, that’s exactly what they’ll do.The problem with a GPA as a metric of learning is that it has little relevance to what a student has actually learned and can later apply outside of the classroom. Encouraging creativity in learning by supplementing the GPA with stories, reflection, and far more opportunities to give in to curiosity would not only develop better students tomorrow, but it would teach them how to approach solutions to the growing number of important questions that we’re asking today.