Evaluating Systems Change

This post is a response to "How Might We Zoom Out to Evaluating With a Systemic View?" Read more of the conversation here. Much of...

This post is a response to "How Might We Zoom Out to Evaluating With a Systemic View?" Read more of the conversation here.

Much of the work we do in the philanthropic and nonprofit sectors is about changing systems to accelerate social progress. Whether we focus on changing or influencing belief systems, operational systems, health delivery systems, educational systems, financial systems, or any other host of systems, ultimately, we must think holistically and expansively if we want to create meaningful and sustainable change. While it is one thing to conceive of an initiative to stimulate systems change, it is quite another to design and evaluate these efforts in ways that address both the effectiveness of the design process and products, as well as the extent to which the short and long term outcomes have been realized.

By its very definition, a system is an arrangement (pattern, design) of parts which interact with each other within the system's boundaries (form, structure, organization) to function as a whole. The saying, "the whole is greater than the sum of its parts," reflects the notion that it is not enough to focus our evaluative gaze on single goals, objectives, actors, processes, activities, and the like, without attempting to understand the larger system in which the initiative lives.

At FSG Social Impact Advisors, we have become increasingly involved in evaluating initiatives that involve multiple sites and varied project implementation strategies. These projects include a diverse set of stakeholders and implementers, all of whom have a key role in creating systems change-from transforming a downtown community, to creating more informed and engaged citizens, to ensuring healthy oceans and sustainable fishing. We have come to understand that evaluating systems change requires us to consider the following questions before we even begin developing an evaluation approach.
  • What is the system?
  • \n
  • What is the system comprised of (activities, structures, processes)
  • \n
  • Who is in the system? What role do different players have in the system? How do members of the system interact?
  • \n
  • How do actors communicate within the system?
  • \n
  • What external forces influence the system we are studying?
  • \n

Once we explore and map the answers to these and other questions that emerge from this discussion, we can begin designing the evaluation. At the same time, we are learning that evaluating systems change initiatives requires a willingness to be comfortable with ambiguity, a willingness to embrace emergence, and a commitment to engaging stakeholders throughout the evaluation process. In essence, evaluations that adopt a systems perspective require being attuned to small changes, exploring interactions among variables (quantitative and qualitative), and looking for patterns in the seemingly disordered data.

Some wonderful folks from the Ball Foundation, who partner with mid-size urban school districts committed to transforming schooling and learning, take a systems view of their work and the evaluation of it. They suggest that, "most traditional ways of generating metrics assume a machine metaphor-if we measure discrete parts in isolation and work to reduce variation and ensure compliance, we will ultimately get the results we seek. This is not necessarily a bad metaphor; the problem is that it is not entirely suited for the more complex and human systems like school systems and other organizations that we live and work in. While some parts of the system may lend themselves to machine-like metrics, it is essential that we expand our perspective to include a systems view." They suggest that when thinking about metrics, we:
  • Measure results in ways that are descriptive as well as quantitative
  • \n
  • Make meaning around holistic system relationships, dependencies and connections
  • \n
  • Provide feedback, generate learnings and guide direction
  • \n
  • Be adaptive as goals evolve and emerge
  • \n
  • Create measures from within-capacity built for people doing the work to create the measures and make meaning of them
  • \n
  • Define accountability as internal, arising from values, principles and commitments
  • \n

In order to accomplish the above, we need to look beyond the traditional tools and methods of evaluation. In addition to using traditional methods such as surveys and interviews, we should consider options such as mapping tools, social network analysis and story-telling. A distinguishing characteristic of these methods is that they explore patterns, interactions, and relationships between the parts; or in other words, how the parts come together in a dynamic way to form the whole. For example, in the Ball Foundation's formative evaluation process in school districts, in addition to measuring changes in competencies of educators and performance of students through traditional evaluation methods, they are creating an Organizational Capacity Mapping Tool that explores changes in patterns of how teachers, administrators, students, and parents come together to interact, make decisions, solve problems, allocate resources, and communicate with each other. Such a tool is by design collaborative, and engages stakeholders not just in generating the data, but in making meaning around it.

Evaluating with a systems perspective also means being intentional about the learning throughout an evaluation process. It means being open to learning from unexpected outcomes, acknowledging that evaluation designs cannot predetermine all factors that will be of interest, being committed to communicating and reporting evaluation results in user-friendly, accessible ways, using evaluation's findings to inform action, and focusing on "differences that make a difference." (See Virginia Lacayo's powerful article on systems and evaluation.)

As I near the end of this blog entry, I realize that what I have written may appear to be answers to the question, "How might we zoom out to evaluating with a systemic view?" In reality, however, I have many more questions than answers. Some of the questions I hope we will explore are:
  • How do we define the boundaries of any initiative's system-in terms of processes and outcomes?
  • \n
  • How do we know what variables to focus on when there are so many?
  • \n
  • How can we begin to actively co-create evaluation questions and processes with stakeholders, and engage them in collaborative meaning-making that builds commitment for action?
  • \n
  • Given the complexity of human systems, and the varied interactions of variables, causation and attribution will sometimes be difficult to attain. How willing are we to accept contribution as an evaluation outcome?
  • \n
  • Are particular evaluation methods most effective for evaluation systems change?
  • \n
  • How willing are we to let an evaluation's questions and findings emerge through an evaluation process, vs. establishing them a priori?
  • \n

Hallie Preskill directs the Strategic Learning and Evaluation Center at FSG Social Impact Advisors and is based in Seattle, WA. Note: Many thanks to my Ball Foundation colleagues who contributed to this entry.

via The Howard Stern Show / YouTube

Former Secretary of State, first lady, and winner of the popular vote in the 2016 presidential election, Hillary Clinton, sat own for an epic, two-and-a--half hour interview with Howard Stern on his SiriusXM show Wednesday.

She was there to promote "The Book of Gutsy Women," a book about heroic women co-written with her daughter, Chelsea Clinton.

In the far-reaching conversation, Clinton and the self-proclaimed "King of All Media" and, without a doubt, the best interviewer in America discussed everything from Donald Trump's inauguration to her sexuality.

Keep Reading Show less

Offering parental leave for new fathers could help close the gender gap, removing the unfair "motherhood penalty" women receive for taking time off after giving birth. However, a new study finds that parental leave also has a pay gap. Men are less likely to take time off, however, when they do, they're more likely to get paid for it.

A survey of 2,966 men and women conducted by New America found that men are more likely to receive paid parental leave. Over half (52%) of fathers had fully paid parental leave, and 14% of fathers had partially paid parental leave. In comparison, 33% of mothers had fully paid parental leave and 19% had partially paid parental leave.

Keep Reading Show less

Bans on plastic bags and straws can only go so far. Using disposable products, like grabbing a plastic fork when you're on the go, can be incredibly convenient. But these items also contribute to our growing plastic problem.

Fortunately, you can cut down on the amount of waste you produce by cutting down on disposable products. And even more fortunately, there are sustainable (and cute) replacements that won't damage the environment.

Coconut bowls


Who says sustainable can't also be stylish? These cute coconut bowls were handmade using reclaimed coconuts, making each piece one of a kind. Not only are they organic and biodegradable, but they're also durable, in case your dinner parties tend to get out of hand. The matching ebony wood spoons were polished with the same coconut oil as the bowls.

Cocostation Set of 2 Vietnamese Coconut Bowls and Spoons, $14.99; at Amazon

Solar powered phone charger


Why spend time looking around for an outlet when you can just harness the power of the sun? This solar powered phone charger will make sure your phone never dies as long as you can bask in the sun's rays. As an added bonus, this charger was made using eco-friendly silicone rubber. It's win-win all around.

Dizaul Solar Charger, 5000mAh Portable Solar Power Bank, $19.95; at Amazon, $19.95; at Amazon

Herb garden kit

Planter Pro

Put some green in your life with this herb planter. The kit comes with everything you need to get a garden growing, including a moisture meter that helps you determine if your herbs are getting the right amount of food to flourish. All the seeds included are certified to be non-GMO and non-hybrids, meaning you can have fresh, organic herbs right at your fingertips.

Planter Pro's Herb Garden Cedar Planter, $39.00; at Amazonedar Planter, $39.00; at Amazon

Reusable Keurig cups

K & J

Keurig cups are convenient, but they also create a ton of plastic waste. These Keurig-compatible plastic cups are an easy way to cut down on the amount of trash you create without cutting down on your caffeine. Additionally, you won't have to keep on buying K Cups, which means you'll be saving money and the environment.

K&J Reusable Filter Cups, $8.95 for a set of 4,; at Amazon

Low-flow shower head


Low-flow water fixtures can cut down your water consumption, which saves you money while also saving one of the Earth's resources. This shower head was designed with a lighter flow in mind, which means you'll be able to cut down on water usage without feeling like you're cutting down on your shower.

Speakman Low Flow Shower Head, $14.58; at Amazon

Bamboo safety razor


Instead of throwing away a disposable razor every time you shave, invest in an eco-friendly, reusable one. This unisex shaver isn't just sustainable, it's also sharp-looking, which means it would make a great gift for the holidays.

Zomchi Safety Razor, $16.99; at Amazon

The Planet