This week marks

A typical tweet: The #IDEO crafted column “Innovation in Evaluation” featured on @GOOD makes an important topic sexier.

So what’s next?
The attention we’ve generated in this series is both energizing and challenging. We hope to find ways to keep this community of passionate people engaged and moving forward together.

IDEO and GOOD plan to co-host “Innovation in Evaluation” events, bringing the community together to share inspiration and maintain a positive momentum. Details are not yet finalized, but we’ve agreed to host the first of these get-togethers in California; the second will take place on the East Coast. Please let us know if you’d like to participate and watch this space for details.

But that’s not all. We’d also like to document the content we’ve generated throughout this discussion in a manner that’s befitting of GOOD and IDEO. So, watch for a shiny publication based on this series.

Key Themes
Some key themes that emerged from this conversation follow.

1. Innovation and evaluation are inseparable
The series started with this statement and the ensuing dialog has reinforced the value of intertwining innovation processes with evaluation mindsets. I have spoken with many measurement professionals, and they’ve each echoed their support for this call to action.

2. Put people at the center of evaluation
This concept was most frequently referenced because the principle is really the center point of the series. Incorporating empathy into the evaluation process ensures that our goal of impacting people doesn’t get lost in the data.

3. Emphasize learning
Overcoming the strong association that people make between “evaluation” and “validation” (after the fact) is one of our primary challenges. We need to emphasize the importance of learning (during an intervention). We also need to determine what is working well and what could be improved, and use this insight to change things for the better.

4. Navigate uncertainty
Uncertainty can be a huge challenge for social enterprises. Anyone engaging in new ideas faces unknown outcomes and risks losing funding opportunities by breaking away from the status quo. Our starting position was to encourage strategies that increase comfort under uncertain conditions, but as this conversation continued over the week, we were surprised that our readers encouraged us to embrace discomfort as an indicator that new ideas are in action.

5. Leverage informed intuition
We’ve hotly debated the issue of intuition: when and how to use it. And we’ve decided that it’s important to distinguish the “informed” intuition of experts from that of some guy with a “gut feeling.” When resources like time and money are scarce, decision-makers can become more efficient and strategic by appropriately leveraging intuition to supplement costly and time-consuming methods.

6. Take a systemic view
This blog consistently referenced the needs of stakeholders. While the constituents, or target audience, are ultimately the focus, it’s also critical to gauge the impact on all stakeholders—including the funders, society at large, and even the surrounding environment. Thinking broadly in this manner increases the likelihood that we can anticipate unintended consequences (both positive and negative).

7. Measure what’s meaningful
You get what you measure, so choose metrics that are meaningful and relevant to the impact you desire. Our readers used this topic as an opportunity to link many of the other blog themes together. We realized that “measuring what’s meaningful” could be a meta-theme for the overall series.

8. Visualize data
Perhaps the hottest topic in the series was on visualization (a topic dear to GOOD readers). It is critical to put information into a form your audience can absorb and use. Results are rendered meaningless if we’re unable to clearly communicate findings.

9. Evaluate in the appropriate time frame
Timing is everything—in car racing, fertility… and evaluation. Balancing short-term and long-term goals is a critical part of the evaluation process—and each requires a different approach.

10. Choose the right tools and methods for the task
We were sorry to report that there is no silver bullet when it comes to evaluation; there’s no one tool or method that is appropriate for every organization. Each team must design an approach for the specific questions they hope to answer.

11. Seek cost-effective tools
The issue of cost was lurking throughout the series—in fact, it underlies most conversations about evaluation. Using cost effective evaluation tools can yield better results with less effort, enabling innovators to do more good with a given amount of resources.

An invitation
In keeping with the style of the series, I will close by posing a few questions.  I invite you to continue the conversation both online and off:

• How might we stay connected as a community?
• How might we implement these themes in our daily work?
• What next steps would you like to see happen (in your organization and in our broader community)?

This post is a response to “How Might We Emphasize Cost Effective Evaluation Tools?” Read more of the conversation here.

We face a time when efficiency, cost cutting, and preparation for the tough times ahead dominate our conversations. Many funding organizations are looking at reduced income due to diminishing returns on their investments, which can mean fewer grants and a much closer eye on what is invested in to begin with.

For some of the organizations I have worked with the reaction to this is to increase fund-raising activity, to apply for more grants, and to ask for less money (to ensure they get any at all). This can result in compromising some of the most important parts of the work to be carried out, and that too often includes the budget allocated to learning and evaluation.

Cutting back on learning and evaluation can help with the survival of a project in the short-term—survive the worst, think about the future later. But while this may be a strategy to get through the tough times, there are problems with this approach.

If an investment is not made in learning and reflection, in the long term it will be difficult to prove its worth. In addition, the project could suffer from not effectively responding to changes in the project’s environment or to developments and challenges as the project progresses.

Some of the most inspirational organizations I have worked with use another approach: they use the urgency of time like these to review resources by their effectiveness to achieve the goal. In practice, this means they question themselves constantly: are we taking the best approach? Are our resources allocated in the best way? What will the resource commitment now mean in the long term? They seek out knowledge and work hard to maximize their resources for the greatest impact of their work.

Cost-effective evaluation tools then become part of the wider picture—will these tools give the quality information needed to make the project work best; and will they give the information required to maximize its impact in the long term? If a tool does not provide critical information vital to the project’s success it will be difficult to categorize it as cost-effective. Superfluous information is a luxury few organizations can indulge in, even in the best of times.

I believe funders need to encourage this approach by ensuring that their investment does not compromise on funds for learning and evaluation, and is flexible enough to allow for the best tools (for the project) to be used, not just traditional or fashionable tools.

Rose Casey-Challies is the Director of Partners in Impact, designing funding programs and working with socially driven organizations to identify their social impact, and share practice that helps to create change. Twitter: @RoseCaseyC

Emphasizing cost effective evaluation tools can get us better results with less effort, enabling innovators to do more good with a given amount of resources. In this week’s discussion, we will think broadly about the costs associated with evaluation throughout the innovation process, and suggest we rethink how we approach evaluation in order to get better results with less effort.

To understand what it means to emphasize cost effective evaluation tools, let’s consider what types of costs are typically associated with evaluating the success of an innovation effort.  These include (what did I miss?):

• Costs of choosing the right methods of evaluation
• Costs of planning and conducting the evaluation
• Costs of processing the evaluation
• Costs of sharing and spreading the results
• Costs of misunderstanding consequences
• Costs of pilot studies and implementation
• Costs of not learning from our successes and failures
• Costs of acting (or not acting) on the evaluation results (opportunity cost of pursuing the wrong path)

I’d like to suggest that it is the last item on this list—opportunity cost—that represents the biggest opportunity for improvement.  What we can least afford is wasting everyone’s time pursuing solutions that aren’t working or that are less effective than other potential solutions that are on the table. This raises the question:

How do we minimize the amount of time being spent on solutions that aren’t working or that should be tweaked to be more effective?

Building on this, here’s an attempt to define what we mean by “cost effective” evaluation tools. Cost effective evaluation tools are (how would you define it?):

• Timely: quick to deploy and minimize time wasted pursuing less effective solutions
• Efficient: use the minimum level of fidelity and rigor needed to inform decision making (sample size, refinement of prototypes, etc.)
• Focused: on the high priority / high uncertainty issues where more learning is needed to move forward (see fig. 1 below)
• Shareable: results are collected, processed, and distributed in a way that tells a compelling story to the relevant stakeholders
• Actionable: findings are distilled down to those which are most meaningful to quickly inform decision making to guide the ongoing innovation process

To get this discussion started, I propose the following starter list of design principles for cost effective evaluation in innovation:

Evaluation is a mindset, not a step in the process.  It should be applied throughout the innovation process to guide the work as it happens. Innovation and development are continuous processes where neat and tidy endpoints rarely exist. Evaluating the work is the work.

Begin with the end in mind. Don’t start an innovation process without clear goals and priorities regarding what you are trying to accomplish and at least some ideas regarding how you will measure how well you’ve accomplished them. Track progress against goals and priorities.

Triage. During the innovation process, take a smart approach to evaluation, recognizing that some unresolved issues will be more important to your success than others. Try this framework for evaluation triage. Rank unresolved issues according to: low to high uncertainty, and  low to high priority (see fig 1).  Priority could mean importance to the success of the project or in terms of impact on decision making.

Then, take action as follows:

• High priority and high uncertainty: be on the lookout for any opportunity to address these issues with the quickest possible evaluation that will enable you to move ahead with sufficient confidence
• Low priority and high uncertainty: measure these issues when it’s convenient; or, wait and see
• High priority and low uncertainty: make an assumption about these issues and move on
• Low priority and low uncertainty: ignore entirely; or, make an assumption and move on

Evaluate the work before the work even starts. Familiarize yourself with how other attempts and solutions have performed in your area and analogous situations. Don’t be afraid to imitate a good idea. Consider probing stakeholders with some “sacrificial concepts” to test the waters before getting too far along.

Be vigilant. As work proceeds, seek out opportunities to clarify issues of high priority and high uncertainty with a quick measurement. Your goal should be to uncover, evaluate, and address as quickly as possible issues affecting your innovation effort where there is both high priority and high uncertainty so that the team can keep moving in the right direction.

Think fast. What could you learn in an hour by talking to one stakeholder? What could you learn in a day by talking to five?

Be efficient. Use the minimum level of fidelity and rigor that you can get away with. Think quick sketch, role-playing exercises, low-res prototypes, small-scale models, etc.

Design the measurements of success while you are designing the offering

Prototype with purpose. Don’t start an evaluation process without clear goals regarding what you are trying to measure.  Design your experiments accordingly, putting your time into the elements that will get you the answers you need.

These principles resonate with what I have seen in my innovation practice at IDEO. Working together with a large energy provider to promote energy conservation, we learned that to innovate effectively, one must have a way to know a good idea from a bad one. And, one wants to know which ideas are working as soon as possible so that they may adjust their efforts for the better. We built a simple tool kit incorporating easy methods for quick idea evaluation, and we created a training program to spread this toolkit across the organization.  It was such a hit that it’s now standard practice that all new energy conservation initiatives will apply these easy evaluation tools as they are being developed.

Thinking about how to incorporate an evaluation mindset into all stages of the innovation process, I often find myself using the idea of The Evaluation Frontier (see fig 2) to navigate the trade-off between accuracy and speed. The idea behind The Evaluation Frontier is that we can choose from a range of evaluation methods during an innovation process, and we can visualize this range on a graph of Accuracy vs. Speed. Since greater accuracy often comes at the expense of speed (in a nonlinear way), we can imagine a frontier along which different evaluation methods lie.

Recognizing the crucial role of timely and efficient evaluation to guide innovation efforts, much of our work at IDEO over the last 20+ years has involved pushing the Evaluation Frontier, particularly in the realm of higher speed / low-to-moderate accuracy evaluations, in pursuit of more effective innovation for our clients. Often this takes the form of new approaches to ethnographic research or quick ways to prototype key elements of a customer experience. These are some of the ways that we minimize the amount of time we spend on solutions that aren’t working or that should be improved. You can find a discussion of some of these methods online here.

I’d love to hear stories about what approaches have worked for you.

Jim Collins leads projects at IDEO with an emphasis on environmental and social impact.

This post is a response to “How Might We Use the Right Tools and Methods for the Task?” Read more of the conversation here.

“Right” means what is right for your organization, your project, your people, your environment, your context, your values, and your beliefs. I get frustrated with people who sell a specific tool or method without a full understanding of what an organization, project, or task is seeking to do. The answer rarely comes in an off-the-shelf prescribed package but rather in methods and tools that provide the absolutely necessary information needed to see whether a task is doing what it intended to do (and if it is not—why not?).  And crucially, the tools and methods provide information that can stimulate and generate improvements.

The combination of tools and methods that provides the vital information then becomes the framework from which relevant information and knowledge can be shared with and communicated to stakeholders. That is, information for stakeholders is a subset of the already gathered information.

Some tips to making a good decision on the “right” tools:

• Really know what your organization, project, or task is about. Get clarity on what you’re trying to achieve and the definition of success. Breakdown and define every word you use; if you can’t explain what you mean, use another word—no sloppiness of words (e.g. by “empower” we mean “a person has gained and now behaves”; by “community” we mean “these people in this area”).
• Predict what you believe will be critical—a collection of truly valuable qualitative data for measuring achievements and challenges. But be prepared to be wrong; the same forum could encourage bias in observation. Sometimes people act in a certain way if they are being assessed, even informally. It may need an independent person to observe or a less invasive observation tool to find out what is really happening.
• Investigate and be open to tool types and methods. Be critical of all tools and methods, and choose the best for your project, which may not be those that an expert suggests (for example, if your project aims to change attitudes of people, it is likely you will know when they have genuinely changed their attitude and whether a tool or method would capture this accurately). It could mean selecting a more resource intensive tool (like involving face-to-face contact) that ultimately gives the most accurate and meaningful data critical to success, rather than a cheaper tool that might give a lot of data but that is less meaningful to your work (such as involving indirect or electronic contact to record attitude changes).
• Break down how you as a person measure  success. What do you look for to see if someone’s behavior has changed?  Does the tool or method include recording of these critical indicators?
• Have ICT caution. Ensure that you are honest when it comes to reviewing the latest technology. It is easy to get swept away by tools that look pretty and are popular but miss collecting really meaningful information.
• Learn lessons from others. It is unlikely that your combination of tools and methods are exactly the same as another organization’s—right is right for what you are trying to achieve in your context. Look for ideas of tools but not a simple adoption of an entire framework.
• Break down any additional information that is required. This could mean financial accountability, ethical accountability, and good management practices. Ensure this information is also being collected (but is not the only information collected).
• Consider separately what is required by funders of the work. A good funder will recognize your logically selected tools and methods, and only request information that has not been proposed but that they need to collectively report across projects (this should be minimal).

When it comes to tools and methods, if it isn’t right for accurately measuring success, providing pivotal learning, or being accountable—why entertain them? And if someone else is insisting you must, what are their motivations—are they really aiming to achieve the same success? Also, are you trying to measure too much? There must always be a balance of resource allocated to measuring, reviewing, and innovating with actually carrying out the work. Keep it simple: prioritize what you must know and work from there.

Questions

• What have been some of the most meaningful tools and methods an organization has used to evaluate its work?
• Have you had experiences of the resources needed to implement the tool or method not justifying the information gained in return?

Rose Casey-Challies is the Director of Partners in Impact, working with socially driven organizations to identify what social impact they want to make and ensuring they have the means to show, learn, and move forward from their experience.

We recently met an organization who complained that it seemed as though evaluation methods are often simply an exercise in “checking the box” to appease their board and funders. For this organization, and many others like them, evaluation can be a time-consuming, costly process that does not lead to better decision-making or better work processes.

This is also at the heart of why so many large and expensive evaluation reports sit on the bookshelf—because before the evaluation was undertaken, no one answered the question:

“What are we trying to provide with our evaluation?”

Answering this question first can lead you to choosing the right methods to use in the evaluation, as well as ensure that the information will be used productively for decision-making and outcomes.

The answer to this question broadly falls into the following two categories:

• Intuitive comfort that we understand the complexity of the problem and the way forward to the right solutions
• Analytical confidence that our solutions are having the intended effects and we can move forward to scale with assurance

Because different sets of tools are better designed to meet either one goal or the other, we first have to address if our organization or funders currently need intuitive comfort or analytic confidence to move forward with a project. At IDEO, we’ve found that both are essential for decision making. 

Some funders that focus on early idea development need smart empirical approaches to feel comfort in pursuing new or risky investments. In these cases, the appropriate tools and methods skew towards the qualitative. Some methods to employ in order to cultivate an understanding of complex systems include ethnographic research, storytelling, and sounding boards. These methods are typically more open-ended, focused on discovery, and inspire the human capacity to synthesize complex data in order to make decisions.

On the other hand, funders that focus on the later phases of optimizing and scaling solutions need confidence in making investment decisions in ideas that have been proven to work. The appropriate tools and methods need to provide analytical robustness and often must be grounded in quantitative data. Some methods to employ in these cases include controlled studies, direct outcome measures, and regression analysis to determine statistically significant causal effects.

There are also some methods that we have found allow the mixture of both richness of understanding and analytical robustness. These hybrid approaches can generate insights that deliver on both the comfort and the confidence level:

• Choice exercises—at IDEO we have used statistical methods that can be applied to learn about how people trade-off value amongst different options. This can be done with very small samples and as part of an interview. The starting point is a comprehensive list of the value drivers underlying a solution. As a result, we understand what drivers or options we need to prioritize in our designs.
• Embedded stories—whenever we build a database, we have to be able to connect to real world stories to validate what we do. Being able to switch back and forth between data and stories is invaluable to not just understand the What but also the Why of observed change.
• System view with data support—linked to the system view approach, we can start building a holistic picture of the world we are operating in. Using stories, we can understand the multivariate system and its causal relationships. After we have achieved this we can extract, abstract, and implement experiments or models to help us measure effectiveness and optimize our offer.

Questions for discussion:

• What methods do you employ to create informed intuition and analytical confidence in decision making?
• Does your organization need complex understanding to move forward or rigorous measures to make decisions?

Tatyana Mamut is an economic anthropologist and content guide for IDEO’s Design for Human Systems work. Johannes Seemann is a business designer at IDEO focusing on bridging qualitative and quantitative methods in insight generation. Read their last entry in this series here.

Many of the blog entries to date have focused on evaluation ranging from longitudinal research-based studies to system theory and the importance of designing the display of data for the greatest impact.

While many of my colleagues and new friends have written important and interesting posts, I am going to attempt to simplify my short entry down to three essential tenants:

• Timing is everything.
• The time to start measuring is now.
• We all need deadlines.

Many of us found the love of our life, our first home, our perfect pet, our dream job etc. based on timing. We were ready and open to the change. Evaluation is the same from the perspective that an organizational leader and the culture of the organization that he or she leads must be ready to make the strategic changes necessary to incorporate evaluation into practice. If you are reading this and are in a position to champion organizational change, read on. If not, read on and then forward the post to the appropriate person/people in your organization.

What kind of evaluation to implement once your organization is ready? Traditional evaluation tends to fulfill a compliance order, retrofitting post-program data into a report, and results in little if any organizational learning. I propose that anyone reading this post should plan to start implementing ongoing performance measurement now—as in, today. The performance measurement I suggest is a process tied to internal improvement. It starts with a discussion to develop a shared vision for success, followed by identifying how that success translates into measurable outcomes and key indicators. Although longitudinal research-based studies of community impact can be useful and serve a purpose, I am outlining a plan for an individual organization to get started. This starts with defining what mission success looks like in a tangible, accessible way, and then developing milestones for data collection, reporting, and management. The goal is program improvement based on the learning now available through the data.

When to implement the process? In short, the indicators or quantifiable data need to occur often enough to allow for course corrections based on that data. I do not propose that the data collection and reporting be so frequent that an organization is in a constant state of flux, trying to make management decisions based on data that is too dynamic to make sense of. But annual data collection is not typically sufficient. The sweet-spot should be driven by internal learning opportunities:  quarterly board meetings, a twice annual management retreat or, dare I write, specific impact gatherings.

I agree with Tim Brown when he writes, “In innovation we have learned that rapid feedback cycles are important when it comes to successful experimentation.” I would expand the idea, as it is clear to me that although we learn something in exit surveys, for example, we gain real meaning after the participant/client/customer leaves our services, returns to their daily routines, and attempts to implement their newly acquired knowledge.

Rapid feedback loops are important but need to be balanced by the settle-in response. After the “wow, that was interesting!” response, we need to learn about the actual results of our work. The problem, of course, is that attrition sets in as soon as participants leave the classroom, so it is incredibly difficult to receive a solid response rate after any amount of time has lapsed.

In working with a large membership organization to develop an outcome dashboard tied to their strategic plan a couple of years ago, I learned that they weren’t actively managing by it. While they found the tool to be useful, their targets fell apart during the economic downturn. Additionally, many of the metrics remained static, so it wasn’t as useful as the initial couple of years. Clearly, it is time for them to reflect and consider how to overcome data fatigue to ensure that the tool is not thrown away for the need to simply update it.  Times change and program offerings and organizations grow and shrink.  Innovation is always at play and there is no question that a performance measurement process and tools need to be updated periodically.

So, you may be thinking great, I get it; this is important and I’m willing to start now or soon. But where do I begin to identify the key outcomes to measure? How do I incorporate the metrics into my process so information, stories, and data are always at my finger tips? What can I realistically ask from a resource-strapped organization at a time of staff layoffs and closing programs? Is now really the time to get started?

I welcome your comments and invite you to visit the outcome/indicators project, a joint and ongoing effort of The Center for What Works and The Urban Institute, for further ideas and free resources including an outcomes portal.

Guest blogger Debra Natenshon is the CEO for The Center for What Works, a Chicago-based nonprofit organization dedicated to performance measurement and benchmarking for the social sector.

We’ve likely all heard these sentiments before (and probably thought them ourselves).

One of the most vexing challenges we face is how to simultaneously balance short-term and long-term results. Some things need to happen today; they just can’t and shouldn’t wait. Other equally or more important projects take longer to develop, prove themselves, or have a noticeable impact. This is especially true when we’re attempting to change a complex, interrelated system.

As a result, the insistence of today often trumps the importance of tomorrow. Standalone, measurable chunks become the focus rather than the impact on people or effectiveness of the system. Performance metrics designed to monitor ongoing performance often aren’t the most appropriate way to assess something new. But unless we pose an alternative, it’s our own fault.

When you’re developing or launching something new you have an opportunity to take the reins and get a handle on how you want to seek and track results. Viewed as a positive, metrics are no more than a discussion starter. With some foresight, we can start the type of discussion that we’d like to start. And, back to the question at hand, we can propose when to pose the question and how to proceed once we have results.

My hunch is that many questions can be answered sooner and with a fair degree of confidence through rapid prototyping, talking more directly with users, and the design of smart experiments. Imagine if you showed up to your next meeting arguing for tracking more concrete results sooner.

In an attempt to get us going, I’d like to propose some guiding questions to help us collect our own time-sensitive results:

• How urgent is our question? How important? Do we have time for patience?
• What’s our time horizon for learning? How quickly could we have an answer (a day, a week, a month, never)?
• How will we know it’s working (learning, change, adoption, something else)?
• What might distract us?
• How will we capture the results?
• Can we design feedback into the experience?
• How aligned are the results and timing of this experiment with the goals and timing of our organization?
• Could we learn faster or more cheaply? If so, how?
• What happens next? What’s our best guess as to what we should do when we “know”?

So here’s my ask to you: go back to your desk right now and give it a shot (see my attempt below) and post your experience in the comments. Based on your experience, what other questions would you add? What doesn’t work for you? For your initiatives what’s the right question? When should you ask it?

Looking at this specific blog post as an example, here are answers to the above questions:

• How important? urgent? patience?: high / low / sure
• Our time horizon for learning: two days
• How we’ll know it’s working: links, comments, people let us know they tried it
• Potential distractions: whiners, flamers, whether “critics” like it
• How we will capture: blog analytics, google searches, stars for the post, retweets
• Designed in feedback: GOODmarks, comments, could and should add share/tweet/digg action links
• Alignment: I think the post fits the goals, but we could be working faster
• Learning faster: I could share it with the guy in 11A next to me (when he wakes up)
• What’s next?: evolve the question list, publish it more broadly

Ryan Jacoby leads IDEO’s business design discipline and writes about innovation strategy on his blog do_mati.

Illustration by Will Etling

This post is a response to “How Might We Visualize Data in More Effective and Inspiring Ways?” Read more of the conversation here.

What’s the difference between saying, “I’m pregnant” and this:

On the one hand, someone who hadn’t seen this visual before might find it unintelligible. The image itself doesn’t say what it’s for, nor how to read it. On the other hand those who know what they are looking at—a pregnancy test—have an immediate, objective status reading. Since they are simply looking at an object, they don’t have to come up with response. Instead they can absorb the information and experience it in their own personal way.

In the budding field of evaluation and impact management, the value comes from communicating the impact—to your team, customers, investors, and others with a stake in what you do. It’s key to make the time to put the results of impact analysis into a form your audience can absorb and use. While it takes a little practice, doing so can be very powerful.

When we introduce impact data to pictures, we tap into a deeply-rooted way humans understand the world, and we can inspire action—which is ultimately the point of measuring impact. But without a factual key or other clear guide that makes the image self-explanatory, data visualization can obscure understanding. All too often such visualizations are used to deceive. This can be hard to see through, partly because people seem to be naturally more skeptical of other people than we are of numbers or images.

One of the most common ways data visualization can be manipulated is by adjusting the scale. For example, CRATX is a publicly-listed fund that invests in debt securities certified under the Community Reinvestment Act, which supports investments in affordable housing and other community development work. How well does CRATX perform?

Here at first glance the returns look huge, until you realize the Y-axis is showing a $0.08 gain.

Another common way data visualization is manipulated is by adjusting the time period. Above, the timeframe was three weeks. Below, CRATX’s percent gain/loss is shown relative to that of the Dow Jones and NASDAQ indices. The time scale is still the same, and CRATX appears to be dwarfed by the two. We are still missing the bigger picture.

Below, if we look at CRATX relative to the Dow Jones Industrial Index and the NASDAQ with the time scale expanded to show June 2007 to October 2009, we finally start to get a good perspective on CRATX relative to the rest of the market. We see that CRATX has been a great, and stable, performer.

The next step would be to identify “peers” of CRATX and plot its relative performance to similarly classed funds.

Things look a lot different over a short period of time than a long period of time. Here are screenshots of the progression by Gapminder (which Engin discussed earlier) of Infant Mortality vs. GDP/Capita, from 1960 to 2000. (Check out the video version.)

Notice how many countries become richer, and that infant mortality decreases significantly, but central and southern Africa are somewhat unchanged. We see strong correlation of infant mortality to income, somewhat regardless of population size.

Also impressive on the Gapminder shots is that the visualization contains four dimensions of data, and five if you press the play button. If all of this data were thrown into a spreadsheet, we could find out the precise number and source, but the patterns would be hard to grok, and the takeaway would not be nearly as memorable.

On a final note, one dimensions of data missing from a lot of visualizations is the personal relativity. With online interaction, you can actually chart where the user stands in relation to the graph you are displaying. Not unlike a “you are here” dot on a map, if you simply enter where you live and your income bracket, you could be personally plotted onto the infant mortality vs. GPD per capita charts.

We hope that keeping these tips in mind will help you understand and create useful data visualizations. Over the next few months we’ll have more examples from SVT’s collaborations with Nonprofitmapping.org, the guys behind Human Translation, Cisco, and we’d love for you to share your examples here.

• What’s the best example of data visualization you know of?
• What’s the worst example of data visualization manipulation you’ve seen?
• What’s your favorite, cost-effective tool to create data visualizations?

Sara Olsen is Founding Partner of SVT Group, an impact advisory and information systems company. Robert Bailey, a 15-year design veteran, recently left Google to co-found agoodproject.com, which will debut later this year.

We have many dashboards available to us today in many contexts: finance, forecasts, consumption, demographics, etc. We witness the emergence of beautiful graphical representations more frequently than ever. Instead of flooding this post with examples of data visualization, I would like to put the spotlight on how we processes them, and what challenges that presents.

A couple of years ago, we were tasked at IDEO to design dashboard visualizations for Ford’s next generation hybrid vehicle, the Ford Fusion. Hybrid cars are efficient only if the driver maximizes the car’s potential; in other words, if the driver learns to make sense of the complex mechanics of two motors and regenerative braking. If the dashboard is the interface between the driver and the car, how might it coach drivers to make sense of this complexity and to adopt efficient driving habits? That is the obvious question, but it’s incomplete. The missing part is, how do we design without interfering with driving and safety?

This is the key question we need to address when we put people at the center of evaluation (as Jocelyn Wyatt puts it). In this context, it means recognizing that people are preoccupied with more important tasks than spending long amounts of time in front of dashboards and data visualizations. This is true in any setting, and in our case it was driving. The role of visualization should not be to demand full attention, but to support the priority task and improve it through feedback loops. The challenge is not just to display how you are doing right now, but also to figure out how you could do better. So, what does this mean for the visualization itself?

Every form of visualization should tell a story. Unfortunately there is limited attention and time to process all the stories. So the gist of the story, or its immediate impact, should be visible right away. The term I like to use for this principle is “glanceability.” What does a visualization tell us before we take time to analyze it? I invite you to look at the following chart and image for 10 seconds each and compare. What did you see? What did you feel?

Modified from Azar Askin’s reproduction of a poster by Muenster Planning Office, Germany

Displaying too many numbers upfront can be overwhelming, yet that does not mean we should avoid them altogether when visualizing data. Pushing the narrative metaphor further, the story of visualizations should unfold as one reads into them.

Al Gore highlights the dramatic increase in CO2 concentration by stepping on an elevated platform in the movie An Inconvenient Truth.

Today, we can use simple interactivity to accommodate an increased level of detail on demand. In more advanced terms, visualizations should allow for direct manipulation as Ben Shneiderman, one of the pioneers of information visualization field, put it in 1983.

Demographic data is often presented in very complex tables and boring charts. Gapminder presents historical data related to poverty in visually compelling and easily maneuverable ways, and actually confronts many of the myths we hold about the developed and developing worlds through the narrative told using the data.

You can explore more examples of data visualization in web and print publications such as Information Aesthetics, Flowing Data, Many Eyes, Wired, The New York Times, and right here at GOOD.

Some questions for further discussion:

• Do you remember a visualization that told you a story?
• What did you see at first glance?
• How did you dig deeper?

Engin Erdoğan designs interactions and leads projects at IDEO.

UPDATE: We’ve removed an image previously posted at the request of the author.

This post is a response to “How Might We Measure What’s Most Meaningful?” Read more of the conversation here.

The dashboard is the holy grail of many non-profit boards—a repository of brightly colored bubbles that provides a quick diagnostic on the health of an organization. Green dots signal “this is going really well—no need to put focus here” and red dots warn “focus an hour of your meeting here—deep well of trouble brewing.” Of course, dashboards are only as good as the goals on which they are predicated, the data on which they are based, and the intelligence of the people who read them. And more importantly, when you are driving on a stormy day (or even on a sunny one), I can’t imagine that many of us think it’s a good idea to stare mesmerized by our odometer or fuel gauge.

But, where to look for meaningful measurement and assessment in a noisy and complex world?

The Draper Richards Foundation selects, funds, and supports early-stage nonprofits with multi-year general operating grants and a range of other resources. We are agnostic with respect to the topics and geographies in which we fund, and instead focus our aggressive grant diligence on visionary entrepreneurial leaders, with distinctive and scalable models, and the promise of sustainable social impact.

We are not experts on the field of measurement or evaluation, but over the years we know that encouraging our portfolio members to focus on impact assessment leads to stronger programmatic outcomes, larger numbers of follow-on funding opportunities, and increased credibility in their fields of endeavor. As our portfolio has increased in size to more than 25 grantees, and portfolio members have scaled, grown their outcomes and impacts, and undertaken assessment in a more mature ways, there are trends that have emerged.

For example, single bellwether outputs can be very meaningful, especially when they reflect the values of the people being served. Agora Partnerships, a social enterprise dedicated to providing talented entrepreneurs in developing countries with tools, networks, and financing, counts the number of jobs created by its supported businesses as a significant output.  Through repeated interactions with people in the regions in which they work, employment has surfaced as a core value, making that number meaningful in that context. Mapendo International, a nonprofit that protects and cares for at-risk and forgotten refugees in Africa, tracks the number of refugees that they help be referred for resettlement to the United States or other locations; again, a strong stated desire of many of the people whom they serve. Similarly, VisionSpring, a social enterprise that reduces poverty and generates opportunity in the developing world through the sale of affordable reading glasses, tracks the number of glasses it sells. Purchasers might not know that they need glasses in the same way that a person hungers for employment, but each sale is a catalyst that improves the lives of the buyer and the seller. The three organizations track lots of things, but in the end, these numbers tell the story of their success.

The obvious point to be made here is that user-centric data is authentic and credible. It is intuitive and honors the perspectives of those receiving intervention head on. Another obvious point? In resource-constrained environments, there is no excuse not to measure—and great opportunities to measure incisively and strategically.

Another trend worth observing is that not all data that indicates change is structured, consistently numerical, or easy to weight. I think that this is especially true of data that signals the change of a system. For example, much of Agora’s work is about changing cultural perceptions of entrepreneurship in Latin America. Greater numbers of regional university course offerings on small business topics, increased press coverage of entrepreneurship in Latin America, and changes in banking policy that support small enterprise are all signals of system change not easily captured in a stock set of numbers, but worth monitoring and celebrating. For Mapendo International, Rose Mapendo, a Congolese refugee with a moving personal story of resettlement, received the 2009 UN Humanitarian of the Year Award and spoke at the White House. And recently, Mapendo has started to get calls from peer NGOs asking how they might get more involved with international refugee resettlement programs, signaling a new understanding of the needs of refugees and the importance of finding solutions to those needs in more collaborative, timely, and humane ways. Being aware that a network of players is looking at their system and considering changing the rules is important to observe, both to hasten a tipping point and bring to bear the wisdom, leverage, and options that a new equilibrium might require. And finally, for VisionSpring, watching a business and distribution landscape—populated with manufacturers, sellers, and buyers of products previously unavailable—unfold over time, signals a new economy and opportunity for many emerging consumers. It maybe be hard to attach numbers to, but nonetheless, it’s still meaningful.

So, what does this mean for the possibly-old-fashioned, yet reliable and so-very-tempting, dashboard? I’ll finish by describing a dashboard that I think works—because it is rich enough in detail to tell a story and skinny enough to hold in your head. One Acre Fund, a nonprofit that helps East African farmers to emerge from persistent poverty and hunger by increasing their farm-based incomes, has a dashboard focused on three concepts—scale, impact, and sustainability—across two focus areas—program and finances. A classic 2-by-3 matrix. Each segment of this impact grid contains a few metrics. On the program front, scale details current footprint as well as capacity for future growth, impact indicates changes in the lives of the farmers served, and sustainability measures cost and repayment figures required for the program to be cost-neutral over time. The financial dashboard has a similar construct, though with a focus on access to various types of capital. The dashboard uses red, yellow, and green dots, based on trajectory toward annual goals, but also contains textual content to add to and generate dialog (some pace-setting indicators, some system-change indicators, and a frame to provide meaning and opportunities for generative discussion).

Would I stare at it while careening down the highway? Perhaps not. But has this set of measurements signaled to our team what is meaningful for our work to be successful for the people we serve and the people we can serve in the future?  Absolutely.

Thanks for getting this far down the page. If you are still with me, I will leave you with a few questions to ponder.

• When measuring impact, who is the audience and what do they consider meaningful?
• Can meaning be derived in a number, a set of numbers, a narrative?  What is just enough and what is too much when seeking a set of indicators and measurements?
• What tools enable meaningful data to be shared, understood and compared with ease and limited cultural friction?

Anne Marie Burgoyne is the portfolio director of the Draper Richards Foundation.  The organizations described above are all DRF portfolio organizations.