How Might We Emphasize Cost Effective Evaluation Tools?

Emphasizing cost effective evaluation tools can get us better results with less effort, enabling innovators to do more good with a given amount of resources. In this week’s discussion, we will think broadly about the costs associated with evaluation throughout the innovation process, and suggest we rethink how we approach evaluation in order to get better results with less effort.

To understand what it means to emphasize cost effective evaluation tools, let’s consider what types of costs are typically associated with evaluating the success of an innovation effort.  These include (what did I miss?):

• Costs of choosing the right methods of evaluation
• Costs of planning and conducting the evaluation
• Costs of processing the evaluation
• Costs of sharing and spreading the results
• Costs of misunderstanding consequences
• Costs of pilot studies and implementation
• Costs of not learning from our successes and failures
• Costs of acting (or not acting) on the evaluation results (opportunity cost of pursuing the wrong path)

I’d like to suggest that it is the last item on this list—opportunity cost—that represents the biggest opportunity for improvement.  What we can least afford is wasting everyone’s time pursuing solutions that aren’t working or that are less effective than other potential solutions that are on the table. This raises the question:

How do we minimize the amount of time being spent on solutions that aren’t working or that should be tweaked to be more effective?

Building on this, here’s an attempt to define what we mean by “cost effective” evaluation tools. Cost effective evaluation tools are (how would you define it?):

• Timely: quick to deploy and minimize time wasted pursuing less effective solutions
• Efficient: use the minimum level of fidelity and rigor needed to inform decision making (sample size, refinement of prototypes, etc.)
• Focused: on the high priority / high uncertainty issues where more learning is needed to move forward (see fig. 1 below)
• Shareable: results are collected, processed, and distributed in a way that tells a compelling story to the relevant stakeholders
• Actionable: findings are distilled down to those which are most meaningful to quickly inform decision making to guide the ongoing innovation process

To get this discussion started, I propose the following starter list of design principles for cost effective evaluation in innovation:

Evaluation is a mindset, not a step in the process.  It should be applied throughout the innovation process to guide the work as it happens. Innovation and development are continuous processes where neat and tidy endpoints rarely exist. Evaluating the work is the work.

Begin with the end in mind. Don’t start an innovation process without clear goals and priorities regarding what you are trying to accomplish and at least some ideas regarding how you will measure how well you’ve accomplished them. Track progress against goals and priorities.

Triage. During the innovation process, take a smart approach to evaluation, recognizing that some unresolved issues will be more important to your success than others. Try this framework for evaluation triage. Rank unresolved issues according to: low to high uncertainty, and  low to high priority (see fig 1).  Priority could mean importance to the success of the project or in terms of impact on decision making.

Then, take action as follows:

• High priority and high uncertainty: be on the lookout for any opportunity to address these issues with the quickest possible evaluation that will enable you to move ahead with sufficient confidence
• Low priority and high uncertainty: measure these issues when it’s convenient; or, wait and see
• High priority and low uncertainty: make an assumption about these issues and move on
• Low priority and low uncertainty: ignore entirely; or, make an assumption and move on

Evaluate the work before the work even starts. Familiarize yourself with how other attempts and solutions have performed in your area and analogous situations. Don’t be afraid to imitate a good idea. Consider probing stakeholders with some “sacrificial concepts” to test the waters before getting too far along.

Be vigilant. As work proceeds, seek out opportunities to clarify issues of high priority and high uncertainty with a quick measurement. Your goal should be to uncover, evaluate, and address as quickly as possible issues affecting your innovation effort where there is both high priority and high uncertainty so that the team can keep moving in the right direction.

Think fast. What could you learn in an hour by talking to one stakeholder? What could you learn in a day by talking to five?

Be efficient. Use the minimum level of fidelity and rigor that you can get away with. Think quick sketch, role-playing exercises, low-res prototypes, small-scale models, etc.

Design the measurements of success while you are designing the offering

Prototype with purpose. Don’t start an evaluation process without clear goals regarding what you are trying to measure.  Design your experiments accordingly, putting your time into the elements that will get you the answers you need.

These principles resonate with what I have seen in my innovation practice at IDEO. Working together with a large energy provider to promote energy conservation, we learned that to innovate effectively, one must have a way to know a good idea from a bad one. And, one wants to know which ideas are working as soon as possible so that they may adjust their efforts for the better. We built a simple tool kit incorporating easy methods for quick idea evaluation, and we created a training program to spread this toolkit across the organization.  It was such a hit that it’s now standard practice that all new energy conservation initiatives will apply these easy evaluation tools as they are being developed.

Thinking about how to incorporate an evaluation mindset into all stages of the innovation process, I often find myself using the idea of The Evaluation Frontier (see fig 2) to navigate the trade-off between accuracy and speed. The idea behind The Evaluation Frontier is that we can choose from a range of evaluation methods during an innovation process, and we can visualize this range on a graph of Accuracy vs. Speed. Since greater accuracy often comes at the expense of speed (in a nonlinear way), we can imagine a frontier along which different evaluation methods lie.

Recognizing the crucial role of timely and efficient evaluation to guide innovation efforts, much of our work at IDEO over the last 20+ years has involved pushing the Evaluation Frontier, particularly in the realm of higher speed / low-to-moderate accuracy evaluations, in pursuit of more effective innovation for our clients. Often this takes the form of new approaches to ethnographic research or quick ways to prototype key elements of a customer experience. These are some of the ways that we minimize the amount of time we spend on solutions that aren’t working or that should be improved. You can find a discussion of some of these methods online here.

I’d love to hear stories about what approaches have worked for you.

Jim Collins leads projects at IDEO with an emphasis on environmental and social impact.