How Might We Emphasize Cost Effective Evaluation Tools?

Emphasizing cost effective evaluation tools can get us better results with less effort, enabling innovators to do more good...

Emphasizing cost effective evaluation tools can get us better results with less effort, enabling innovators to do more good with a given amount of resources. In this week's discussion, we will think broadly about the costs associated with evaluation throughout the innovation process, and suggest we rethink how we approach evaluation in order to get better results with less effort.

To understand what it means to emphasize cost effective evaluation tools, let's consider what types of costs are typically associated with evaluating the success of an innovation effort. These include (what did I miss?):
  • Costs of choosing the right methods of evaluation
  • \n
  • Costs of planning and conducting the evaluation
  • \n
  • Costs of processing the evaluation
  • \n
  • Costs of sharing and spreading the results
  • \n
  • Costs of misunderstanding consequences
  • \n
  • Costs of pilot studies and implementation
  • \n
  • Costs of not learning from our successes and failures
  • \n
  • Costs of acting (or not acting) on the evaluation results (opportunity cost of pursuing the wrong path)
  • \n

I'd like to suggest that it is the last item on this list-opportunity cost-that represents the biggest opportunity for improvement. What we can least afford is wasting everyone's time pursuing solutions that aren't working or that are less effective than other potential solutions that are on the table. This raises the question:

How do we minimize the amount of time being spent on solutions that aren't working or that should be tweaked to be more effective?

Building on this, here's an attempt to define what we mean by "cost effective" evaluation tools. Cost effective evaluation tools are (how would you define it?):
  • \nTimely: quick to deploy and minimize time wasted pursuing less effective solutions
  • \n
  • \nEfficient: use the minimum level of fidelity and rigor needed to inform decision making (sample size, refinement of prototypes, etc.)
  • \n
  • \nFocused: on the high priority / high uncertainty issues where more learning is needed to move forward (see fig. 1 below)
  • \n
  • \nShareable: results are collected, processed, and distributed in a way that tells a compelling story to the relevant stakeholders
  • \n
  • \nActionable: findings are distilled down to those which are most meaningful to quickly inform decision making to guide the ongoing innovation process
  • \n

To get this discussion started, I propose the following starter list of design principles for cost effective evaluation in innovation:

Evaluation is a mindset, not a step in the process. It should be applied throughout the innovation process to guide the work as it happens. Innovation and development are continuous processes where neat and tidy endpoints rarely exist. Evaluating the work is the work.

Begin with the end in mind. Don't start an innovation process without clear goals and priorities regarding what you are trying to accomplish and at least some ideas regarding how you will measure how well you've accomplished them. Track progress against goals and priorities.

Triage. During the innovation process, take a smart approach to evaluation, recognizing that some unresolved issues will be more important to your success than others. Try this framework for evaluation triage. Rank unresolved issues according to: low to high uncertainty, and low to high priority (see fig 1). Priority could mean importance to the success of the project or in terms of impact on decision making.

Then, take action as follows:
  • High priority and high uncertainty: be on the lookout for any opportunity to address these issues with the quickest possible evaluation that will enable you to move ahead with sufficient confidence
  • \n
  • Low priority and high uncertainty: measure these issues when it's convenient; or, wait and see
  • \n
  • High priority and low uncertainty: make an assumption about these issues and move on
  • \n
  • Low priority and low uncertainty: ignore entirely; or, make an assumption and move on
  • \n

Evaluate the work before the work even starts. Familiarize yourself with how other attempts and solutions have performed in your area and analogous situations. Don't be afraid to imitate a good idea. Consider probing stakeholders with some "sacrificial concepts" to test the waters before getting too far along.

Be vigilant. As work proceeds, seek out opportunities to clarify issues of high priority and high uncertainty with a quick measurement. Your goal should be to uncover, evaluate, and address as quickly as possible issues affecting your innovation effort where there is both high priority and high uncertainty so that the team can keep moving in the right direction.

Think fast. What could you learn in an hour by talking to one stakeholder? What could you learn in a day by talking to five?

Be efficient. Use the minimum level of fidelity and rigor that you can get away with. Think quick sketch, role-playing exercises, low-res prototypes, small-scale models, etc.

Design the measurements of success while you are designing the offering

Prototype with purpose. Don't start an evaluation process without clear goals regarding what you are trying to measure. Design your experiments accordingly, putting your time into the elements that will get you the answers you need.

These principles resonate with what I have seen in my innovation practice at IDEO. Working together with a large energy provider to promote energy conservation, we learned that to innovate effectively, one must have a way to know a good idea from a bad one. And, one wants to know which ideas are working as soon as possible so that they may adjust their efforts for the better. We built a simple tool kit incorporating easy methods for quick idea evaluation, and we created a training program to spread this toolkit across the organization. It was such a hit that it's now standard practice that all new energy conservation initiatives will apply these easy evaluation tools as they are being developed.

Thinking about how to incorporate an evaluation mindset into all stages of the innovation process, I often find myself using the idea of The Evaluation Frontier (see fig 2) to navigate the trade-off between accuracy and speed. The idea behind The Evaluation Frontier is that we can choose from a range of evaluation methods during an innovation process, and we can visualize this range on a graph of Accuracy vs. Speed. Since greater accuracy often comes at the expense of speed (in a nonlinear way), we can imagine a frontier along which different evaluation methods lie.

Recognizing the crucial role of timely and efficient evaluation to guide innovation efforts, much of our work at IDEO over the last 20+ years has involved pushing the Evaluation Frontier, particularly in the realm of higher speed / low-to-moderate accuracy evaluations, in pursuit of more effective innovation for our clients. Often this takes the form of new approaches to ethnographic research or quick ways to prototype key elements of a customer experience. These are some of the ways that we minimize the amount of time we spend on solutions that aren't working or that should be improved. You can find a discussion of some of these methods online here.

I'd love to hear stories about what approaches have worked for you.

Jim Collins leads projects at IDEO with an emphasis on environmental and social impact.

via Honor Africans / Twitter

The problem with American Sign Language (ASL) is that over 500,000 people in the U.S. use it, but the country has over 330 million people.

So for those with hearing loss, the chances of coming into contact with someone who uses the language are rare. Especially outside of the deaf community.

Keep Reading Show less

Looking back, the year 1995 seems like such an innocent time. America was in the midst of its longest streak of peace and prosperity. September 11, 2001 was six years away, and the internet didn't seem like much more than a passing fad.

Twenty-four years ago, 18 million U.S. homes had modem-equipped computers, 7 million more than the year before. Most logged in through America Online where they got their email or communicated with random strangers in chat rooms.

According to a Pew Research study that year, only 32% of those who go online say they would miss it "a lot" if no longer available.

Imagine what those poll numbers would look like if the question was asked today.

RELATED: Bill and Melinda Gates had a surprising answer when asked about a 70 percent tax on the wealthiest Americans

"Few see online activities as essential to them, and no single online feature, with the exception of E-Mail, is used with any regularity," the Pew article said. "Consumers have yet to begin purchasing goods and services online, and there is little indication that online news features are changing traditional news consumption patterns."

"Late Night" host David Letterman had Microsoft founder and, at that time the richest man in the world, on his show for an interview in '95 to discuss the "the big new thing."

During the interview Letterman chided Gates about the usefulness of the new technology, comparing it to radio and tape recorders.

Gates seems excited by the internet because it will soon allow people to listen to a baseball game on their computer. To which Letterman smugly replies, "Does radio ring a bell?" to laughter from the crowd.

But Gates presses Letterman saying that the new technology allows you to listen to the game "whenever you want," to which Letterman responds, "Do tape recorders ring a bell?"

Gates then tells Letterman he can keep up with the latest in his favorite hobbies such as cigar smoking or race cars through the internet. Letterman shuts him down saying that he reads about his interests in magazines.

RELATED: Bill Gates has five books he thinks you should read this summer.

The discussion ends with the two laughing over meeting like-minded people in "troubled loner chat room on the internet."

The clip brings to mind a 1994 segment on "The Today Show" where host Bryant Gumbel and Katie Couric have a similar discussion.

"What is internet anyway?" an exasperated Gumball asks. "What do you write to it like mail?"

"It's a computer billboard but it's nationwide and it's several universities all joined together and it's getting bigger and bigger all the time," a producer explains from off-stage.

Photo by Li-An Lim on Unsplash

The future generations will have to live on this Earth for years to come, and, not surprisingly, they're very concerned about the fate of our planet. We've seen a rise in youth activists, such as Greta Thunberg, who are raising awareness for climate change. A recent survey indicates that those efforts are working, as more and more Americans (especially young Americans) feel concerned about climate change.

A new CBS News poll found that 70% of Americans between 18 and 29 feel climate change is a crisis or a serious problem, while 58% of Americans over the age of 65 share those beliefs. Additionally, younger generations are more likely to feel like it's their personal responsibility to address climate change, as well as think that transitioning to 100% renewable energy is viable. Overall, 25% of Americans feel that climate change is a "crisis," and 35% feel it is a "serious problem." 10% of Americans said they think climate change is a minor problem, and 16% of Americans feel it is not a problem that worries them.

The poll found that concern for the environment isn't a partisan issue – or at least when it comes to younger generations. Two-thirds of Republicans under the age of 45 feel that addressing climate change is their duty, sentiments shared by only 38% of Republicans over the age of 45.

Keep Reading Show less
The Planet