Report

Evaluation and Program Analysis (from Pew Prospectus 2008)

Evaluation was not yet a distinct field in the 1940s, but those years did see various trends emerge that contributed to a blossoming of the discipline in subsequent decades.

Psychologist Kurt Lewin, for example, coined the term "action research" to describe an ideal approach to investigating social change, outlining a series of steps now quite familiar to contemporary evaluators: plan, act and then evaluate to inform the plan's next iteration. World War II boosted applied social research as scientists working for the military evaluated programs designed to improve soldier training and morale. When program evaluation emerged as a distinct field in the 1960s and 1970s, a few nonprofit organizations began to see it as a tool they could use to better understand the effectiveness of their work and help improve their performance.

At Pew, evaluation and evaluative thinking figure in our initiatives from design to conclusion. We view evaluative thinking as a disciplined process of gathering and assessing information to guide action that will raise the level of performance of individuals and programs. There is no better example than the Canadian wilderness protection program. This strategy was first articulated in 1999 by the director of the Pew Environment Group, Joshua Reichert, and Steven Kallick, then a senior officer in the program and now director of Pew's Boreal Conservation Campaign.

When Steve joined Pew in 1997, the evaluation unit was reviewing the Environment program's efforts to protect old-growth forests through a series of regional campaigns. Reflecting on those evaluations, Steve noted, "What jumped out at us was that the campaigns were securing tangible results, getting protection for remaining old-growth tracts in remote wilderness areas, in contrast to some of our other efforts that were harder to measure, such as forest management reform. And when we looked at all of the regional campaigns that Pew was supporting, the biggest gains by far were in British Columbia. So we drilled down into that finding."

Our staff is expected to develop programs with clear objectives and strategies, informed whenever possible by institutional knowledge and the best available evidence of what might be achieved. Past evaluations are a key source of this knowledge and evidence. In Steve's words, "Starting from the evaluations, Josh and I then launched into a strategic planning effort, which also included the evaluators and other colleagues at Pew. We looked for large tracts of intact wilderness that were feasible to protect and where there was sufficient rule of law to ensure that protections would be enforced. Canada scored high on these criteria."

Skillful planning cannot substitute for careful oversight of a strategy's implementation. Josh and Steve initially set specific milestones to track program progress in their paper outlining a strategy for protecting wilderness in Canada, but they revised these milestones as necessary in Pew's annual planning process. The ten-year goal was to protect 100 million acres of boreal forest by 2010. An evaluation confirmed that the approach had substantively contributed to the protection of 62 million acres through 2006, suggesting that it was largely on track for meeting the 2010 target. Thanks in part to changes instituted as a result of the evaluation, there were huge gains in 2007. As a result, the program was able to announce that it reached its 100-million-acre goal last year.

In the best cases, participating in the evaluation process itself can be a valuable source of learning. "During the recent evaluation of the Canadian wilderness program, we discovered that the scientific work we supported wasn't on track," Steve noted. "The evaluators kept asking us how we knew which areas were the most important ones to protect. In essence, they were asking us about the scientific basis underlying our policy recommendations. And we found we couldn't give good answers to these questions. I realized then that we had to support better science."

To promote discussion at the end of an evaluation, we share core findings and recommendations with project staff and their host organizations and with Pew's president and board. Program staff responds to this summary by raising points of consensus and disagreement with the evaluation. Pew's board then engages in a discussion with evaluation and program staff about the findings and their implications for the program going forward.

Evaluation is challenging. To be most useful, it requires a commitment by the entire organization. The common thread that links evaluators to the rest of the organization is the passion to improve performance. As the recent review of the Canadian wilderness protection program drew to a close, Steve Kallick offered these final remarks:

"Once I got past my early anxieties, I looked at the evaluation as a tool that would give us a second opinion on how we were doing. Of course we wanted a positive review, but we didn't want to hide anything from the evaluators. Doing that would have been like withholding information from my doctor but then expecting a reliable and useful diagnosis. My advice to my colleagues: Be transparent and open. Our evaluation helped us get better at what we do."

Michael J. Dahl
Managing Director, Evaluation

Lester W. Baxter
Deputy Director, Evaluation