Evaluating a Project That Measures Effectiveness (Summer 2006 Trust Magazine article)
Justice Louis Brandeis's 1932 description of states as the laboratories of democracy— trying out innovative approaches to the country's most pressing problems—has become a common way to think about the work that the states do.
It's a useful image—not least because it suggests that policy could take 50 different forms. Leaders, whether in politics, business, advocacy or other areas, know this, and they often look across state lines to see how their neighbors are handling the same concerns.
The Pew Center on the States (PCS) was designed to enable states to benefit from the experience of other states. In December 2004, the Trusts launched PCS, which helps the Trusts and its partners examine effective policy approaches to critical issues facing states.
Based at the Trusts' Washington, D.C., office, PCS supports credible research, brings together diverse perspectives, analyzes states' practices to determine what works and what doesn't, and collaborates with other funders and organizations to advance nonpartisan, pragmatic solutions grounded in thorough research.
In developing PCS's core capacities, program staff were faced with an important question with respect to a long-standing project of the Trusts, the Government Performance Project (GPP), a research-based initiative that since 1996 has assessed how well, or how poorly, state governments manage their money, employees, infrastructure and information.
Program staff asked themselves: What role might GPP play in contributing to the goals of PCS? What are GPP's assets, and what value would they bring to PCS? What synergies, if any, exist between GPP's focus on management capacity and PCS's emphasis on effective policy solutions?
These questions prompted program staff to commission an evaluation of GPP that sought to examine the project's impact (in particular, that of its report cards) and to inform the Trusts' deliberations about whether the project should continue and how it could be usefully integrated into PCS.
Over its 10-year history, the objective of GPP has been consistent— improving government management. As part of its assessment, GPP evaluates whether and to what extent states are measuring for results—e.g., undertaking strategic planning, tracking agencies' progress toward meeting their goals, and evaluating not just outputs, but also outcomes. The findings are produced in a 50-state report card, which represents the culmination of rigorous methodology and collaboration among a team of academic experts and journalists.
GPP issued grades to the 50 states in 1999, 2001 and 2005. In 1999 and 2001 it covered fiscal management, capital management, human resources management, information technology management and managing for results. These reports were completed by a team of academic researchers at the Maxwell School of Citizenship and Public Affairs at Syracuse University, in partnership with Governing magazine.
In 2005, GPP reorganized the report card into four areas: money, people, infrastructure and information. This most recent iteration was compiled by a new team consisting of a project director and assistant based at the University of Richmond, four academic teams at other institutions and an overall academic coordinator, plus two journalists and a team of editors at Governing magazine. The results were published in Governing in February 2005 and also made available through the project Web site, www.gpponline.org.
In 2004, the Trusts, seeking a better understanding of GPP's strengths and weaknesses in informing states about their assessments, sponsored an evaluation by Lawrence Jacobs, Ph.D., the Walter F. and Joan Mondale Chair for Political Studies at the Humphrey Institute of Political Science at the University of Minnesota.
His study drew upon three data sources: internal Trusts documents and interviews with Trusts staff and GPP staff; interviews with 92 state government officials, their senior staff and experts in state government; and case studies of management reform in five states.
Summary of Findings
According to Jacobs's evaluation, GPP gathers a tremendous amount of data from the 50 states through interviews, an online survey and extensive document review; it analyzes the data; and it publishes its findings in an easily understood format.
Among its target audience of officials and experts in state government, GPP is visible, respected and connected to their work. Some 66 percent of those interviewed reported that they considered the quality of GPP's report card “very” or “somewhat” strong, and 55 percent credited GPP with motivating states to take an interest in public management.
In a small number of states, GPP appeared to affect government management by playing a supportive role for officials who were already committed to management reforms. GPP provided these reformers with credible and independent measures of their success, which they used to garner recognition and support for their efforts within state government and with the public.
A significant number of state officials said they would welcome an even larger GPP role through targeted outreach and technical assistance to willing recipients.
In considering whether GPP could play a useful role in the newly formed PCS, the evaluation identified four assets that GPP offers PCS: data collection and reporting infrastructure; widespread regard for GPP and recognition of its credibility; a role in demonstrating the importance of effective government; and a role in affirming the link between management and policy development.
Given these assets, Jacobs recommended that GPP's grading be continued and integrated into PCS and accompanied by a strengthened outreach program that would help states apply the effective practices and lessons learned identified through the project's report card.
He found that the comparative ratings in GPP's report card are valued by a core audience of PCS, and state influentials see effective management as an important ingredient in developing new policy—a critical focus of PCS.
State officials are eager to learn from the practices of other states and from the expertise that GPP has to offer; and PCS could help the project disseminate its information to broader audiences, strengthen partnerships with national associations of state officials and work more closely with individual states that want assistance in improving their practices.
Informed by the evaluation and staff's analyses, the Trusts' board in June decided to integrate the Government Performance Project into the Pew Center on the States.
The project will publish its next 50- state report card in February 2008. The areas studied by GPP are critical to successful policy implementation across a wide range of issues, so the data collected will be useful to PCS in its efforts to inform and advance effective state policy more broadly.
At the same time, GPP will expand its outreach efforts by sponsoring multi-state meetings, responding to requests for tailored information and assistance from individual states interested in improving their systems, and building a network of peer advisors and practitioners with expertise and extensive experience.
Finally, GPP's profile in tracking and assessing state performance should amplify PCS's message that it matters where you live—and enhance PCS's ability to encourage states to pursue data-driven solutions that have proven effective and efficient elsewhere.
Nichole Rowles is an officer in Planning and Evaluation at the Trusts, and Marshall Ledger is editor of Trust.