How Nongovernmental Stakeholders Can Support States in Advancing Evidence-Based Policymaking

Overview

Evidence-based policymaking has helped many states across the country ensure that their budget and policy decisions are informed by the best available data. Yet, even for those states that have made significant progress using this approach, barriers remain to institutionalizing its use. By ensuring that the creation and use of evidence becomes embedded in the way state governments make decisions, leaders and policymakers can better serve their populations effectively and equitably.

This report offers guidance on ways that nongovernment stakeholders can help evolve states’ progress by addressing persistent challenges to the routine use of evidence.

Since 2010, the Results First initiative has partnered with 27 states to help them advance an evidence-based policymaking approach to invest in programs that are proved to work. At the onset, few state policymakers knew how to find rigorous evidence, how to assess it, or how to use it when making budget and policy decisions. Since then, with the support of Results First tools and resources, states have built their capacity to examine the effectiveness of state-funded programs, conduct cost-benefit analyses, and incorporate this information into decision-making. But although Results First has helped the most committed states institutionalize and sustain their evidence-based policymaking efforts, there’s more to be done.

Thanks to the important advancements that states have made, Results First will end in March 2023, transferring its key tools and resources to other organizations that will continue to support state governments. This transition will require broader commitment from nongovernmental stakeholders—such as philanthropies, state membership groups, and research and policy organizations—to provide instrumental supports in order for states’ evidence-based policymaking efforts to continue and grow. Fortunately, many organizations now provide a wide range of assistance for states’ efforts, including conducting evaluations, facilitating peer networking among government stakeholders, supporting government capacity building, and providing technical assistance. Given the progress that Results First and its state partners have made in the past decade, the reputable organizations now supporting the field are well positioned to advance the institutionalization of this work.

To help these groups and other stakeholders, this report describes the kinds of supports state governments need as they work toward more consistent use of evidence, particularly through state budget offices. The report also highlights key areas in which nongovernmental stakeholders can boost state efforts to institutionalize evidence use. These key areas include:

• Building and supplementing state governments’ research capacity.
• Improving availability of, and state governments’ access to, relevant evidence.
• Fostering knowledge-sharing and support for evidence work.

Nongovernmental stakeholders can use this knowledge, together with consideration of local contexts, to help ensure that states’ evidence-based policymaking efforts remain a thriving and evolving endeavor that can help them better meet the needs of their communities.

How Results First and states evolved evidence use over the past decade

During the Great Recession (December 2007 to June 2009), state governments reeled under the pressure of deep revenue declines. They were forced to slash spending and often did so indiscriminately across budget areas. As they did so, they lacked evidence to know which programs were producing beneficial outcomes (e.g., reducing recidivism, increasing high school graduation rates, preventing substance use disorders) for their residents. In response, The Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation launched Results First in 2010 to provide states with an evidence-based policymaking approach that could help them make the most of their limited resources. The Results First approach empowers leaders to incorporate the best available research into their policy and budget decisions and includes inventorying state programs, using evidence to analyze their effectiveness, and assessing the costs and benefits of a subset of these programs.

Results First developed tools to facilitate this work: program inventory¹ templates and instructions to catalog state-funded programs; the Results First Clearinghouse Database² to understand program effectiveness; a customizable cost-benefit model³ to examine state programs’ return on investment; and the Evidence-Based Policymaking Resource Center⁴ to highlight guidance and examples of the approach in action.⁵ Results First also worked with 27 states to build their capacity to carry out this work. This hands-on training helped develop state leaders’ understanding, support, and use of rigorous research in budget and policy decisions. ⁶ Subsequently, these state government partners collectively shifted more than $1 billion to evidence-based programs, moving funds away from ineffective programs and investing additional resources in programs proved to work.

Besides moving money, Results First’s state partners have made significant policy changes to improve evidence use. At least seven states have developed budget or contracting guidelines that require information on the effectiveness of programs, thereby routinizing the process of consulting evidence as contracts and budget proposals are crafted. For instance, Colorado requires its budget office and state agencies to describe the research that supports changes to a program’s funding in budget requests and directs the General Assembly’s Joint Budget Committee staff to independently assess relevant evidence and consider it in their funding recommendations, when appropriate. And the New York State Division of Criminal Justice Services has incorporated guidelines on evidence-based practices, fidelity principles, and assessments of program effectiveness into its contracts. At least 10 states have taken steps to ensure the longevity of their efforts, passing laws or resolutions to mandate effectiveness or cost-benefit analyses of state-funded programs. In 2019, New Mexico approved a law requiring state agencies to indicate how much of the funding they are requesting will be spent on evidence-based programs and compelling them to prioritize these programs when possible. It also requires the legislative and executive branches to collaboratively assist agencies with implementing the law, ensuring cross-branch support.⁷ Furthermore, eight of our long-standing state partners (Alabama, Colorado, Iowa, Minnesota, Mississippi, New Mexico, New York, and North Carolina) are well positioned to serve as ambassadors and exemplars for other states to follow, having taken significant steps to ensure that their budget and programmatic planning are routinely informed by evidence.

Lastly, given its approaching end date, Results First decided to transfer four of its key assets to other organizations via a competitive request for applications process.⁸ In July 2021, Results First transferred its peer- learning community, which facilitated knowledge exchange and helped states advance their evidence-based policymaking efforts, to the National Conference of State Legislatures (NCSL),⁹ in collaboration with The Council of State Governments (CSG)¹⁰ and The Policy Lab at Brown University.¹¹ In October 2021, Results First transferred the cost-benefit model, Clearinghouse Database, and the Evidence-Based Policymaking Resource Center to Pennsylvania State University (PSU)’s Evidence-to-Impact Collaborative (EIC).¹² These organizations are drawing on their expertise to manage, expand, and improve the resources, which will ensure that states not only have continued access to them, but that these tools can continue to play an important role in shaping and furthering evidence-based policymaking. Pew also gave EIC the right to use the Results First name, so that the name and the important work associated with it continue to live on after March 2023.

Charting the path ahead

We—Results First staff—conducted three key research activities to identify opportunities for nongovernmental stakeholders¹³ to address the challenges states have in institutionalizing the use of evidence in their decision- making. (This report focused on barriers to existing state-level evidence-based policymaking efforts and did not center questions of how to modify the approach to promote more inclusive and equitable use of evidence by state governments. See Appendix 1 for more details on the methodology and Appendix 1A for limitations of our findings.)

First, we developed a list of challenges (Appendix 1B) to institutionalizing evidence use in state governments, based on more than 10 years of experience working with Results First partner states and a literature review on the topic. The list included examples of how these challenges manifest in state work.

Second, we developed a questionnaire to solicit feedback on the list. We shared the questionnaire with key government stakeholders, including elected officials, gubernatorial appointees, budget directors, and other legislative and executive staff in 10 states. (These states we reached out to have made significant progress with and are actively engaged in using evidence and are considered leaders in this field. See Appendix 1 for the criteria used to identify these states). We did not reach out to agency staff, focusing on a narrower scope of budget offices, legislators, and their staff whom Results First has primarily worked with and is knowledgeable about.

Overall, the challenges resonated with the state stakeholders, who offered additional detail on the ways they experience them.

Third, we convened a panel to discuss solutions to the identified challenges and roles for nongovernmental stakeholders to help address states’ needs. The panel included 13 experts in evidence-based policymaking from national, nonpartisan organizations that have already played a role in supporting these efforts in states, as well as from state governments. Ten of these experts were nongovernmental stakeholders from philanthropy, research and policy organizations, and state membership groups; the other three were state government stakeholders who shared their experiences with implementing and expanding evidence-based policymaking in their states. (See Appendix 1C for the members of the panel.) The rest of this report focuses on the main takeaways from this discussion and refers to these four stakeholder groups.

Nongovernmental stakeholders can partner with states to advance evidence-based policymaking

The following section outlines several priorities that nongovernmental stakeholders (including individuals from nonpartisan think tanks, philanthropies, member organizations for government officials, and higher education institutions) identified to better support state efforts to institutionalize evidence-based policymaking. The activities posited in this list are intended to spark discussion and help enable nongovernmental stakeholders to determine what they can do, in alignment with their own goals and missions. This list should not be interpreted as a comprehensive set of opportunities for nongovernmental stakeholders, nor should it prevent them from considering how to evolve the approach in other ways, such as helping states employ evidence more equitably to better address community needs. Instead, this list should be viewed as the beginning of a roadmap for organizations to consider and act upon as Results First exits the field.

Build and supplement government capacity

Many state stakeholders report having insufficient staff time and capacity to carry out evidence-based policymaking in a sustainable way. States need staff with specialized training in research, evaluation, and data analysis to carry out functions such as interpreting evidence, successfully managing and contributing to external evaluations (e.g., asking questions about the methodology, fully understanding the results), or drawing critical connections between research and policy. “Most state agencies lack the in-house expertise to select, properly implement, and evaluate evidence-based programs that address the needs of their specific service populations,” said Linda Triplett, former performance accountability director at the Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review. “The process isn’t as off the shelf as it would seem.” Centralized offices—such as those equipped with analysts that support evidence-based policymaking work across government—also report limited time and capacity.

"Most state agencies lack the in-house expertise to select, properly implement, and evaluate evidence-based programs that address the needs of their specific service populations. The process isn’t as off the shelf as it would seem.”

Linda Triplett, former performance accountability director Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review

In-state funding for evidence-based policymaking is usually limited because governments are striving to meet competing needs with finite resources. As Jon Courtney, deputy director for program evaluation at the New Mexico Legislative Finance Committee, put it, “Policymakers are often reluctant to fund administration ahead of direct services.” But the scarcity of funds is precisely why states need evidence-based policymaking, which can save money by directing investments to approaches that are proved to work.

This research identified several ways that nongovernmental stakeholders can help address these challenges.

Support government leaders’ investment in their research workforce

Nongovernmental stakeholders can support states’ efforts to build capacity by helping them determine the most appropriate approach for their needs and resources, sharing examples of different funding models, sponsoring government positions, and providing trainings for existing staff.

These stakeholders can work with states to help them identify which capacity needs—staff functions, skills, and competencies—are best served by increasing in-government capacity versus those that may be more practically addressed through partnerships with nongovernmental stakeholders. For instance, a state might ask which option best fits its needs: investing in its own robust evaluation unit, an office that coordinates with universities to conduct relevant research, or one to two staff positions that manage external evaluations? Minnesota, for example, took the first approach, creating an impact evaluation unit with full-time staff in 2019.¹⁴ And North Carolina took the second approach, creating a director of strategic partnerships position in 2018, which has since evolved into the North Carolina Office of Strategic Partnerships¹⁵ and is tasked with fostering collaborations between state government and the research and philanthropic sectors.¹⁶

During the expert panel, philanthropists also mentioned sponsoring research that highlights best practices from different funding models that states have successfully employed to sustainably build capacity for evidence use. Other states could use the examples to better understand the myriad approaches and replicate those aspects that are most relevant to their situation while avoiding pitfalls that other states have encountered.

When states need in-government expertise, philanthropists can invest in sustainable capacity by supporting the establishment of positions and offices that can last.

When states need in-government expertise, philanthropists can invest in sustainable capacity by supporting the establishment of positions and offices that can last. This includes subsidizing positions that can later be funded through the state budget, working with states to secure additional resources (e.g., from a different philanthropy), and investing in positions for longer than an administration’s term. For example, through a grant, Arnold Ventures supports an evidence adviser in North Carolina’s Office of Strategic Partnerships to increase the development and use of high-quality evidence supporting agency spending decisions.¹⁷ Although philanthropies may not be able to endow such a position in perpetuity, initial and bridge funding can help demonstrate the position’s value so that it can be a better candidate for state funding.

Nongovernmental stakeholders can also help states develop internal capacity by providing education for existing staff. Trainings should target states’ specific skills gaps and could encompass topics such as learning to judge research quality, understanding what makes something evidence-based, or gaining experience translating evidence into policy actions.

Foster deeper partnerships between state governments and nongovernmental experts

Even after investing in staff capacity for research, states frequently need additional outside expertise that can fill gaps in those internal skills. For example, states commonly cite the need for partnerships with external experts to conduct evaluations and to keep current on emerging research in specific policy areas. They also need support for effective program implementation and oversight and education for state policymakers. Although there are “never enough people or resources,” as Othni Lathram, director at the Alabama Legislative Services Agency, put it, “The access to subject area experts is key.” However, staff often don’t have time to find and assess these researchers and subject matter experts.

"[There are] never enough people or resources. The access to subject area experts is key.”

Othni Lathram, director Alabama Legislative Services Agency

Nongovernmental stakeholders can help fill this gap in several ways, including connecting states to external experts, cultivating entities that coordinate between government and researchers, growing a workforce of boundary spanners (individuals who can draw connections between research findings and potential policy actions), and following best practices in partnership.

These stakeholders can help states understand which entities, such as local universities or nonprofits, could supplement their internal capacity for evidence-based policymaking. State membership groups in particular believe they can play a role in helping states make meaningful connections with such experts; this may involve facilitating initial conversations between partners and making sure the two groups understand each other’s needs.

Nongovernmental stakeholders can also address state governments’ need for more sustainable, long-term coordination with researchers by helping develop and fund entities whose role is to connect universities with state policymakers and produce research responsive to policy needs and questions. These could be internal government offices devoted to forming such relationships or policy labs that are situated within or outside of government. For example, the Colorado Evaluation and Action Lab, which launched with philanthropic support, works with state and local governments to understand the challenges they face and then activates the research community to generate actionable and timely insights.¹⁸ The Colorado Lab began as a partnership with the governor’s office but has grown to include the legislative and judicial branches as well as local government offices.¹⁹ The Policy Lab at Brown University²⁰ and the California Policy Lab²¹ similarly bring government, universities, and others together to collaborate on research designed to inform public policy.

But building the infrastructure is not enough; states need a workforce with the skills to seamlessly work with both academics and policymakers. These individuals, described by expert panelists and the literature as “boundary spanners” or “knowledge brokers,” understand these different groups and can facilitate learning exchanges and build long-term, trusting relationships between them.²² Boundary spanners can help state agencies understand the different types of evaluations available, including what policy questions the evaluations can help answer and when it is best to use them; how to articulate clear research questions; and how to interpret and apply evaluation results. They can work with researchers on how to respond to agency requests, identify relevant research projects, and communicate results to an audience of political leaders and key policymaking staff. The evidence adviser position in North Carolina’s Office of Strategic Partnerships is a great example of a boundary spanner.

Unfortunately, boundary spanners are difficult to find because they are often embedded in academia and don’t know these types of government positions are available. Nongovernmental stakeholders can help build a pipeline of these experts by providing funding for these positions (which could be internal to government or in outside entities such as policy labs), along with training and fellowship opportunities. Doing so can offer proof that there is a pathway to work in applied research outside of the academic tenure track. Nongovernmental stakeholders can also help boundary spanners form professional networks so they can make connections, share resources, and problem solve with others.

Lastly, expert panelists agreed that partnerships between states and nongovernmental stakeholders should prioritize partnering early to establish a common language and mutually agreeable goals and ensuring that nongovernmental stakeholders understand state contexts and needs by hearing directly from them. Panelists also stressed that anyone facilitating connections between researchers and states should consider racial, ethnic, and gender equity in researchers’ access to policymakers. “There’s a real set of structural barriers in terms of which researchers have access to policymakers or to the intermediaries who are giving them access to policymakers,” said Vivian Tseng, former senior vice president, program at the William T. Grant Foundation. “You would hope that the researchers that are brought to the table early are also those who really understand the lived experiences and populations that are most marginalized and that are supposed to benefit most from policy and budgeting decisions.”

"There’s a real set of structural barriers in terms of which researchers have access to policymakers or to the intermediaries who are giving them access to policymakers. You would hope that the researchers that are brought to the table early are also those who really understand the lived experiences and populations that are most marginalized and that are supposed to benefit most from policy and budgeting decisions.”

Vivian Tseng, former senior vice president, program William T. Grant Foundation

Clarify and advocate for capacity-building opportunities leveraging federal funds

Expert panelists identified two key opportunities for nongovernmental stakeholders to expand states’ ability to use federal government resources for evidence-based policymaking: by illuminating relevant funding options and advocating for more flexible grant opportunities.

Federal funding can present an option to support states’ evidence-based policymaking efforts, but those opportunities are limited and frequently tied to specific uses and policy areas and therefore not well suited to broad, long-term capacity building. Moreover, it can be difficult for states to discern the various ways these resources can be used, and they often lack the time, capacity, or expertise to seek out or decipher federal funding opportunities.

Expert panelists saw a role for themselves in identifying and offering guidance around such funding opportunities, as well as a chance to collaborate among themselves. For example, NCSL, CSG, and The Policy Lab at Brown University worked together to publish a brief with guidance on the ways states and the District of Columbia can and have used the $195 billion in stimulus funding provided by the American Rescue Plan (ARP)²³ for evidence-based policymaking work.²⁴ Similarly, Results for America and Mathematica partnered to create a tool—the ARP Data and Evidence Dashboard—that shows “how local governments are investing their [ARP] funds, including assessing how they are using evidence and data, tracking outcomes, engaging with the public, and ensuring an equitable recovery for their residents.”²⁵ Government leaders and staff can use this tool to understand how other states and localities are maximizing the impact of their ARP funds, connect with other governments, and better use their investments to improve lives. Collaboratively produced reports and tools of this nature can streamline information about federal funding opportunities that states can leverage for capacity building.

In addition, philanthropic expert panelists see potential to advocate as a group for federal funding to be available for capacity building or general support instead of tied to specific policy areas or sets of activities. This could provide a potentially significant avenue for states to obtain resources to sustain their evidence-based policymaking. Although this long-term pursuit would require a considerable shift in the way the federal government has historically provided funding, building such a movement could help evidence-based policymaking evolve more effectively.

Improve availability of and access to relevant evidence

State governments require greater access to evidence that illuminates whether their programs are effective at improving outcomes, how effective they are, and how they can be improved to better meet community needs. Although there is a robust research base for bigger, often national, name-brand programs, homegrown programs (often created to meet a state’s unique needs and resources) are most likely to lack evaluations.

Moreover, this lack of data may disadvantage these programs when it comes to funding decisions. Filling this evaluation gap cannot typically be done only with internal capacity. Pete Bernardy, former results management director at Minnesota Management and Budget, said, “On average, 50% of the services that we’ve inventoried have qualifying evidence. The other 50% does not. We’ve started to make progress building impact evaluation capacity, but at our current pace it will take a very long time to measurably move the needle.”

"On average, 50% of the services that we’ve inventoried have qualifying evidence. The other 50% does not. We’ve started to make progress building impact evaluation capacity, but at our current pace it will take a very long time to measurably move the needle.”

Pete Bernardy, former results management director Minnesota Management and Budget

Even when evidence is available, it often doesn’t address a state’s specific needs or target populations (such as rural or predominately Spanish-speaking populations). This deficit hampers policymakers’ ability to respond to inequities between and within communities in their states. And evaluations often take too long to conduct, so key evidence is not available when policymakers need it.

This research identified two ways nongovernmental stakeholders can help grow the body of available policy- relevant research and states’ ability to use it.

Incentivize institutions and researchers to create more applied research

States need access to more evidence that speaks to policy challenges in their communities. Nongovernmental stakeholders can push for incentives to create more policy-relevant research and help scholars produce such studies within existing structures. They can also ensure that these two actions are carried out equitably, improving—rather than worsening—gaps for academics in underrepresented groups.

Relevant, timely, and actionable evidence is limited in part because there are not enough academics conducting research that is pertinent to state government policy questions. Some expert panelists believe the impediment is largely structural, noting that the academic tenure track process does not sufficiently incentivize or reward applied research. Although researchers would like to see their work used in the real world, they must meet rigid demands if they wish to advance in academic careers. “This is not a proportionate barrier that’s shared equitably,” said Max Crowley, director of the Evidence-to-Impact Collaborative at Penn State University. “Female academics and academics who identify as individuals of color experience these [barriers] at higher rates, and there’s also more risk to them. We’ve seen that through both quantitative and qualitative study.” Furthermore, even when funding is available, academics often do not know how to identify the type of research policymakers would find most valuable or how to make it more accessible to them (see the next section for more on making research more accessible to policy actors).

"This is not a proportionate barrier that’s shared equitably. Female academics and academics who identify as individuals of color experience these [barriers] at higher rates, and there’s also more risk to them. We’ve seen that through both quantitative and qualitative study."

Max Crowley, director Evidence-to-Impact Collaborative at Penn State University

While recognizing this shift in structures as a long-term and field-changing pursuit, some expert panelists see it as a movement-building opportunity. They believe actors in the research and policy spheres and philanthropists can join forces to align incentives across academic and nonacademic research communities that prioritize the creation of evidence needed to answer pressing policy questions. And some are already attempting to do so. For example, the William T. Grant Foundation, in collaboration with the Spencer Foundation, the Doris Duke Charitable Foundation, and the American Institutes for Research, funds the Institutional Challenge Grant program, which encourages academic institutions “to build sustained research-practice partnerships with public agencies or nonprofit organizations,” challenging them “to shift their policies and practices to value collaborative research … [and] to build the capacity of researchers to produce relevant work and the capacity of agency and nonprofit partners to use research.”²⁶ The grants provide funding to higher education institutions to generate more research that supports public agencies. The grants also encourage those institutions to expand academic norms and incentives—including through changes to tenure and promotion systems—in order to bolster and reward the faculty who partner with policymakers and practitioners.

Panelists also suggested ways to incentivize more applied research even within the existing academic framework. For example, higher education institutions can explicitly reward researchers with a service mindset whose work is an asset to state government, elevating the value of such research. Panelists noted that efforts to address the conflicts between tenure pressures and the production of applied research should also consider inequities in the system, including unequal burdens²⁷ felt by researchers in underrepresented groups. Although they did not discuss this in depth, other scholars have demonstrated the centrality of this issue. For example, a 2021 study on racial inequality in academia pointed out that “knowledge produced within academia is less innovative and impactful because is it largely created by—and applicable to—a narrow slice of humanity.” This study suggested methods for developing a more equitable system, including increasing transparency in hiring, promotion, and compensation practices and retaining scholars of color through tracking, funding, and mentorship.²⁸ Making academia and other centers of research production more welcoming to underrepresented scholars, such as scholars of color, will help ensure that the knowledge base generated reflects a more diverse perspective on social issues and is ultimately more impactful.

Provide support for broader interpretation and communication of evidence

Even when evidence is available, states face challenges to understanding and applying it. Research that is germane to state government policy questions is not always communicated broadly, translated into policy recommendations, or presented in ways that policymakers see as relevant. Additionally, it is not always clear what is rigorous evidence and what is not. Evaluations for similar programs in other states may not be applicable. States need credible intermediaries to help judge the quality of evaluations and interpret research findings in a way that accounts for state context and helps governments identify specific opportunities to use evidence. “Lack of contextual nuance, ‘bad’ timing, and trying to apply other states’ experiences to our state are all constraints to more and better takeup of evidence in policy and practice,” said Jenni Owen, director of the North Carolina Office of Strategic Partnerships. “I believe that better communication about the relevance of evidence would mitigate much of this. Because what appears for good reason not to be relevant to North Carolina, for example, might actually be helpful and relevant, and having tailored communication could make that clearer.”

"Lack of contextual nuance, ‘bad’ timing, and trying to apply other states’ experiences to our state are all constraints to more and better takeup of evidence in policy and practice. I believe that better communication about the relevance of evidence would mitigate much of this. Because what appears for good reason not to be relevant to North Carolina, for example, might actually be helpful and relevant, and having tailored communication could make that clearer.”

Jenni Owen, director North Carolina Office of Strategic Partnerships

As described earlier, one solution is providing funding to build an evidence workforce within government, whether that entails building those skills in existing staff or hiring to fill those needs. Nongovernmental stakeholders can also help states interpret evidence and make it actionable, particularly in states where building in-state capacity is not feasible. Expert panelists in state membership groups believed it would be helpful to have technical assistance efforts that focus on sharing best practices and examples of evidence-based policymaking and helping states contextualize that information. “A lot of times that’s where a connection might be missed,” said Sarah Needler, director of research for the Center of Innovation at the CSG.

Additionally, expert panelists suggested that funding and budgets for applied research (regardless of funder) should value interpretation, communication, and application of research findings, particularly among policy audiences. This could mean creating high-quality briefs for a broader audience on research conducted in one state or engaging with stakeholders and communities to ensure that they understand the findings of that research. Incentives for researchers to do this work may include giving them adequate time, funds, or staff support, which requires some investment but ensures that existing research—which takes plenty of time and resources to produce—goes to better use.

Expert panelists from philanthropy also pointed to public research clearinghouses as valuable intermediaries, as they review and summarize evidence in an easy-to-understand way. (However, most clearinghouses do not have the capacity to provide direct technical assistance to states; instead, they provide the information and general guidance on how to use it.) Nongovernmental stakeholders can fund these clearinghouses so they can continue to operate and add new programs to their collections. For instance, Blueprints for Healthy Youth Development examines, catalogs, and recommends programs that have strong evidence of effectiveness. Although it began with Colorado state funding, it has shifted and expanded its scope through the support of philanthropic funding from Arnold Ventures and, historically, the Annie E. Casey Foundation.²⁹ States have also developed their own tools that help to interpret and contextualize research evidence, in response to local needs. Minnesota’s Evidence Base Demographics tool can help states match a program’s demographic information (such as data on race, ethnicity, education, income, and sex) from national clearinghouses to their populations.³⁰

Foster knowledge-sharing and support for evidence work

To sustain and expand their evidence-based policymaking, states need to develop better practices and strategies. State stakeholders point to the importance of having other states’ examples to learn from and being able to discuss how these came to be with the staff who were involved. As Owen put it, “More time with people who are thinking about the same things would allow all of us to pick up opportunities from each other and replicate them or tailor them to our own context.”

"More time with people who are thinking about the same things would allow all of us to pick up opportunities from each other and replicate them or tailor them to our own context.”

Jenni Owen, director North Carolina Office of Strategic Partnerships

States also need to build a stable base of support from legislative and executive leadership. State stakeholders stressed how crucial buy-in and champions are to ensure the work gets funded and succeeds. “We’ve experienced [this] the hard way, funding new data systems and entire new units of evaluation within state agencies, [and learning] that if it’s not valued from the top, those things flounder,” said Courtney. But states also report that buy-in can be difficult to obtain, especially across branches of government or political parties. Furthermore, turnover— whether it be from legislative term limits, administrative change, or retirement—is a prominent challenge to maintaining a sufficient bench of these champions. In Mississippi, for example, Triplett noted, “Our state’s very limited pool of champions resulted in major setbacks to the effort when a champion would leave.”

This research identified two ways that nongovernmental partners can support states to foster new and better approaches and to increase buy-in from leaders.

Help states learn from one another

Peers from other states can serve as credible and effective resources, as they work in similar environments and encounter similar constraints and opportunities. However, people in one state often don’t know who to connect with or even which state would be the best resource. They also don’t have the time and resources to bring together larger groups of peers.

Nongovernmental stakeholders, particularly those in state membership groups, are well positioned to take on this role. This could mean connecting leaders or staff in a state with counterparts in another that have experience or expertise in a given area. It could also mean holding webinars with subject-matter experts during which states can ask questions, share experiences, and discuss strategies. Likewise, state membership organizations can host open discussions without fixed topics in which states can discuss whatever issues arise within contexts of changing policy demands.

Some organizations are already undertaking some of these activities. For example, the National Governors Association and Results for America have hosted What Works Bootcamps that prioritize a peer-learning approach among senior executive branch officials to learn from national leaders and one another how to increase enterprisewide, results-driven governing.³¹ Results for America regularly hosts convenings and other events with leading states (as designated in its annual Invest in What Works State Standard of Excellence) to facilitate peer exchange.

But some state stakeholders pointed to the value of transforming more engagements between nongovernmental and state entities into opportunities for knowledge-sharing between states. For example, this could mean expanding an invitation to a one-on-one meeting between a state and a nongovernmental entity (such as a funder or technical assistance provider) to other states that may benefit from being a third party to that conversation.

Nongovernmental stakeholders can also document and elevate evidence-based policymaking approaches, adding to resources such as the Results First Evidence-Based Policymaking Resource Center. These examples not only serve to help states develop their own policies, they also can help them build buy-in and support by showing that this type of work is already being done successfully elsewhere. To be most useful, these examples should be in response to states’ requests.

In addition, state stakeholders see value in having a one-stop comprehensive collection of relevant evidence- based policies, including legislation, evidence guidelines, and position descriptions. The partners at NCSL, CSG, and The Policy Lab at Brown University have already begun working on a catalog of state legislative and executive actions related to evidence use.

Cultivate and maintain evidence champions

Leaders in the executive and legislative branches who champion and support the use of evidence-based policymaking are vital to ensuring that this work thrives. Expert panelists saw opportunities for state membership groups, who have relationships with these individuals, and research and policy organizations, who have expertise in key topic areas, to help build and sustain such champions. This section highlights three ways nongovernmental stakeholders can grow leadership for evidence-based policymaking.

First, state membership groups, research and policy organizations, and other independent nongovernmental stakeholders can deliver repeated and consistent education to state leaders about the value of evidence-based policymaking and describe successful efforts undertaken in other states. As nonpartisan outsider entities, they are likely to be viewed as less contentious, politically motivated, or self-serving, and thus potentially better messengers for this information. As Robin Smart, chief legislative budget and policy analyst with the Colorado General Assembly, said, “External stakeholders can serve as a powerful force in encouraging legislators to understand how important this is and to prioritize it.”

"External stakeholders can serve as a powerful force in encouraging legislators to understand how important this is and to prioritize it.”

Robin Smart, chief legislative budget and policy analyst Colorado General Assembly

One way to provide such education is for state membership groups and research and policy organizations to collaboratively develop and deliver introductory trainings for incoming legislators and agency heads, as well as refresher courses for other lawmakers and executive leaders. Collaboration between varied nongovernmental stakeholders also ensures more cohesive support, rather than overlapping technical assistance, as well as streamlined, consensus information for state leaders from experts in the field. Nongovernmental stakeholders also can create a suite of take-home materials for participants. To make this effort more feasible, trainings could be given to multiple states at the same time and ideally would include both legislative and executive leaders.

Convening them would ensure that participants have access to the same information, build cross-branch support for evidence-based policymaking, and bring state leaders together to learn from each other.

In some instances, however, it may be more effective to provide training to an individual state. For those cases, nongovernmental stakeholders should be paired with state officials and staff who can help design the education materials and provide relevant examples and explanations of state resources and processes. Bernardy, former results management director at Minnesota Management and Budget, suggested that training be project-based to ensure that “the learning [is] pegged to real-world work.” Individual state training could enable a tailored approach that dives deeper into specific policy or program needs.

Second, nongovernmental stakeholders can partner with states to identify possible new evidence champions in both branches of government and help cultivate their interest and commitment. This would require considerable outreach by nongovernmental stakeholders, including setting up calls with leaders, inviting them to conferences or webinars, sending them tailored information, and putting them in touch with evidence champions in other states. Again, nonpartisan external stakeholders may be best positioned to do this work. Moreover, it may be inappropriate (or outside the scope of their responsibilities) for state staff to engage in some of these initial activities.

Third, nongovernmental stakeholders can keep highlighting and sharing examples to potential supporters, proving that this work has been successful in other states, and speaking more broadly about the importance of evidence.

One state stakeholder noted that 50-state scans, such as Results First’s national assessment of evidence-based policymaking³² or Results for America’s State Standard of Excellence,³³ help model behavior, providing states with a valuable tool to motivate lawmakers to use the approach. As Sophia DiCaro, executive director of Utah’s Governor’s Office of Planning and Budget, said, “External stakeholders can assist in continuing the momentum for evidence-based considerations in policymaking by remaining vocal about its importance.”

"External stakeholders can assist in continuing the momentum for evidence- based considerations in policymaking by remaining vocal about its importance.”

Sophia DiCaro, executive director Utah’s Governor’s Office of Planning and Budget

Conclusion

Over the past decade, states have increasingly built their capacity to use evidence, and more policymakers have come to rely on it when making decisions. Some states have institutionalized this work by incorporating steps to gather, analyze, and use research into their day-to-day work, but challenges persist to routinely using this approach to inform crucial decisions about government programs and services. The recommendations outlined above describe how nongovernmental stakeholders can help states institutionalize evidence-based policymaking. This information, together with consideration of states’ unique contexts, can help guide the efforts of organizations who are well positioned to lead and evolve this work after Results First exits the field. Doing so will ensure that the progress that states, Results First, and others have made around evidence use is not only sustained but continues to evolve and improve.

Appendix 1

Methodology

This study employed a three-pronged approach to data collection and analysis of challenges inhibiting states’ efforts to routinely gather and use evidence in their decision-making, potential solutions, and possible roles for nongovernmental stakeholders. To understand the nature of state challenges and related needs, we (Results First staff) conducted a literature review, a solicitation of Results First team members’ experience and expertise derived from more than a decade of working with states to advance their evidence use, and a questionnaire of state stakeholders. We also convened an expert panel discussion to understand how nongovernmental stakeholders could help support states’ efforts.

In the first phase of the research, we created a list of the five main challenges to state-level evidence-based policymaking efforts based on findings from the literature review and team solicitation. (See Appendix 1B.) In the second phase, we sent a structured, open-ended questionnaire to 28 state stakeholders (including elected officials, appointees, and government staff) in 10 states. In it, we solicited feedback on our list and sought to understand which of the challenges the stakeholders experience (including adding new experiences to the list), potential solutions for overcoming them, and opportunities for nongovernmental involvement in the solutions. We used the following criteria to determine which states would receive the questionnaire: a) membership in the Results First Peer Learning Community, a formal network of state leaders who are committed to advancing and sustaining the use of data and evidence in budget and policy decisions, and/or b) leading state status in Results for America’s 2020 Invest in What Works State Standard of Excellence. Both criteria highlighted states that have made significant progress and are actively engaged in broad evidence-based policymaking efforts. We excluded states with evidence-based policymaking efforts that are exclusively agency-based, not integrated in any meaningful way into state government, or lapsed (inactive for several years). The 10 states that met the criteria for inclusion were: Alabama, Colorado, Minnesota, Mississippi, New Mexico, North Carolina, Oregon, Tennessee, Utah, and Washington. We received 15 responses, including at least one response from each state targeted, and a near-even split between branches of government (six responses from legislative stakeholders, eight executive branch stakeholders, and one response from a government-affiliated research entity).

To select which stakeholders within these 10 states would receive the questionnaire, we used a purposive sampling approach based on a specific criterion: current or former state leaders who are working to expand and advance evidence-based policymaking, including elected officials, gubernatorial appointments, budget directors, and other legislative and executive staff. Most importantly, we included those individuals with the most familiarity and expertise on a state’s evidence-based policymaking efforts. We used NVivo software³⁴ to code questionnaire responses, analyze them thematically, and summarize them.

In the third phase, we convened a live, virtual closed-door expert panel to discuss the primary challenges, potential solutions, and roles for nongovernmental stakeholders identified in the first two phases of the research. The expert panel included 13 individuals: 10 individuals from nongovernmental stakeholders— experts representing national research and policy organizations, state government membership organizations, researchers, evaluators, and philanthropists—as well as three leaders in evidence-based policymaking from state budget offices. (See Appendix 1C.) This enabled discussion both of nongovernmental stakeholders’ potential contributions and state governments’ needs. We used a purposive sampling approach to identify and recruit an expert panel comprising individuals from the types of entities listed above, prioritizing those that have already played a role in supporting state efforts to expand their evidence use (e.g., state membership groups) and are thus familiar with the field. The convening included:

• An overview presentation of states’ needs. (See Appendix 1D.)
• A small panel discussion on the institutionalization of evidence use in state government with state stakeholders.
• Multiple breakout groups for generative discussion.
• Full group sessions on how nongovernmental stakeholders could support states, the feasibility and impact of those ideas, priorities among these, types of collaboration needed, and opportunities for state stakeholders to respond to the generated and prioritized solutions.

Results First staff recorded the sessions, transcribed them, and used NVivo software³⁵ to code the transcripts, analyze them thematically, and summarize them.

Appendix 1A: Limitations

Improving evidence-based policymaking requires considerations of equity. As David E. Kirkland wrote in 2019, “The use of research evidence is not only embedded in systems of power, it is a system of power.”³⁶ He says that using research and evidence is not a neutral act, and although it can bring tremendous value when it is used to begin conversations rather than end them, evidence must be questioned and understood within the contexts in which it is generated and used. As Martha Fedorowicz and Laudan Y. Aron wrote in 2021, “Improving evidence- based policymaking requires interrogating structural racism and oppressive systems and treating them as evidence of why a policy or program may not be effective (or may appear less effective for nonprivileged groups than they are for privileged groups).”³⁷ The primary research questions in this paper did not explicitly explore this tension, but further consideration of potential methods for addressing state challenges to evidence-based policymaking would benefit from their examination. Some researchers suggest that broadening conventional views of what constitutes evidence and taking steps to improve its production and application (such as through valuing lived experience, capturing learnings in addition to effects, and co-producing research with communities) can help make for more equitable policymaking.³⁸ As Kirkland notes, because the problem pervades the social sciences, it must be addressed from all angles: the ways research teams are composed, grants are made, agencies are funded, and so on.³⁹ For example, because funders have a great deal of influence over the communities they work in, Fedorowicz and Aron suggest that they share decision-making power with those communities and invest in research that seeks to improve evidence use (they acknowledge the William T. Grant Foundation as a rare funder that does this).⁴⁰

Results First staff have substantial experience with the implementation of evidence-based policymaking in states, as well as many connections with key stakeholders and an understanding of available data and potential challenges. Although these helped the team conduct a comprehensive study and opened channels to crucial insights from state and nongovernmental stakeholders, we still faced some limitations. Given that we gathered data for the questionnaire from a small, purposive sample, we could not ensure all types of diversity among participants. Also, because the focus of our research was on challenges and solutions to improve evidence use in centralized government offices, we did not explicitly seek to include perspectives from individuals with lived experience (e.g., current program participants or populations). However, the individuals included have intersectional identities that likely include both lived experience as well as positions of influence in government.

Additionally, we did not focus on all aspects of evidence-based policymaking but rather limited our attention to the collection, analysis, and use of evidence in budget and policy decisions. As a result, we excluded some barriers noted by state stakeholders, such as challenges in monitoring the implementation of public programs, from our discussions.

However, we sought a variety of viewpoints and triangulated our data from the literature review, questionnaire, and expert panel convening to strengthen our confidence in our findings. We used self-administered questionnaires and provided three email reminders over the course of seven weeks to enable participants to consider detailed responses when they had sufficient time. We protected individuals and their responses as much as possible by providing participants with an opportunity to review quotes attributed to them prior to publication (and making this process known to them at the time of data collection). Finally, we made the expert panel proceedings a private event, open only to the panelists, a facilitator, and three members of the Results First team.

Appendix 1B: List of challenges to state-level evidence-based policymaking efforts

• Difficulty interpreting or applying evidence (e.g., evaluations are not written for a policymaker audience and thus can be difficult for them to understand and apply; credible, independent intermediaries that can help translate and judge the quality of research are limited; there’s a lack of consistency in the field around what is considered to be evidence or evidence-based; states struggle to develop and apply their own clear, comprehensive definitions of evidence across multiple policy areas).
• Inadequate resources and staff capacity to support evidence use (e.g., insufficient funding for evaluations; staff don’t have the skills and/or time to conduct, manage, or interpret evaluations; lack of resources to train staff).
• Insufficient leadership support (e.g., inability to develop champions at the leadership level that support and advocate for evidence use; inability to sustain champions because of turnover; lack of sufficient and ongoing education for leadership on the value of evidence-based policymaking).
• Lack of agency buy-in (e.g., agencies see evidence-based policymaking as yet another trend that will be short-lived; agencies feel threatened by evidence requirements and see them as a way for leadership to cut their budgets; agencies aren’t provided sufficient incentives for using evidence and thus see it as simply another mandate; agencies aren’t given adequate training on what evidence is and how to apply it).
• Limited access to practical high-quality evidence (e.g., programs administered in the state, including homegrown ones, have not been evaluated nationally or locally; available evidence doesn’t address a state’s specific needs or target populations; evidence takes too long to produce and is not available when policymakers need it).

Appendix 1C: Expert panel participant list

The expert panel was composed of 10 nongovernmental stakeholders—experts representing national research and policy organizations, state government membership organizations, and philanthropists—as well as three leaders in evidence-based policymaking from state government, in or affiliated with state budget offices. Participants are listed in alphabetical order by first name.

Philanthropic stakeholders:

• Chris Kingsley, director of data tools and measurement at Annie E. Casey Foundation
• David Anderson, former vice president of evidence-based policy at Arnold Ventures
• Vivian Tseng, former senior vice president, program at William T. Grant Foundation

Research and policy organization stakeholders:

• David Yokum, director at The Policy Lab at Brown University
• Laura Feeney, co-executive director at J-PAL North America
• Max Crowley, director, Evidence-to-Impact Collaborative at Penn State University

State membership group stakeholders:

• Josh Inaba, senior manager, state and federal policy at Results for America⁴¹
• Kristine Goodwin, director, Center for Results-Driven Governing at National Conference of State Legislatures
• Ryan Martin, deputy director, National Governors Association Center for Best Practices
• Sarah Needler, director of research, Center of Innovation at The Council of State Governments

State stakeholders:

• Jenni Owen, director at the North Carolina Office of Strategic Partnerships
• Jon Courtney, deputy director at New Mexico Legislative Finance Committee
• Pete Bernardy, former results management director at Minnesota Management and Budget

Appendix 1D: List of state needs emerging from questionnaire of state stakeholder

After analysis of state stakeholder questionnaire responses, Results First shared the following list (along with the list of challenges in Appendix 1B) of components that states need in order to more fully embed an evidence- based policymaking approach across government and make evidence use business as usual: 

• More evaluations, especially of homegrown programs and programs replicated in their state. For many programs, rigorous evidence does not exist.
• Better access to researchers and experts (e.g., from local universities) who can help state governments gather, assess, and apply evidence.
• Better translation of evidence to state-specific contexts, including determining what evidence is relevant.
• More buy-in from leadership and staff within legislative and executive offices to support and prioritize evidence-based policymaking efforts.
• Help demonstrating the value of evidence-based policymaking to decision-makers, particularly through external entities emphasizing its importance.

  We also noted that to make progress on the areas listed above, states had two significant needs:

• Internal capacity (e.g., to conduct evaluations, foster long-term partnerships with local experts, and train staff on how to translate, assess, apply, and communicate about evidence).
• Funding (e.g., for commissioning or conducting evaluations, additional staff, or staff time and trainings from external experts).

Endnotes

1. The Pew Charitable Trusts, “How to Use the Results First Program Inventory,” April 10, 2018, https://www.pewtrusts.org/en/research-and-analysis/fact-sheets/2018/04/how-to-use-the-results-first-program-inventory. A program inventory is a complete list of programs funded by a jurisdiction, usually within a particular policy area.
2. The Pew Charitable Trusts, Results First Clearinghouse Database, accessed July 2, 2020, https://www.pewtrusts.org/en/research-and-analysis/data-visualizations/2015/results-first-clearinghouse-database.
3. The Pew Charitable Trusts, “Results First Cost-Benefit Model Aids Policymakers in Funding Decisions” (2019), https://www.pewtrusts.org/en/research-and-analysis/fact-sheets/2019/05/results-first-cost-benefit-model-aids-policymakers-in-funding-decisions.
4. The Pew Charitable Trusts, “Evidence-Based Policymaking Resource Center,” accessed May 3, 2022, https://www.pewtrusts.org/en/research-and-analysis/articles/2018/12/18/evidence-based-policymaking-resource-center. The Resource Center describes the five key components of the evidence-based policymaking framework developed by Results First in 2014. It includes briefs describing each component and articles describing promising practices and cases where states and counties are using them.
5. S. Dube and A. Sileo, “Results First to Help States Sustain Evidence-Based Policymaking,” The Pew Charitable Trusts, Aug. 19, 2020, https://www.pewtrusts.org/en/research-and-analysis/articles/2020/08/19/results-first-to-help-states-sustain-evidence-based-policymaking.
6. Results First has also worked with 10 counties.
7. New Mexico Evidence and Research Based Funding Requests S.B. 58 (2019), https://nmlegis.gov/Legislation/Legislation?Chamber=S&LegType=B&LegNo=58&year=19.
8. S. Dube, “Results First to Transfer Project Assets to Other Organizations,” The Pew Charitable Trusts, Nov. 22, 2021, https://www.pewtrusts.org/en/research-and-analysis/articles/2021/11/22/results-first-to-transfer-project-assets-to-other-organizations.
9. National Conference of State Legislatures, “NCSL,” accessed May 3, 2022, https://www.ncsl.org/. NCSL is a membership organization that serves the legislatures in all 50 states, territories, and the commonwealths of the United States through convenings, nonpartisan research, capacity-building technical assistance, and training.
10. The Council of State Governments, “Welcome to CSG,” accessed May 3, 2022, https://www.csg.org/. CSG is a nonpartisan, nonprofit organization serving all three branches of state government.
11. The Policy Lab at Brown University, “Welcome to The Policy Lab,” accessed May 3, 2022, https://thepolicylab.brown.edu/. The Policy Lab specializes in conducting randomized controlled trials and applied policy research to improve the rigor of evidence used in policymaking.
12. Evidence-to-Impact Collaborative, “Evidence-to-Impact Collaborative,” accessed May 3, 2022, https://evidence2impact.psu.edu/. The Evidence-to-Impact Collaborative is a research center that works to improve the use of research evidence through collaboration between practitioners, researchers, and policymakers.
13. Some nongovernmental entities (such as higher academic institutions that operate under state systems) may face similar limitations as state government entities.
14. S. Lize, “Minnesota Boosts State’s Capacity to Evaluate Effectiveness of Agency Programs,” The Pew Charitable Trusts, April 28, 2022, https://www.pewtrusts.org/en/research-and-analysis/articles/2022/04/28/minnesota-boosts-states-capacity-to-evaluate-effectiveness-of-agency-programs.
15. North Carolina Office of State Budget and Management, “NC Office of Strategic Partnerships,” accessed April 11, 2022, https://www.osbm.nc.gov/operational-excellence/nc-office-strategic-partnerships.
16. S. Dube, A. Doom, and S. Lize, “North Carolina Leverages State Research Capacity to Improve Government Performance,” The Pew Charitable Trusts, Feb. 18, 2021, https://www.pewtrusts.org/en/research-and-analysis/articles/2021/02/18/north-carolina-leverages-state-research-capacity-to-improve-government-performance.
17. Arnold Ventures, “North Carolina Office of State Budget and Management,” accessed April 11, 2022, https://www.arnoldventures.org/grantees/north-carolina-office-of-state-budget-and-management.
18. Colorado Evaluation and Action Lab, “Colorado Evaluation and Action Lab,” accessed May 3, 2022, https://coloradolab.org/.
19. Colorado Evaluation and Action Lab, “Colorado Evaluation and Action Lab: 3 Years in Action,” accessed May 3, 2022, https://coloradolab.org/impact-report/.
20. The Policy Lab at Brown University, “Welcome to The Policy Lab.”
21. California Policy Lab, “What We Do,” accessed May 3, 2022, https://www.capolicylab.org/.
22. K. Peterman et al., “Boundary Spanners and Thinking Partners: Adapting and Expanding the Research-Practice Partnership Literature for Public Engagement With Science (PES),” Journal of Science Communication 20, no. 7 (2021), https://jcom.sissa.it/archive/20/07/JCOM_2007_2021_N01.
23. ARP funds, which are meant to aid states in their recovery from the COVID-19 pandemic, are flexible and can be used to invest in data or technology infrastructure, conduct data analysis and evaluations, and invest in evidence-based programs.
24. The National Conference of State Legislatures, The Council of State Governments, and The Policy Lab at Brown University, “How States Can Utilize American Rescue Plan Act Funds for Evidence-Based Policymaking V2.0” (2022), https://web.csg.org/recovery/wp-content/uploads/sites/24/2022/04/ARP_Resource2_March2022_AS_v2.pdf.
25. Results for America, “Standards of Excellence,” accessed April 11, 2022, https://results4america.org/tool/index/; Results for America, “New ARP Dashboard Highlights How Cities and Counties Are Investing Federal Recovery Funds,” news release, Dec. 2, 2021, https://results4america.org/press-releases/new-arp-dashboard-highlights-cities-counties-investing-federal-recovery-funds/.
26. William T. Grant Foundation, “Institutional Challenge Grant,” accessed April 11, 2022, http://wtgrantfoundation.org/grants/institutional-challenge-grant.
27. D.M. Crowley et al., “Cultivating Researcher-Policymaker Partnerships: A Randomized Controlled Trial of a Model for Training Public Psychologists,” American Psychologist 76, no. 8 (2021): 1307-22, https://doi.org/10.1037/amp0000880.
28. C.H. Dupree and C.M. Boykin, “Racial Inequality in Academia: Systemic Origins, Modern Challenges, and Policy Recommendations,” Policy Insights From the Behavioral and Brain Sciences 8, no. 1 (2021): 11-18, https://cydneydupree.com/s/Dupree-Boykin-2021.pdf. The authors provide the suggestions described in the text to help address the devaluation of scholars of color as well as the resulting lack of visibility and increased barriers to career advancement.
29. Blueprints for Healthy Youth Development, “FAQs,” accessed May 3, 2022, https://www.blueprintsprograms.org/faq/.
30. S. Dube and R. Sen, “2 Online Tools in Minnesota Help Policymakers Assess Effectiveness of Social Policy Programs,” The Pew Charitable Trusts, Aug. 10, 2021, https://www.pewtrusts.org/en/research-and-analysis/articles/2021/08/10/2-online-tools-in-minnesota-help-policymakers-assess-effectiveness-of-social-policy-programs.
31. Results for America, “What Works Bootcamps,” accessed April 11, 2022, https://results4america.org/what-works-bootcamps/.
32. Pew-MacArthur Results First Initiative, “How States Engage in Evidence-Based Policymaking: A National Assessment” (2017), https://www.pewtrusts.org/en/research-and-analysis/reports/2017/01/how-states-engage-in-evidence-based-policymaking.
33. Results for America, “Standards of Excellence.”
34. QSR International Pty Ltd., “NVivo,” (2020), https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home.
35. Ibid.
36. D.E. Kirkland, “No Small Matters: Reimagining the Use of Research Evidence From a Racial Justice Perspective” (William T. Grant Foundation, 2019), http://wtgrantfoundation.org/digest/no-small-matters-reimagining-the-use-of-research-evidence-from-a-racial-justice-perspective.
37. M. Fedorowicz and L.Y. Aron, “Improving Evidence-Based Policymaking: A Review” (Urban Institute, 2021), https://www.urban.org/sites/default/files/publication/104159/improving-evidence-based-policymaking.pdf.
38. Ibid.
39. Kirkland, “No Small Matters.”
40. Fedorowicz and Aron, “Improving Evidence-Based Policymaking.”
41. Although Results for America is a policy nonprofit, its work more closely aligned with that of state membership groups for the purposes of the expert panel discussion.