Using Data to Measure Performance of Home Visiting

The Pew Charitable Trusts

Overview

Across the country, state legislatures are turning to evidence-based policymaking to ensure that taxpayer dollars are spent efficiently and effectively. One example is family support and coaching. In response to research confirming that the early years of childhood affect learning, behavior, and health for a lifetime, many states have invested in these programs, commonly referred to as “home visiting.” Evidence shows that families that participate in home visiting programs, which focus on strengthening vulnerable families with children under age 5, are often more self-sufficient and better able to handle the challenge of parenting and to raise healthier, safer children.¹

However, for many reasons, including differences in family needs, culture, and the availability of supportive community services, past evidence of effectiveness alone does not necessarily lead to positive outcomes. Evidence must play an essential role throughout the life of the program, from legislation and planning to design and implementation. Ongoing performance monitoring is vital to understanding whether desired family and child outcomes are being realized. Several states have passed legislation to make home visiting programs more effective and accountable by requiring the agencies that oversee them to set goals and measure results.

Ongoing performance monitoring is vital to understanding whether desired family and child outcomes are being realized.

Until recently, however, performance measures for home visiting have been used primarily not to assess outcomes, but instead to track “inputs”—the number of visits, basic client demographics, and completed tasks such as screenings for depression. This type of data, while helpful for understanding client characteristics and what services are being delivered, provides little insight into program effectiveness. Not all family support and coaching models and programs are the same; they vary in focus, content, and target population, and states typically invest in more than one approach, including promising and locally developed programs, designed to address specific state priorities. This can make it difficult to determine the overall impact of a state’s investment in family support and coaching.

To help overcome this challenge, several states joined with The Pew Charitable Trusts beginning in 2013 to develop the Home Visiting Data for Performance Initiative, the first significant effort to promote common performance measures that states can adopt to determine whether their goals are being achieved across a portfolio of home visiting programs. It identified key indicators in three areas—maternal health and achievement; child health, development, and safety; and parental skills and capacity—as well as descriptive factors, which include demographic, geographic, and basic service-delivery information about participants.

These indicators focus on achievable results and build on data that states and programs are already collecting. To effectively monitor performance, they include measures of both outcomes and the processes used to reach them and are based on a recognition that measurement methodologies will become more sophisticated as state efforts mature.

The initiative also emphasizes key strategies for enhancing the quality and usefulness of states’ data, including collecting raw—as opposed to only aggregated—data at the participant level and doing so at multiple intervals, as well as comparing parent and child outcomes with those of similar families not receiving services.

This report outlines the indicators developed by the initiative as well as key descriptive factors that state and local managers can use to form a clearer picture of family and child outcomes, who is being served, and the availability of other community supports. With this information, they can, in turn, provide the most effective services for all families, to support parents in promoting their children’s healthy emotional, social, physical, and cognitive development.

Performance indicators

The recommended indicators are intended to improve decision-making in regard to programs, policies, and funding, so they needed to be easily understandable for policymakers and the engaged public. Likewise, the measurement approaches and recommendations for enhancing data quality and utility had to be as methodologically rigorous as possible, to make the data and findings credible and reliable.

The indicators reflect a mixture of process and outcome measures because, although outcome indicators that capture the results of an intervention’s effect on participants are preferable, reliable data may not always be available. In that context, process measures have value because they track important steps assumed to contribute to the participant-level results. For example, because screenings are critical to identifying and treating depressive symptoms in new mothers, ensuring that participating women are assessed is vital to reducing maternal depression.

In particular areas, programs have substantial influence over the process but exert limited control over the corresponding outcome. For example, family support and coaching programs make referrals to community mental health providers but have little power over the availability of services or the effectiveness of any treatment received. In these instances, the initiative recommended indicators that are process rather than outcome measures.

• Despite the diversity of program designs, target populations, service durations, and provider credentials, most programs’ primary objectives are to:
• Cultivate parents’ ability to form strong, positive attachments with their children and to keep them safe.
• Promote children’s healthy physical, cognitive, and social-emotional development by monitoring their progress, guiding parents in recognizing their children’s and their own needs, and accessing appropriate services.
• Improve maternal and child health.

By using the following performance indicators, states can assess whether programs are meeting these primary objectives. As noted in the criteria, the indicators focus on outcomes and activities that are under the direct and immediate control of family support and coaching programs rather than any long-term aspirational goals for participating children and families. For additional information on these indicators, see the Appendix.

Maternal health and achievement indicators

Maternal depression screening and referral

Maternal depression and stress negatively affect the entire family and strain relationships. A lack of maternal engagement and bonding early in an infant’s life can stunt the child’s long-term health and learning. Researchers have found that between 28 percent and 61 percent of mothers in home visiting programs have clinically elevated symptoms of depression at some point during their participation.²

Although evidence is limited that family support and coaching programs alone can significantly decrease symptoms of clinical depression, family support providers who build trusting relationships with mothers are better able to identify women struggling with depression, connect them to mental health services, reduce new parents’ sense of isolation, and improve their ability to ask for help when they feel overwhelmed.³

To maximize the potential benefits that programs can yield, it is critical that home visiting programs screen for maternal depression. States should measure whether these screenings take place and how many mothers identified as having symptoms receive referrals for services in the community.

Postpartum health care visit

Following the child’s birth, a mother receives a checkup from her health provider who examines her physical recovery from pregnancy and delivery, assesses her emotional state, and gives her health guidance for herself and her baby. This postpartum health care visit is critical to identifying any potential risks for a woman’s health and any future pregnancies.

The use of postpartum visits is below expected levels generally, but among low-income, uninsured, and Medicaid-enrolled women, less than three-quarters receive the recommended visit.⁴ For women with a previous poor pregnancy outcome, such as premature birth and low birth weight, the postpartum visit is especially important. Home visiting programs encourage postpartum visits and timely follow-up and increase awareness of the value associated with obtaining regular primary health care. Interbirth interval An interbirth interval—the length of time between births—of less than 18 months is significantly associated with increased risk of adverse maternal and infant health outcomes, including low birth weight, preterm birth, obstetrical complications, and infant mortality. Studies also suggest a positive association between longer intervals and maternal educational achievement, employment, and family self-sufficiency, particularly among teen mothers who had not completed high school when their first child was born.⁵ Teens who have a subsequent pregnancy before they are able to complete high school find it increasingly difficult to accomplish their educational goals.

Family support providers discuss with mothers the effects of birth spacing. States need to track birth intervals to determine if program efforts are having positive effects.

Maternal educational achievement

Maternal educational achievement contributes to an array of good outcomes for both the mother and her infant. Increased education is associated with higher earning potential and better odds for family self-sufficiency. Maternal educational achievement has also been shown to have positive effects on child development outcomes, such as cognitive development and future performance in school. Home visiting programs work with families to identify education, training, and employment goals, create plans for achieving those goals, and connect parents to the resources they need. States should measure whether mothers pursue high school or GED completion after enrolling in a home visiting program.

The Pew Charitable Trusts

A mother, son, and family support provider focus on early literacy skills.

Child health, development, and safety indicators

Child development screening and referral

During the first three years of life, the foundations for communication, movement, and social and emotional functioning are formed. Identification of developmental delays and referral to appropriate early intervention services are key to mitigating the long-term impacts or severity of physical or cognitive impairments. Early treatment increases the effectiveness of interventions and reduces long-term costs.

Most home visiting programs conduct developmental screenings to identify children at risk for delays and refer children for additional diagnostic assessments, evaluations, and treatment when screening uncovers possible problems. Programs also reinforce the importance of timely follow-up and teach parents developmentally appropriate activities to address areas of concern. To ensure that these efforts are delivering the best results, states should measure whether screenings take place and how many children identified as needing additional intervention receive referrals.

Child development

Parents play a critical role in cultivating their children’s physical and emotional well-being through their interactions and by creating a safe and nurturing physical environment. Helping parents with these objectives is an important component of early home visiting programs. By documenting, on a regular basis, how children are developing in key domains—including literacy, executive functioning, socio-emotional security, and fine and gross motor skills—family support providers gain critical information for improving program content, and states gain confidence in the ability of these investments to improve school readiness.

This indicator needs further development.

These indicators focus on achievable results and build on data that states and programs are already collecting.

Child maltreatment

Incidents involving children under the age of 4 make up a disproportionately high percentage of child abuse and neglect reports. Infants are at the greatest risk for child maltreatment because they are completely dependent on their caregivers and more physically vulnerable. Prevention and reduction of child maltreatment through improvement in parenting skills, reduction of parenting stress, promotion of parent-child attachment, and early identification of risks are among the primary goals of home visiting.

Home visiting programs promote positive parenting behaviors, including problem solving and behavior management skills and understanding the importance of attending to children’s needs. States should monitor programs to ensure that they lead to a decrease in abuse and neglect reports for participating children.

Well-child visits

To promote healthy development, experts recommend that infants receive seven checkups during the first year of life that include physical examinations, developmental screening, immunization, and guidance for parents on healthy development. Toddlers and preschoolers should receive an additional seven well-child visits before their 6 fifth birthdays. The visits are important for early detection of health or developmental issues, and home visitors can help families connect with health care providers and follow up on recommended actions. By tracking whether participating children are up-to-date on well-child visits, states can measure the effectiveness of these supports.

Maternal smoking or tobacco use

Smoking during pregnancy increases the risk of miscarriage, low birth weight, preterm birth, and birth defects, and smoking during and after pregnancy is associated with greater risk of sudden infant death syndrome. Although many women stop smoking during pregnancy, 40 percent begin again within six months after delivery,⁶ increasing children’s exposure to secondhand smoke and their risk for ear infections and asthma attacks. Home visitors facilitate smoking cessation through education, support, and connections to treatment; measuring how many mothers quit using tobacco will allow states to gauge the efficacy of these efforts.

Parental skills and capacity indicators

Parental capacity

Parents are a child’s first and best teachers. Although myriad issues will affect children’s development, infants’ earliest interactions with their parents have profound impacts on the emotional well-being, social skills, and language capacity that are important for success in school, work, and society. As such, building parents’ capacity to meet this responsibility is a foundational goal of all home visiting. Through a variety of mechanisms, programs work to improve parents’ understanding of how children develop and how best to nurture and guide them. To ensure that programs are accomplishing this objective, it is important to track if participants are forming strong parent-child attachment, gaining skills and knowledge, and demonstrating confidence in their ability to care for their children and keep them safe.

This indicator needs further development.

Breastfeeding

Breast milk provides the optimal nutrition for infants and contains immunological and anti-inflammatory properties that guard against illness. Studies suggest that breastfed children are significantly less likely than are their bottle-fed peers to be obese; develop asthma; have autoimmune diseases, such as Type 1 diabetes; and be diagnosed with childhood cancers.⁷ Moreover, infant feeding practices appear to be associated with cognitive ability during childhood: Full-term infants who are breastfed, as opposed to bottle-fed, score three to six points higher on IQ tests.⁸ Family support providers can influence the initiation and continuation of breastfeeding by promoting, teaching, and supporting nursing; states can maximize potential benefits by tracking how many mothers start and continue breastfeeding for at least three months.

States with a concentration of programs designed to focus on birth outcomes should also consider tracking low birth weight and preterm births.

Descriptive factors

In addition to obtaining information on outcomes, states need to know who is being served. To that end, the Home Visiting Data for Performance Initiative recommends that states collect data on at least the descriptive factors outlined in Table 1. Collecting these factors will help assess whether demographic differences are associated with variations in outcomes.

In particular, it is critically important to collect the dates of visits. This information can be used to answer questions such as “How long does a family need to be in a program before it is reasonable to expect a change in one of the indicators?” Participant race, ethnicity, and language data can help states determine whether their existing portfolios of home visiting programs are meeting the needs of diverse families. Location data (e.g., residential address or ZIP code) enable states to identify geographic service gaps and account for the role that community context plays in the capacity of programs and participants to achieve their goals. Program data can also provide useful information about how implementation affects outcomes.

Data development agenda

Improving parents’ capacity to form strong, positive attachments with their children and enhancing healthy child development are key objectives of home visiting. However, considerable variation exists in how programs define and support them. Therefore, identifying reliable and valid measures that can be applicable across programs is essential to understanding the effects of home visiting.

Standardizing these measures will also help states assess the effectiveness of other childhood interventions in enhancing parental capacity and supporting healthy child development. These two objectives are central to a range of parenting education programs, early care and education, early intervention services; and family support services. Securing agreement on how best to define and monitor change in these areas will improve states’ ability to effectively and efficiently direct their investments.

Parental capacity

Experts generally identify four areas of parental capacity for measurement: parent-child attachment, the presence and growth of protective factors, specific parenting skills and knowledge, and general parental efficacy.

However, none of the measurement approaches commonly employed by programs capture this full scope in a standardized manner. Several of the tools programs use to assess parental capacity—such as the Parenting Interactions with Children: Checklist of Observations Linked to Outcomes, the Keys to Interactive Parenting Scale, and the Dyadic Assessment of Naturalistic Caregiver-child Experiences—focus on the quality of the parent-child interaction and the degree to which parents are able to nurture and support their child’s healthy development and learning. Others, such as the Healthy Families Parenting Inventory and the Protective Factors Survey, examine a range of skills parents may have (e.g., child development knowledge, emotional well-being) as well as situational challenges, such as the quality of the home environment or access to social supports, that enhance or limit their capacity to meet their child’s needs.

In addition to their different focuses, some tools are observational while others rely on participant self-reporting. Any approach adopted by the home visiting field should capture the multiple elements of parental capacity listed above, discern meaningful differences in the ability of participants to meet their parenting responsibilities at the time they enroll in home visiting services, document program impacts over time, and be mindful of placing a data burden on the parent-provider relationship.

Past evidence of effectiveness alone does not necessarily lead to positive outcomes.

Child development

In the case of child development, several tools are available to measure the physical, cognitive, and socialemotional status of children. However, the cost, time, and level of training required to administer these tools are inappropriate for home visitors, who are frequently paraprofessionals with demanding caseloads. Moreover, the screening tools typically used by home visitors in the field are sufficient only to capture a snapshot of a child’s development but are not diagnostic assessments and should not be used to measure change over time. For example, the Ages and Stages Questionnaire, which many home visiting programs use to screen for delays, is not appropriate for identifying changes in a child’s developmental trajectory or assessing program impacts.

The field needs a tool to capture change in one or more of the following developmental areas: cognitive and executive functioning, social and emotional development, language and literacy, physical and motor development, and self-help and adaptive behaviors.

Although researchers and program developers working with home visiting programs have led much of the work around determining parental capacity, those working within the broader early intervention and education fields have made the most robust progress in the area of child development assessments. Groups such as the federally sponsored Network of Infant and Toddler Researchers are engaged in active discussions about how to improve child development assessment and how to generate an appropriate tool for family support and other early education providers.

Enhancing data quality and utility

The data initiative identified the following key strategies for enhancing the quality and rigor of family support and coaching data. In some cases, these best practices exceed the minimum recommended measures captured in the indicators.

Strategies for strengthening rigor

Collect data at the participant level

Performance indicators aggregate participant-level data into summary statistics, but disaggregated participantlevel data can also provide a valuable opportunity for in-depth analysis. By collecting these data, states build a more robust database that can be used to analyze critical nuances, such as which types of participants benefit most from home visiting programs. Furthermore, states can combine participant-level data across programs to discover trends not identified by the individual programs.

Participant-level data are already typically collected at the program or site level for internal performance monitoring and quality improvement purposes, so a state requirement should not place any additional burden on family support providers.

Collect basic, raw data

Collecting details such as scores, number of incidents, and incident dates provides multiple angles from which to analyze, interpret, and report. Raw, unprocessed data offer more flexibility and opportunities for responding to a broad array of questions about a state’s family support and coaching system without creating an additional data collection burden.

States can combine participant-level data across programs to discover trends.

These data also can help assess a variety of program elements, from the timeliness of screenings to whether child maltreatment reports declined following intervention. For example, beyond recording only whether a screening was performed—a yes or no question—capturing specific scores, such as from maternal depression screenings, can provide contextual information about the broader population and its needs.

Measure at least 2 points in time for outcome indicators

Collecting data at two or more points in time allows for comparisons and provides the opportunity to show improvement and change over time. Moreover, measuring indicators at multiple times offers insight into required program dosage, or the number of visits necessary to yield a desired result. States will be able to determine relationships between the number of home visits and observed outcomes.

For many of the recommended indicators, one pre- and one post-home visiting measurement is sufficient, but other indicators should be measured more frequently. For instance, states might assess maternal smoking at enrollment and then six, 12, and 18 months later to capture changes in behavior.

Determine dosage or threshold

States should only measure outcomes for participants who received at least a minimum “dose” of family support and coaching services that is sufficient to influence the targeted outcome. The data initiative did not recommend specific service thresholds or minimum dosages for the indicators because states need to examine their own data to understand how many visits or weeks or months of services are necessary to affect outcomes. States can use participant-level data and program enrollment dates to determine the thresholds that make sense for each indicator based on state-specific information. These dosages may vary based on differences in the characteristics of home visiting participants and the array of available service models. For example, a program that focuses on prenatal and infant visits may require greater intensity and frequency than one serving 3- and 4-year-olds.

Use valid and reliable measurement tools

Two of the data initiative’s indicators—child development screening and maternal depression screening—require the use of screening and assessment instruments. These indicators will only be meaningful if the instruments used are reliable and valid. A reliable instrument measures an indicator consistently across time, individuals, and situations. A valid tool accurately captures the indicator it is intended to measure. Both reliability and validity can be assessed statistically. In the case of the child development screening indicator, the data initiative specifies employment of the Ages and Stages Questionnaire, which has strong evidence of its reliability and validity across diverse populations.

Construct a comparison group of families that did not receive home visiting

Although tracking indicators over time will produce useful information about the performance of family support and coaching systems, it does not reveal whether outcomes are directly attributable to program participation. The most scientifically rigorous way to determine causality is to conduct a randomized control trial in which eligible families are randomly grouped, with one group assigned to receive home visiting and the other not offered home visiting services. Then the outcomes of both groups are compared.

Unfortunately, randomized trials are frequently costly and impractical. To assess the impact of home visiting programs without the significant investments of dollars and time, states can use various data sources to create comparison groups, such as data from administrative databases, state or national surveys, or Medicaid.

Conclusion

Investing in high-quality family support and coaching programs, adopting performance indicators, and relying on credible data will help states accurately measure the outcomes of their home visiting programs. Using this assessment framework will ensure that such programs provide effective early intervention for vulnerable young families and achieve the strongest results.

Download the full report.