NRCFCP
Home
About NRCFCP
Training
Research & Evaluation
 
Iowa Center for Evaluation & Research
DMC Resource Center



Outcomes consultation: lessons from the field (part 1)
by: Miriam J. Landsman and Brad Richardson
(from The Prevention Report, 1998 #2)

The NRC Research Division is finishing up a busy year of working with state, local, and private-non profit agencies on a variety of outcome-related projects - from working with state agencies to develop valid and reliable statewide evaluations and data collection systems, to helping communities to synthesize community level outcomes with those of individual programs, to working with multi-service agencies to develop outcomes for their varied programs.

The demand for assistance in developing outcome based systems is coming from all levels - from individual programs seeking to validate their work through state agencies looking for ways to extract measurable results from often unwieldy management information systems.  While there is no "one way" to develop an outcome based system to satisfy all needs, and each situation requires an approach that fits with the desired goals, the community of stakeholders, and the nature of the program(s), there are several principles that guide the process:

Principle #1:  No matter what system is taking the lead role in developing an outcome based system (state, local, etc.), the involvement of stakeholders in developing outcomes and measures of those outcomes is critical to developing a valid outcome system.  An outcome system developed at the state level must be accountable not only to federal funding sources but also to the local level, actively involving localities in the process.  Therefore, whatever system is leading the outcome based project, stakeholders must be prepared the length of time and extent of commitment of effort that the project will require.

Principle #2:  The desire for uniform outcomes must be considered in light of the nature of the service system, the diversity of the population, and a variety of community factors.  While uniform outcomes are the easiest to measure and report, several important questions must be raised: 1) are the same outcomes relevant to all locales within a region or to all regions within a state?; 2) are outcome measures developed in ways that are appropriate to culturally diverse populations?; and 3) can stakeholders reach consensus on how desired outcomes can best be measured?

Principle #3:  As a rule, the greater the heterogeneity, the more complex the task of developing an outcome based system.  A statewide evaluation of a relatively homogeneous program with clear outcomes will be less complicated to implement than an evaluation of a community based collaborative of assorted programs, each serving different target populations, providing different types of services, and having varied (perhaps even intangible) outcomes.

Principle #4:  Key to developing an outcome based service system is maintaining a focus not exclusively on the measurable outcomes, but on the linkages between outcomes and services.  In the current fervor for developing measurable outcomes, too often the relationship between outcomes and programs and services is neglected.  Outcomes are intended to represent changes which occur as a result of interventions.  Tracking indicators in the absence of a service context, therefore, provides little useful information about the role of programs in attaining those changes.

Principle #5:  The logical sequence of developing an outcome based system begins with a thorough understanding of the needs which prompt a service or system of services and the goals developed to address those needs.  If a community is experiencing a high rate of adolescent pregnancy, an important goal will be to reduce the rate of adolescent pregnancy.  Outcomes, then, are the measurable results by which attainment of the goal will be evaluated.  Outcomes do not develop out of thin air, or apart from a context of needs.

Having stated a few important principles which guide the development of an outcome based system, we wish to offer some examples from the field.  In this article, we describe an approach to synthesizing community and program level outcomes, based on work with one of the decategorization projects in the state of Iowa.  In the next issue, we will present an approach to developing statewide outcomes and a computerized data collection system to evaluate progress toward family self-sufficiency.

Synthesizing Community and Program Level Outcomes:  A Case Example

The NRC Research Division has recently completed a child welfare outcomes project with the Johnson County Decategorization Project.  Decategorization is an initiative which permits a county or a cluster of counties to develop a plan to blend broadly categorized as self-sufficiency, safety, health, and stability - which led to decisions to fund certain types of programs.  However, it was difficult to tell the extent to which these programs were achieving outcomes consistent with the specified community level outcomes.  The individual programs varied considerably in the extent to which they had developed measurable outcomes, as well as the extent to which they were reporting on those outcomes.

An advisory committee for the outcomes project was formed early on and, in collaboration with the NRC research consultants, a strategy was devised to implement the outcomes project. The plan was to first work with each individual program to articulate measurable outcomes, and second, to work from the community outcome plan and assess how the individual programs fit within the larger framework of community level outcomes.

The individual program work involved consultation to each program, beginning by holding meetings/discussions with program staff to understand the nature of the program, the target population, the services provided, and the desired results of each program.  For each program, the research consultants prepared an evaluation framework, essentially a simplified logic model which specified: 1) the program's activities and interventions; 2) available measures of those interventions, whether those were units of service indicated in case logs, attendance sheets at program events, etc.; 3) the desired results of the program; and 4) how the desired results were measured currently or could be measured in the future.

Working through this process served as a training technique to help program staff differentiate between processes and outcomes and to see how these two were related.  However, it was not always a simple linear process - in some cases we went through a number of modifications to the framework, based on further discussions, advice from the outcomes project advisory committee, and decisions by individual programs to substitute different outcome measures.  In other cases we worked with program staff to develop measurement instruments where none previously existed, or assisted in modifying instruments that were then in use but posed problems.  In the long run, developing a simple evaluation framework proved to be a useful tool for programs not only for purposes of this project, but a tool that programs could use for such purposes as seeking additional funding (by demonstrating a plan for evaluation) and for replicating the process with other programs administered by their agencies.

To illustrate the evaluation framework at the program level, Figure 1 presents an example from a sample program, in this case a teen parent program.

Figure 1: Sample Evaluation Framework for a Teen Parent Program

PROCESS
Program activities and components 
PROCESS MEASURES
Observable events or behaviors measuring 
program implementation
OUTCOMES
Desired (measurable) results of the program
INDICATORS
Specific measure of program outcomes
Prenatal education (individual) Staff observations

Contact records

Babies of pregnant teens are born healthy 0% low birth-weight infants
 
Preventive health services Health care notes documented in case records Prevention of second pregnancies among program participants % of program participants with subsequent pregnancy
Individual and group counseling Attendance at sessions              
Home visits Case notes Reduce social isolation among pregnant and parenting teens 90% participants report less isolation on post-test
Parent skill building sessions Staff observations

Attendance at sessions

Prevention of child abuse and neglect among program participants

Children develop within normal limits

No reports of abuse and neglect among participating teen parents

% children who pass in all areas on the Denver Developmental screening

The second task involved achieving a "fit" between the community level outcomes and those of individual programs.  This required reviewing the outcomes specified within the broader categories described earlier (self-sufficiency, safety, etc.) and matching key program outcomes to the appropriate community level outcomes.  In this way, the decategorization project could see which of their funded programs were working toward specific community level outcomes, and how these outcomes were being measured by each program.  Not every program level outcome has a place in the community level outcomes framework - each program can have multiple outcomes, some of which are applicable to the community outcomes and some of which are responsive to other priorities and funding sources.

Figures 2 and 3 illustrate how the teen parent program example described above fits within the community level health and safety outcomes.  Readers should note that these are excerpts from the larger document - there are many more community level outcomes with corresponding program outcome measures than are indicated in these figures:

Figure 2: Community Level Health Outcomes

Community level outcome Program outcome measure Program
Children are healthy and able to reach their full potential. 0% low birth weight infants born to participating teens

% program participants without subsequent pregnancy

% children passing all areas on the Denver Developmental screening

Teen Parent Program

Figure 3: Community Level Safety Outcomes

Community level outcome Program outcome measure Program
Parents provide care to children in a manner that keeps them safe from abuse and meets their needs for critical care. No reports of abuse and neglect among participating teen parents. Teen Parent Program

Through this process, all of the prevention programs were represented in the community level outcomes framework, and the decategorization project was able to begin "visualizing" the outcomes for all of their programs. One observation from the process was that there seemed to be many more programs working toward outcomes for parent and youth self-sufficiency than on outcomes for health, safety, and stability.  This realization prompted further discussion and reconceptualization of the self-sufficiency goals into economic self-sufficiency (adult focused) and  behavioral self-sufficiency (youth centered).  It has also contributed to the discussion about current and future programming priorities.

We point out that much of the work that has been done is at the conceptual level: clarifying desired outcomes of the programs and fitting them within the larger community outcomes structure.  The next challenges that this group will take on are measurement issues: 1) convening discussions with programs working on similar outcomes to consider developing a core set of common measures (currently some programs are using the same measures of similar outcomes, while others are using slightly different measures), and 2) bringing data into the outcomes framework.  While this effort remains a "work in progress" due to new programs being funded and other programs changing focus, the work that has already been done toward conceptualizing and synthesizing outcomes provides an ongoing structure for understanding and evaluating a broad range of prevention oriented programs.

For more information contact us at:
Miriam-Landsman@uiowa.edu; Phone (319) 335-4934
Brad-Richardson@uiowa.edu; Phone (319) 335-4924

 
NRCFCP - 319.335.4965

Copyright 2009, The University of Iowa. All Rights Reserved.
Contact NRCFCP


The University of Iowa