The demand for assistance in developing outcome based systems is coming from all levels - from individual programs seeking to validate their work through state agencies looking for ways to extract measurable results from often unwieldy management information systems.  While there is no "one way" to develop an outcome based system to satisfy all needs, and each situation requires an approach that fits with the desired goals, the community of stakeholders, and the nature of the program(s), there are several principles that guide the process:

Principle #1:  No matter what system is taking the lead role in developing an outcome based system (state, local, etc.), the involvement of stakeholders in developing outcomes and measures of those outcomes is critical to developing a valid outcome system.  An outcome system developed at the state level must be accountable not only to federal funding sources but also to the local level, actively involving localities in the process.  Therefore, whatever system is leading the outcome based project, stakeholders must be prepared the length of time and extent of commitment of effort that the project will require.

Principle #2:  The desire for uniform outcomes must be considered in light of the nature of the service system, the diversity of the population, and a variety of community factors.  While uniform outcomes are the easiest to measure and report, several important questions must be raised: 1) are the same outcomes relevant to all locales within a region or to all regions within a state?; 2) are outcome measures developed in ways that are appropriate to culturally diverse populations?; and 3) can stakeholders reach consensus on how desired outcomes can best be measured?

Principle #3:  As a rule, the greater the heterogeneity, the more complex the task of developing an outcome based system.  A statewide evaluation of a relatively homogeneous program with clear outcomes will be less complicated to implement than an evaluation of a community based collaborative of assorted programs, each serving different target populations, providing different types of services, and having varied (perhaps even intangible) outcomes.

Principle #4:  Key to developing an outcome based service system is maintaining a focus not exclusively on the measurable outcomes, but on the linkages between outcomes and services.  In the current fervor for developing measurable outcomes, too often the relationship between outcomes and programs and services is neglected.  Outcomes are intended to represent changes which occur as a result of interventions.  Tracking indicators in the absence of a service context, therefore, provides little useful information about the role of programs in attaining those changes.

Principle #5:  The logical sequence of developing an outcome based system begins with a thorough understanding of the needs which prompt a service or system of services and the goals developed to address those needs.  If a community is experiencing a high rate of adolescent pregnancy, an important goal will be to reduce the rate of adolescent pregnancy.  Outcomes, then, are the measurable results by which attainment of the goal will be evaluated.  Outcomes do not develop out of thin air, or apart from a context of needs.

Having stated a few important principles which guide the development of an outcome based system, we wish to offer some examples from the field.  In this article, we describe an approach to synthesizing community and program level outcomes, based on work with one of the decategorization projects in the state of Iowa.  In the next issue, we will present an approach to developing statewide outcomes and a computerized data collection system to evaluate progress toward family self-sufficiency.

Synthesizing Community and Program Level Outcomes:  A Case Example

The NRC Research Division has recently completed a child welfare outcomes project with the Johnson County Decategorization Project.  Decategorization is an initiative which permits a county or a cluster of counties to develop a plan to blend broadly categorized as self-sufficiency, safety, health, and stability - which led to decisions to fund certain types of programs.  However, it was difficult to tell the extent to which these programs were achieving outcomes consistent with the specified community level outcomes.  The individual programs varied considerably in the extent to which they had developed measurable outcomes, as well as the extent to which they were reporting on those outcomes.

An advisory committee for the outcomes project was formed early on and, in collaboration with the NRC research consultants, a strategy was devised to implement the outcomes project. The plan was to first work with each individual program to articulate measurable outcomes, and second, to work from the community outcome plan and assess how the individual programs fit within the larger framework of community level outcomes.

The individual program work involved consultation to each program, beginning by holding meetings/discussions with program staff to understand the nature of the program, the target population, the services provided, and the desired results of each program.  For each program, the research consultants prepared an evaluation framework, essentially a simplified logic model which specified: 1) the program's activities and interventions; 2) available measures of those interventions, whether those were units of service indicated in case logs, attendance sheets at program events, etc.; 3) the desired results of the program; and 4) how the desired results were measured currently or could be measured in the future.

Working through this process served as a training technique to help program staff differentiate between processes and outcomes and to see how these two were related.  However, it was not always a simple linear process - in some cases we went through a number of modifications to the framework, based on further discussions, advice from the outcomes project advisory committee, and decisions by individual programs to substitute different outcome measures.  In other cases we worked with program staff to develop measurement instruments where none previously existed, or assisted in modifying instruments that were then in use but posed problems.  In the long run, developing a simple evaluation framework proved to be a useful tool for programs not only for purposes of this project, but a tool that programs could use for such purposes as seeking additional funding (by demonstrating a plan for evaluation) and for replicating the process with other programs administered by their agencies.

To illustrate the evaluation framework at the program level, Figure 1 presents an example from a sample program, in this case a teen parent program.

Figure 1: Sample Evaluation Framework for a Teen Parent Program

The second task involved achieving a "fit" between the community level outcomes and those of individual programs.  This required reviewing the outcomes specified within the broader categories described earlier (self-sufficiency, safety, etc.) and matching key program outcomes to the appropriate community level outcomes.  In this way, the decategorization project could see which of their funded programs were working toward specific community level outcomes, and how these outcomes were being measured by each program.  Not every program level outcome has a place in the community level outcomes framework - each program can have multiple outcomes, some of which are applicable to the community outcomes and some of which are responsive to other priorities and funding sources.

Figures 2 and 3 illustrate how the teen parent program example described above fits within the community level health and safety outcomes.  Readers should note that these are excerpts from the larger document - there are many more community level outcomes with corresponding program outcome measures than are indicated in these figures:

Figure 2: Community Level Health Outcomes

Figure 3: Community Level Safety Outcomes

Through this process, all of the prevention programs were represented in the community level outcomes framework, and the decategorization project was able to begin "visualizing" the outcomes for all of their programs. One observation from the process was that there seemed to be many more programs working toward outcomes for parent and youth self-sufficiency than on outcomes for health, safety, and stability.  This realization prompted further discussion and reconceptualization of the self-sufficiency goals into economic self-sufficiency (adult focused) and  behavioral self-sufficiency (youth centered).  It has also contributed to the discussion about current and future programming priorities.

We point out that much of the work that has been done is at the conceptual level: clarifying desired outcomes of the programs and fitting them within the larger community outcomes structure.  The next challenges that this group will take on are measurement issues: 1) convening discussions with programs working on similar outcomes to consider developing a core set of common measures (currently some programs are using the same measures of similar outcomes, while others are using slightly different measures), and 2) bringing data into the outcomes framework.  While this effort remains a "work in progress" due to new programs being funded and other programs changing focus, the work that has already been done toward conceptualizing and synthesizing outcomes provides an ongoing structure for understanding and evaluating a broad range of prevention oriented programs.

For more information contact us at:
Miriam-Landsman@uiowa.edu; Phone (319) 335-4934
Brad-Richardson@uiowa.edu; Phone (319) 335-4924