by: Miriam J. Landsman and Brad Richardson
(from The Prevention Report, 1998 #2)
NRC Research Division is finishing up a busy year of working with state,
local, and private-non profit agencies on a variety of outcome-related
projects - from working with state agencies to develop valid and reliable
statewide evaluations and data collection systems, to helping communities
to synthesize community level outcomes with those of individual programs,
to working with multi-service agencies to develop outcomes for their varied
demand for assistance in developing outcome based systems is coming from
all levels - from individual programs seeking to validate their work through
state agencies looking for ways to extract measurable results from often
unwieldy management information systems. While there is no "one
way" to develop an outcome based system to satisfy all needs, and each
situation requires an approach that fits with the desired goals, the community
of stakeholders, and the nature of the program(s), there are several principles
that guide the process:
#1: No matter what system is taking the lead role in developing
an outcome based system (state, local, etc.), the involvement of stakeholders
in developing outcomes and measures of those outcomes is critical to developing
a valid outcome system. An outcome system developed at the state
level must be accountable not only to federal funding sources but also
to the local level, actively involving localities in the process.
Therefore, whatever system is leading the outcome based project, stakeholders
must be prepared the length of time and extent of commitment of effort
that the project will require.
#2: The desire for uniform outcomes must be considered in
light of the nature of the service system, the diversity of the population,
and a variety of community factors. While uniform outcomes are the
easiest to measure and report, several important questions must be raised:
1) are the same outcomes relevant to all locales within a region or to
all regions within a state?; 2) are outcome measures developed in ways
that are appropriate to culturally diverse populations?; and 3) can stakeholders
reach consensus on how desired outcomes can best be measured?
#3: As a rule, the greater the heterogeneity, the more complex
the task of developing an outcome based system. A statewide evaluation
of a relatively homogeneous program with clear outcomes will be less complicated
to implement than an evaluation of a community based collaborative of
assorted programs, each serving different target populations, providing
different types of services, and having varied (perhaps even intangible)
#4: Key to developing an outcome based service system is
maintaining a focus not exclusively on the measurable outcomes, but on
the linkages between outcomes and services. In the current fervor
for developing measurable outcomes, too often the relationship between
outcomes and programs and services is neglected. Outcomes are intended
to represent changes which occur as a result of interventions. Tracking
indicators in the absence of a service context, therefore, provides little
useful information about the role of programs in attaining those changes.
#5: The logical sequence of developing an outcome based
system begins with a thorough understanding of the needs which prompt
a service or system of services and the goals developed to address those
needs. If a community is experiencing a high rate of adolescent
pregnancy, an important goal will be to reduce the rate of adolescent
pregnancy. Outcomes, then, are the measurable results by which attainment
of the goal will be evaluated. Outcomes do not develop out of thin
air, or apart from a context of needs.
stated a few important principles which guide the development of an outcome
based system, we wish to offer some examples from the field. In
this article, we describe an approach to synthesizing community and program
level outcomes, based on work with one of the decategorization projects
in the state of Iowa. In the next issue, we will present an approach
to developing statewide outcomes and a computerized data collection system
to evaluate progress toward family self-sufficiency.
Community and Program Level Outcomes: A Case Example
NRC Research Division has recently completed a child welfare outcomes
project with the Johnson County Decategorization Project. Decategorization
is an initiative which permits a county or a cluster of counties to develop
a plan to blend broadly categorized as self-sufficiency, safety, health,
and stability - which led to decisions to fund certain types of programs.
However, it was difficult to tell the extent to which these programs were
achieving outcomes consistent with the specified community level outcomes.
The individual programs varied considerably in the extent to which they
had developed measurable outcomes, as well as the extent to which they
were reporting on those outcomes.
advisory committee for the outcomes project was formed early on and, in
collaboration with the NRC research consultants, a strategy was devised
to implement the outcomes project. The plan was to first work with each
individual program to articulate measurable outcomes, and second, to work
from the community outcome plan and assess how the individual programs
fit within the larger framework of community level outcomes.
individual program work involved consultation to each program, beginning
by holding meetings/discussions with program staff to understand the nature
of the program, the target population, the services provided, and the
desired results of each program. For each program, the research
consultants prepared an evaluation framework, essentially a simplified
logic model which specified: 1) the program's activities and interventions;
2) available measures of those interventions, whether those were units
of service indicated in case logs, attendance sheets at program events,
etc.; 3) the desired results of the program; and 4) how the desired results
were measured currently or could be measured in the future.
through this process served as a training technique to help program staff
differentiate between processes and outcomes and to see how these two
were related. However, it was not always a simple linear process - in
some cases we went through a number of modifications to the framework,
based on further discussions, advice from the outcomes project advisory
committee, and decisions by individual programs to substitute different
outcome measures. In other cases we worked with program staff to
develop measurement instruments where none previously existed, or assisted
in modifying instruments that were then in use but posed problems.
In the long run, developing a simple evaluation framework proved to be
a useful tool for programs not only for purposes of this project, but
a tool that programs could use for such purposes as seeking additional
funding (by demonstrating a plan for evaluation) and for replicating the
process with other programs administered by their agencies.
illustrate the evaluation framework at the program level, Figure 1 presents
an example from a sample program, in this case a teen parent program.
1: Sample Evaluation Framework for a Teen Parent Program
Program activities and components
Observable events or behaviors measuring
Desired (measurable) results of the program
Specific measure of program outcomes
of pregnant teens are born healthy
low birth-weight infants
care notes documented in case records
of second pregnancies among program participants
of program participants with subsequent pregnancy
and group counseling
social isolation among pregnant and parenting teens
participants report less isolation on post-test
skill building sessions
of child abuse and neglect among program participants
develop within normal limits
reports of abuse and neglect among participating teen parents
who pass in all areas on the Denver Developmental screening
second task involved achieving a "fit" between the community level outcomes
and those of individual programs. This required reviewing the outcomes
specified within the broader categories described earlier (self-sufficiency,
safety, etc.) and matching key program outcomes to the appropriate community
level outcomes. In this way, the decategorization project could
see which of their funded programs were working toward specific community
level outcomes, and how these outcomes were being measured by each program.
Not every program level outcome has a place in the community level outcomes
framework - each program can have multiple outcomes, some of which are applicable
to the community outcomes and some of which are responsive to other priorities
and funding sources.
2 and 3 illustrate how the teen parent program example described above
fits within the community level health and safety outcomes. Readers
should note that these are excerpts from the larger document - there are
many more community level outcomes with corresponding program outcome
measures than are indicated in these figures:
2: Community Level Health Outcomes
are healthy and able to reach their full potential.
low birth weight infants born to participating teens
program participants without subsequent pregnancy
children passing all areas on the Denver Developmental screening
3: Community Level Safety Outcomes
provide care to children in a manner that keeps them safe from abuse
and meets their needs for critical care.
reports of abuse and neglect among participating teen parents.
this process, all of the prevention programs were represented in the community
level outcomes framework, and the decategorization project was able to
begin "visualizing" the outcomes for all of their programs. One observation
from the process was that there seemed to be many more programs working
toward outcomes for parent and youth self-sufficiency than on outcomes
for health, safety, and stability. This realization prompted further
discussion and reconceptualization of the self-sufficiency goals into
economic self-sufficiency (adult focused) and behavioral self-sufficiency
(youth centered). It has also contributed to the discussion about
current and future programming priorities.
point out that much of the work that has been done is at the conceptual
level: clarifying desired outcomes of the programs and fitting them within
the larger community outcomes structure. The next challenges that
this group will take on are measurement issues: 1) convening discussions
with programs working on similar outcomes to consider developing a core
set of common measures (currently some programs are using the same measures
of similar outcomes, while others are using slightly different measures),
and 2) bringing data into the outcomes framework. While this effort
remains a "work in progress" due to new programs being funded and other
programs changing focus, the work that has already been done toward conceptualizing
and synthesizing outcomes provides an ongoing structure for understanding
and evaluating a broad range of prevention oriented programs.
more information contact us at:
Miriam-Landsman@uiowa.edu; Phone (319)
Brad-Richardson@uiowa.edu; Phone (319)