STUDYING COLLEGE OUTCOMES IN THE 1990s:
OVERVIEW AND ORGANIZATION OF THE RESEARCH
When we completed and published our synthesis of nearly 2,600 studies on the impact of college conducted primarily during the 1970s and 1980s (How College Affects Students, Pascarella & Terenzini, 1991), we believed that we were in a position to make a fairly large number of conclusions about the institutional and programmatic factors that facilitate student growth and development across a wide range of outcomes. At the same time, however, we also grappled with the vague feeling that we had completed our synthesis at the front edge of a major period of demographic transformation in American postsecondary education (Adelman, 1998b; Astin, 1998; Callan, 1997; Henderson, 1995; Levine & Cureton, 1998; National Center for Education Statistics, 1996b, 1998; E. O'Brien, 1993; Rhoads, 1995a; Silverman & Casazza, 1999). As a result, over time we (along with others) began to seriously question the extent to which the conclusions of research from the 1970s and 1980s would help us understand the impact of college on students as we move into the 21st century (Kuh, 1992a; Pascarella & Terenzini, 1998; Stage, 1993). Indeed, as we reviewed the research on the impact of college conducted since the publication of How College Affects Students we began to realize that the “ground rules” of American postsecondary education were being rewritten and that the research, with some natural lag time, was beginning to seriously reflect these changes.
If there is a single adjective that describes the body of research on the impact of college conducted during the decade of the 1990s it is expansive; and this expansiveness is manifest along a number of different dimensions. First, there has been an expansion of the notion of who constitutes the students worthy of study in the American postsecondary system. With some notable exceptions, the research base for How College Affects Students was strongly biased toward “traditional” white undergraduates, age 18-22, who attended four-year institutions full-time, who lived on-campus, who did not work, and who had few, if any, family responsibilities. To some extent, this bias still exists in the research base of the 1990s, perhaps reflecting characteristic student populations at those institutions that employ the majority of scholars doing research on college impact. Yet it is also true that the literature of the 1990s has shifted its focus in major ways to reflect the changing, and increasingly diverse, national undergraduate student body.
Thus, we witness an appreciably greater volume of evidence in the 1990s that attempts to account for variations in such factors as age, work responsibilities, ethnicity, sex, full- or part-time (or even interrupted) attendance, and resident versus commuter status, in estimating the impact of college. In short, as compared to the body of evidence synthesized in How College Affects Students, the literature of the 1990s appears more sensitive to a major implication of the growing diversity of American undergraduates—namely, that not all students will necessarily benefit to the same extent, or perhaps even in the same direction, from the postsecondary experience. When different kinds of students benefit differently from the same experience, social scientists typically refer to this difference as a conditional (or interaction) effect. (This effect is often contrasted with a general effect in which all students appear to benefit about the same from a given experience.) We found an increased, if not universal, concern for the estimation of conditional effects in the literature of the 1990s which we believe represents an expanded vision of the impact of college that is consistent with the growing diversity of American undergraduate students.
Second, the literature of the 1990s evidenced an expanded notion of the kinds of postsecondary institutions worthy of study. This phenomenon is certainly reflected in a continuing and more sophisticated body of evidence on women’s colleges and historically Black institutions, but it is perhaps most pronounced in the case of two-year, community colleges. The dramatic increase in student diversity in American postsecondary education has been paralleled by the equally dramatic growth in the importance of two-year colleges. For example, between 1978 and 1991, enrollments in community colleges increased by 31% (versus 23% for four-year institutions); and two-year college enrollments are expected to increase another 11% by 2003 (Chronicle of Higher Education: Almanac Issue, 1995). In 1996, community colleges constituted about 28% of all U.S. colleges and universities and about 39% of all public institutions; and, these colleges enrolled about 37% of all U.S. undergraduates and about 50% of all undergraduates in public institutions (Callan, 1997; Terenzini, 1996).
Despite the fact that community colleges are clearly major players in the national system of postsecondary education, they were largely ignored, with a few notable exceptions, in the literature base we synthesized for How College Affects Students. A liberal estimate is that only about 5-10% of the literature we reviewed focused on community college students. Fortunately, that trend changed substantially in the literature we reviewed for this book. Though community colleges are still significantly under-represented in the total body of evidence on college impacts, there is a growing body of evidence, conducted largely in the 1990s, that greatly contributes to our understanding of how these important educational institutions influence students (Pascarella, 1999).
Third, there is an expanded vision of how students learn that is clearly evident in the literature base of the 1990s. Scholars no longer regard learning solely as an act of acquiring or absorbing a set of objectively verifiable facts and concepts and, subsequently, incorporating them into long-term memory. Instead, they argue that a substantial amount of knowledge is actively constructed by the learner. There are a number of excellent, and detailed, discussions of the psychological and philosophical foundations for such constructivist approaches to student learning (Baxter Magolda, 1993; Baxter Magolda & Buckley, 1997; Cross, 1999; Dykstra, 1996; Fassinger, 1995, 1996; Felder, 1995; Mayer, 1999; Stage, Muller, Kinzie, & Simmons, 1998; Twomey Fosnet, 1996). What appears central to all of these discussions, however, is that the learner does not just passively receive knowledge or “truth” from others (i.e., faculty members). Rather, students work actively and collaboratively with faculty members and student peers to create their own knowledge by trying to make personal sense out of the material that is presented to them (Brooks & Brooks, 1993).
Accompanying this expanded view of how students learn and develop intellectually has been an expanded use of innovative instructional approaches in postsecondary education, many of which incorporate the underlying assumptions of constructivist learning in their pedagogy. Research on instructional approaches such as collaborative and cooperative learning, learning communities, freshman interest groups, supplemental instruction, problem-based learning, and service learning was largely absent, or in its embryonic form, in the literature we reviewed for How College Affects Students. Findings pertaining to the effectiveness of these instructional innovations, however, are quite evident in the literature we reviewed for this book.
Fourth, the research on college impact from the decade of the 1990s reflects an expanded set of policy concerns that were not particularly evident in the literature we reviewed for How College Affects Students. In many ways, these policy concerns reflect societal changes and technological advances that have come to fruition in the last 10-15 years. Whatever their cause, however, they are playing a major role in reshaping the policy research environment of American postsecondary education. For example, the increasing racial diversity of American society in general has been paralleled by a line of research that focuses not only on how college changes students’ attitudes toward, or openness to, diversity, but also on how experiencing diversity during college influences the outcomes of college itself. Similarly, recent national concern with the rapidly increasing costs of attending college ("College affordability concerns", 1996; McPherson & Shapiro, 1991; Morganthan & Nayar, 1996); has been reflected in a line of inquiry that focuses on the costs as well as the benefits of not only attending college but also of attending colleges that differ in their tuition structure.
In addition to societal changes and economic concerns, technological advances have assumed a major role in creating new policy issues that have shaped the research agenda of the past decade. None of these advances has been more significant than the availability of a rapidly expanding and increasingly powerful array of computer and information technologies. It would appear that, while there is not uniform adoption across institutional types (Gladieux & Swail, 1999), postsecondary institutions in general have embraced these new technologies with great fervor. For example, a recent report issued by The Institute for Higher Education Policy (1999) indicated the extent to which computer use in postsecondary education has increased during the second half of the last decade. Based on data from the 1998 National Survey of Information Technology in Higher Education, the report by the Institute indicated that in 1994 about 8% of postsecondary classes used electronic mail (e-mail). This proportion increased to 44% by 1998. Similarly, the percentage of classes reporting that they used internet resources increased from 15% in 1996 to 30% in 1998. By 1998, it was estimated that 46% of all institutions had a mandatory student information technology (computer use) fee.
When used appropriately the power of these technologies for enhancing student learning and cognitive growth in college is, indeed, substantial (Ehrmann, 1995; Green, 1996; Kuh & Vesper, 1999; Upcraft, Terenzini, & Kruger, 1999; T. West, 1996). This potential has not been lost on scholars concerned with the impact of college. While there was some research focusing on the cognitive impacts of computers in the literature we reviewed for How College Affects Students, that line of research has grown substantially and is significantly more evident in the body of literature we reviewed for this book. Moreover, the growing sophistication of information technologies such as teleconferencing, computer conferencing, the World Wide Web, gophers, file-transfer protocols, listservs, and bulletin boards has enhanced the potential for off-campus or remote site instruction. The relative impacts of such “distance education” on learning is another line of research that is markedly more evident in the literature we reviewed for this book than it was in the literature we reviewed for How College Affects Students.
Fifth, and finally, the literature we reviewed for this book evidences a more expanded repertoire of research approaches for estimating the impact of college than did the literature we synthesized for How College Affects Students. To be sure the positivist, quantitative paradigm still dominates the total body of research we reviewed, with true experiments, quasi-experiments, and correlational designs with statistical controls for salient confounding variables being the methodological tools of choice. Most of what we learned in our present synthesis of evidence is based on such approaches. At the same time, however, we also noted an increased use of naturalistic, qualitative methodologies in the total body of literature. Our prediction toward the conclusion of How College Affects Students “that in the next decade important contributions to our understanding of college impact will be yielded by naturalistic investigations” (Pascarella & Terenzini, 1991, p. 634) has certainly been born out, although this was probably more a case of reading the writing on the wall than of any great prescience on our part. In addition to the increased importance of naturalistic, qualitative studies, the current body of evidence also reflects an expanded repertoire of sophisticated statistical approaches which have helped refine and extend our understanding of college impact. These include hierarchical linear modeling, structural equation modeling, correction for selection procedures, and the estimation of effect sizes.
The warrant for this book stems largely from the confluence of the trends and changes noted above: (1) a rapid increase in student diversity and an accompanying expansion of the notion of who constitutes the students who should be studied in American postsecondary education; (2) an expanded interest in the impacts of postsecondary institutions that were largely ignored in previous research, particularly community colleges; (3) an expanded and evolving vision of how students learn that has been accompanied by new bodies of evidence on teaching, instruction, and learning; (4) an expanded set of policy issues, such as college costs and the impacts of student diversity and new information technologies, that have spawned new lines of research; and (5) an expanded repertoire of methodological approaches for estimating and understanding the impact of college on students. These five influences have had a major role in shaping a body of literature between 1989 and 2000 that is substantially more diverse and complex, and nearly as extensive in sheer number of studies, as the 20+ years of research (i.e., 1967-1989) we synthesized for How College Affects Students.
This book is an attempt to comprehensively synthesize the new research evidence on the impact of college on students that has accumulated since the 1991 publication of How College Affects Students. It covers primarily the body of research produced from about 1989 through the end of 1999, although at times we cite studies earlier than 1989, either because they are important to an understanding of the total body of evidence or because we somehow overlooked them in our previous synthesis. Since the body of research is an ever-expanding entity, we have also tried, when possible, to include important studies produced during the 2000-2002 period. To be sure, there have been important and well-conducted research syntheses on the impact of college on students since the publication of How College Affects Students (Gardiner, 1994, 1998; Kuh, Vesper, & Krehbiel, 1994; Love & Goodsell Love, 1995; Stage et al., 1998). However, these syntheses have been focused on relatively specific dimensions of impact, such as teaching and learning or the developmental impact of students’ out-of-class involvement. None of them is, nor do they make any claim to be, an attempt at a comprehensive review of the research on college impact.
Much of the heavy lifting in terms of developing a conceptual framework, or set of organizing principles, for this synthesis was done when we wrote How College Affects Students. We employ the same general conceptual framework in this book that we did in its predecessor. There are two reasons for this decision. First, while any conceptual framework for organizing such a vast and complex body of evidence clearly has weaknesses, the one we used in How College Affects Students seemed to work reasonably well (Baird, 1992). Second, using the same conceptual framework gives us a measure of continuity and consistency with our previous synthesis. We reasoned that a consistent conceptual framework across the two volumes would facilitate comparison of the dimensions along which the literature has advanced and where new findings have changed or reinforced previous conclusions. This conceptual continuity between volumes not only helped us organize and conduct the present synthesis of evidence, but we think that it also will help the reader see clearer connections between the conclusions of this synthesis and the conclusions of How College Affects Students.
In this synthesis, as in How College Affects Students, we organize the evidence in terms of different types of outcomes (e.g., cognitive development, values and attitudes, career) rather than in terms of the potential sources of influence on college outcomes (e.g., major field, place of residence, interactions with peers). Astin’s (1973) taxonomy of outcomes was particularly influential in defining the content and scope of our synthesis in How College Affects Students and it continues its strong influence in this volume. Astin reasoned that college outcomes can be organized along three dimensions: type of outcome, type of data, and time span. The first two dimensions can be thought of as a 2 x 2 matrix where type of outcome tends to be either cognitive or affective and type of data tends to be either psychological or behavioral. It makes less sense to consider the temporal dimension as a dichotomy. Rather, it can be considered a continuous variable tapping the time span over which outcomes are assessed (e.g., during college, ten years after graduation from college). Under the first dimension, cognitive outcomes have to do with the utilization of higher-order intellectual processes such as knowledge acquisition, decision making, application, and reasoning. Affective outcomes are attitudes, values, self-concepts, aspirations, and personality dispositions. The second dimension of Astin’s taxonomy refers to the operations required to assess the cognitive or affective outcomes under consideration. Psychological data reflect the internal states or traits of the individual and are typically assessed indirectly by means of a test or examination. Thus, an individual’s level of skill in evaluating arguments or critical thinking is typically inferred from responses to a set of questions. Behavior measures on the other hand, are based on direct observation of, or reporting by, the individual. Consequently, there is much less to infer.
Astin’s (1973) 2 x 2 taxonomy of college outcomes permits us to look at four different general clusters of outcomes based on the intersection of the two dimensions: cognitive-psychological (e.g., subject matter knowledge, critical thinking), cognitive-behavioral (e.g., level of educational attainment, occupational attainment, income, and the like), affective-psychological (e.g., attitudes, values, personality orientations), and affective-behavioral (e.g., leadership, choice of a major, career choice, use of leisure time, and so on). Using Astin’s model as an organizing principle and guide for defining the parameters of the evidence to be considered, the chapters of this book address different broad categories of college outcomes designed to provide coverage of the four taxonomic cells. Some chapters fit rather neatly into a single cell. For example, chapters on the acquisition of subject matter knowledge and academic skills and on the development of general cognitive competencies and skills (Chapters and ) fall into the cognitive-psychological cell; chapters on psychosocial development and values and attitudes (Chapters through ) fit generally into the affective-psychological cell; and the chapter on educational attainment (Chapter ) falls into the cognitive behavioral cell. Other chapters, however, include more than one cell. The chapter on moral development (Chapter ) probably taps both the cognitive-psychological and affective-psychological cells. Similarly, the chapters on career and economic returns and quality of life (Chapters and ) tap both the cognitive-behavioral and affective-behavioral cells.
It is important to state explicitly that the focus of this volume, as with How College Affects Students, is on the outcomes of college for individual students. Certainly, evidence concerning the impact of postsecondary education on society is a scholarly topic worthy of attention. However, as in its predecessor, we judged a synthesis of the societal benefits of postsecondary education to be beyond the scope of this volume.
Obviously, some artificiality is inherent in any separation of the outcomes of postsecondary education into discrete categories. We know from the evidence we reviewed for How College Affects Students, as well as from other research and research reviews (Buczynski, 1991; Davis & Murrell, 1993; Kuh et al., 1991; Love & Goodsell Love, 1995), that a student does not develop in discrete, unrelated pieces, but rather grows as an integrated whole. Indeed, one of our major conclusions from How College Affects Students was that student growth along any one dimension is often highly related to, and perhaps even dependent upon, growth along other dimensions. For example, principled moral judgment may develop with, and perhaps even depend upon, growth in formal or abstract reasoning. Still, the daunting size, diversity, and complexity of the existing literature makes some reasonable taxonomy or categorization necessary if one is to make sense of the evidence.
One might also reasonably take issue with our organizing the chapters of this book around outcomes rather than influences. Indeed, the seminal work in this area of inquiry, Feldman and Newcomb’s (1969) The Impact of College on Students, takes the latter approach with considerable success. However, attempting to isolate the discrete influences of postsecondary education on students (e.g., residence, peers, faculty, college environment, work experiences, instructional and classroom experiences) is probably fraught with the same limitations of artificial categorization as is an organizing scheme built around different outcomes. Certain aspects of students’ intellectual development during college, for example, may result from the intersection and interaction of a number of influences (e.g., interactions with peers, classroom experiences, work experiences), whose separate effects cannot be clearly disaggregated. Irrespective of the conceptual structure one might use to organize the immense body of evidence on college impact, there are tradeoffs to be made and limitations within which one has to work. We acknowledge that focusing the chapters in the present volume on outcomes is only one of several equally valid ways of conceptually organizing the evidence.
The second major dimension of the conceptual framework of the present synthesis concerned critical questions to be asked of the evidence within each of the broad categories of outcomes. Here, we once again adopted the framework employed in How College Affects Students. This framework, which developed out of previous work by G. Gurin (1971), Nucci and Pascarella (1987) and Pascarella (1985), asks six basic questions within each category of outcomes:
1. What evidence is there that individuals change during the time in which they are attending college? In many ways this is frequently regarded as the most fundamental question and one on which hinges the relevance of many subsequent questions that might be asked concerning the impact of college. It is tempting to believe that if individuals fail to change during college other questions regarding effects of college versus noncollege or the effects of institutional environments are essentially moot. As we observed frequently in How College Affects Students, however, it’s more complicated than that. Put simply, the presence of change during college is no guarantee that college is having an impact. Other influences such as simply growing older (maturation), or the natural improvement that may occur when one takes the same test or instrument twice (practice effect) could account for much or all of the change we attribute to exposure to postsecondary education. Conversely, the absence of change during college does not always mean that college is failing to have an impact. In How College Affects Students, we observed a number of instances where average change or growth during college was either non-existent or trivial. That is, college seniors were about at the same level, or only slightly higher, as freshmen. Yet, students not exposed to postsecondary education tended to substantially retrogress (i.e., change in a negative direction) during the same period of time. In short, college had an impact by anchoring development and preventing its retrogression. In this volume, we continue to summarize new evidence on documented change that occurs during college. In terms of estimating the actual impact of postsecondary education on students, however, it is perhaps the least relevant of the six questions that guide our synthesis. A shorthand expression for the question will be “Change During College.”
2. What evidence is there that change or development during college is the result of college attendance? This question is more specific and, therefore, more difficult to answer than our first one. It is not merely concerned with whether or not change or growth occurs during college but focuses instead on the extent to which whatever change does occur can be attributed to college attendance rather than other causes or influences (e.g., normal maturation, differences in background traits between those who attend and do not attend college). In different contexts and different disciplines this has been referred to as the “unique,” “value-added,” or “net” effects of college. Our shorthand for this question is “Net Effects of College.”
3. What evidence is there that different kinds of postsecondary institutions have a differential influence on student change or development during college? This question is essentially asking whether or not discernible differences in student development or the outcomes of college are attributable to the characteristics of the particular institution attended (e.g., institutional type, student body selectivity, size, financial resources). Since this question is primarily concerned with differential impacts between and among institutions, the shorthand phrase for this question will be “Between-College Effects.”
4. What evidence exists on effects of different experiences within the same institution? This question is concerned with identifying different subenvironments or experiences within the institution (e.g., residence arrangement, academic major, quality of instruction, peer group involvement, extracurricular activities, interaction with faculty) that may have influences on student change or development. The shorthand expression will be “Within-College Effects.”
5. What evidence is there that the collegiate experience produces conditional, as opposed to general, effects on student change or development? The question essentially asks whether various influential collegiate experiences have the same aggregate or general effect for all students or whether these experiences vary in their influence for different kinds of students (e.g., men versus women; minority versus non-minority; low-versus high-aptitude students). While a general effect suggests that a particular experience is the same in magnitude and direction for all students experiencing it, a conditional effect suggests that the magnitude or direction of the effect is conditioned by or varies according to the specific characteristics of the individuals being considered. Thus, for example, a particular experience may have stronger developmental effects for male than for female students. Conditional effects are sometimes referred to as interaction effects in that individual subject differences are said to interact with the particular experience or exposure thought to influence a particular outcome. Our shorthand label will be “Conditional Effects of College.”
6. What are the long-term effects of college? This question addresses the durability or permanence of the collegiate experience, or differences in that experience, on students’ postcollege activities, attitudes, beliefs, and behaviors. Our shorthand phrase will be “Long-Term Effects of College.”
Obviously, and as in our previous synthesis, not all six questions will be meaningful in terms of each category of college outcome considered. The influence of postsecondary education, for example, is manifest much earlier on such outcomes as cognitive or moral development than it is on occupational or economic attainments. Indeed, for the latter two outcomes it makes little sense to talk about development or changes during college.
In How College Affects Students, we reviewed evidence generally covering the time period 1967-1989. The temporal, chronological focus for the present synthesis is from 1989-90 to the present. In several instances, however, we review studies conducted prior to l989-90. We do this for two basic reasons. First, such studies may help us place the synthesis of more recent evidence in context; and second, in case we missed significant studies in our previous synthesis. To identify applicable investigations we initially conducted searches of various abstracting documents and data bases (e.g., Sociological Abstracts, Psychological Abstracts, Dissertation Abstracts, Higher Education Abstracts). We also reviewed conference proceedings from such scholarly and professional associations as the American Educational Research Association, the Association for the Study of Higher Education, the American Sociological Association, and the Association for Institutional Research. This strategy allowed us to obtain studies that had yet to be published or that had never been published. Finally, we also used an extensive network of colleagues to obtain unpublished papers and technical reports that dealt with college impact.
It would be foolhardy to claim that the results of our literature search were exhaustive. In How College Affects Students we reviewed in the neighborhood of 2,600 studies and discovered over the last decade that we missed some studies. If anything, the literature of the last decade is even more complex and extensive, sometimes to the point of being overwhelming. With such a mass of literature to screen we know that some studies were missed. Nevertheless, we believe our search methodology has been thorough and extensive and that the studies we synthesize here comprise a comprehensive representation of the existing evidence.
This volume continued the tradition of the three most comprehensive previous syntheses of evidence on the impact of college, those of Feldman & Newcomb (1969), Bowen (1977), and Pascarella & Terenzini (1991), by being a narrative or explanatory literature review. That is, the synthesis and conclusions were based on a logical explanatory analysis of the literature and are presented in narrative form. That this type of literature review or research synthesis has a strong and lengthy tradition in education and the social and behavioral sciences is evidenced by scanning such journals as Review of Educational Research and Psychological Bulletin and annual reviews such as Review of Research in Education, Annual Review of Sociology, and Higher Education: Handbook of Theory and Research.
To be sure, there are some advantages to using more quantitative techniques, such as meta-analysis, to synthesize a large body of literature. Among these are greater standardization in reporting results, ease of comparability across different bodies of research, and an objective method for resolving conflicting findings in a body of evidence. Despite these advantages, we did not employ meta-analysis as the primary method for synthesizing the literature we reviewed for one important reason. The remarkable diversity of ways in which inquiry on the impact of college is conducted and reported made meta-analysis, with its strict requirements for information from each study, an unwieldy tool for synthesizing the full range of existing evidence. This diversity perhaps reflects the multi-disciplinary nature of research on college impact. Different disciplines often take very different approaches to the conceptual models they employ, the analytical methods they use, and the statistical detail they report. As a result, within any particular category of outcomes one can frequently confront a mass of diverse statistical information. This information may not only be in different form (e.g., partial correlations, increases in explained variance, unstandardized regression weights, standardized regression weights, logistic regression coefficients, adjusted means, or total, direct, and indirect effects in structural equation models), it may also have different meanings depending on how prediction equations are specified (i.e., what controls are effected) or on the form of the outcome itself (e.g., actual versus natural logarithmic transformations of wages or income). Furthermore, a substantial percentage of studies simply did not report adequate information (e.g., standard deviations) for computing effect sizes - the sine qua non for meta-analysis.
Confronted with such overwhelming complexity and diversity in the studies reviewed, we judged it virtually impossible to compute comparable study effect sizes or to aggregate them in a manner that would produce meaningful conclusions. Related to this issue was our concern that the meta-analytic requirement of quantifying study results in a comparable metric exclude studies based on naturalistic inquiry or other relevant investigations whose results were simply not amenable to the computation of effect sizes.
In using a narrative explanatory synthesis as our primary approach to the analysis of evidence, we were, as in our predecessor volume, guided by the criterion of “weight of evidence.” That is, given a logical analysis of the studies conducted, what does the weight of evidence allow us to conclude about the impact of college or the influence of different aspects of the collegiate experience? When operationalized as “box scores” or “vote counts” (i.e., the percentage or proportion of studies that show positive results versus those that do not), the simple criterion of weight of evidence has been found to yield conclusions quite similar to those based on effect sizes computed in meta-analysis. Indeed, there is evidence that the correlations between the two approaches are in the .77 to .87 range (Walberg, 1985).
Our own operationalization of the weight of evidence criterion, while not always in the form of box scores or vote counts, nevertheless has two important characteristics. First, it is not exclusionary in that we try to synthesize all the available studies pertaining to an outcome, not just those that report a certain level of statistical detail. Second, we attempt to take into account variations in the methodological characteristics of studies and to place a greater inferential burden on those investigations that are the most methodologically sound. This effort is particularly important in those instances where findings conflict or where findings from one or two methodologically sound studies conflict with findings from a larger number of less well conducted investigations.
Although we have continued the narrative, explanatory approach which characterized How College Affects Students in this volume, we have still made supplementary use of meta-analytic techniques and results. In several of our chapters, we review the results of meta-analytic work and, in several cases, have employed meta-analytic techniques to corroborate findings or estimate the magnitude of an effect. We have also used subsets of studies in other syntheses to conduct our own meta-analysis of studies in those areas that pertain directly to postsecondary education. In short, wherever we can estimate the magnitude of an effect with reasonable accuracy we try to do so.
A Brief Note on Methodology
What we can confidently conclude about the influence of college or the influence of different collegiate experiences on students is highly dependent on methodological rigor. There is simply no escaping this fact. Throughout our present synthesis, we attempt to deal with issues of research design, measurement, and data analysis as they arise and then as simply and benignly as possible. Nevertheless, since we employ several terms frequently throughout the book, it is important to define them here.
The first of these terms is net effect. The easiest way to explain net effect is through an example. Suppose one wishes to estimate the effect of attending versus not attending college on critical thinking while at the same time controlling for the confounding influence of differences in initial intelligence between college and noncollege groups. If one were to compute the association or correlation between college attendance and a measure of critical thinking while statistically controlling for (or statistically removing) the effect of intelligence, the result would be an estimate of the effect of college on critical thinking net of (or independent of) the confounding influence of initial intelligence. Thus, the term net effect has a relative meaning, depending upon what potentially confounding variables are controlled or taken into account.
The second term is direct effect. A direct effect can be thought of as the unmediated influence of one variable on another (that is, the impact is direct and does not pass through an intervening variable). Although the descriptor direct is only occasionally used in the literature we reviewed, direct effects are by far the most frequently estimated effects in educational and social science research. Using our previous example, if going to college has a significant association with critical thinking when initial intelligence is controlled, then it can be said to have a direct effect on critical thinking net of intelligence. Throughout the text we periodically employ the complete descriptor direct effect. For purposes of brevity and variety, however, we also use the short hand (and more common) term effect to stand for “direct effect.” Thus, whenever the term effect is used without an antecedent modifier, it signifies the direct or unmediated effect of a variable.
Although it is estimated less often than direct effects, a variable may also have an “indirect” or mediated effect on an outcome. An indirect effect occurs when the effect is transmitted through an intervening variable or variables. For example, it is possible that college attendance (versus nonattendance) may have a substantial indirect effect on adult critical thinking by influencing a person’s reading habits. Thus, the path of indirect influence would be college attendance directly influencing reading habits and reading habits, in turn, directly affecting adult critical thinking. In this and similar ways, college could have a significant indirect impact on a range of outcomes in addition to having a direct effect on them—or even without having a direct effect on them.
A final term is total effect. This term means nothing more than the sum of the direct and indirect effects of one variable on another. In some instances the total effect of a variable will consist largely of its direct effect on an outcome. In other instances most of the total effect may be indirect. In still other cases a variable may have substantial direct and indirect impacts.
Our brief introduction of the above terms should afford the reader a basic understanding of them when they are used in the remainder of the book. For a more detailed discussion of each, including their statistical estimation, the reader is referred to the Appendix of How College Affects Students. We attempt to provide a brief working definition of each new statistical or methodological term as it is introduced in the text.
Finally, it is important to make a distinction between the causal meaning of the term “effect” when it is derived from experimental versus correlational studies. The simple fact is that we can make stronger causal statements from the results of experimental studies than we can from correlational studies. The easiest way to illustrate this distinction is through an admittedly contrived example adopted from Pascarella & Terenzini (1991). Suppose we randomly provide half the entering students in a small liberal arts college with a dictionary-thesaurus combination and withhold it from the remaining half. At the end of the first year of college we give the entire class a test of vocabulary and find that those who received the dictionary-thesaurus scored a statistically significant 15% higher than the nonrecipients. Given this randomized, true experiment, we could conclude that the typical improvement in vocabulary achievement we could routinely get by purposefully providing incoming students with a dictionary-thesaurus is around 15%. In short, we can be reasonably confident that the 15% advantage in vocabulary achievement is attributable to (or caused by) the college’s providing students with a dictionary-thesaurus.
Conversely, suppose in a correlational study we find that, with precollege level of vocabulary achievement controlled statistically, having a dictionary-thesaurus (versus not having one) is associated with a statistically significant 15% advantage on the same measure of vocabulary achievement at the end of the first year of college. In this situation we have not been able to manipulate and control the conditions under which the relationship between having a dictionary-thesaurus and end-of-first-year vocabulary achievement is observed. Consequently, the 15% advantage is only an estimate of the average difference in vocabulary knowledge between students who own a dictionary-thesaurus and those who do not, net of precollege vocabulary achievement. We cannot conclude to the same degree we can in the randomized experiment described above that purposefully providing students with a dictionary-thesaurus would produce the same effect.
In short, even though the terms “effect” or “causal effect” are often used in correlational research, they do not have the same meaning with respect to causal certitude as they have in experimental investigations. In the correlational study described above, the strongest inference we can make from the findings is that a causal link between having a dictionary-thesaurus and improved vocabulary cannot be ruled out. Correlational investigations that effect controls for salient confounding influences are extremely useful in identifying plausible causal associations among variables. In the vast majority of investigations on the influence of college, however, statistically significant net associations from correlational research are a necessary but not sufficient condition for inferring causality (Light, Singer, & Willett, 1990).
Evidence on such a diverse topic as the impact of postsecondary education on students varies not only in methodological approach and rigor but also in the focus of research, the characteristics of the samples, and the operational definitions of variables. Consequently, it may be useful for the reader to be aware of several general limitations or problems in the overall body of evidence. Most of these problems were evident in the literature we reviewed for How College Affects Students and remain problems in the literature we reviewed for the present synthesis.
The first problem with the evidence is that the characteristics of samples in the research vary dramatically, from single-institution samples with only a few students to multi-institution, nationally representative samples with hundreds and even thousands of students. In some areas of research, such as the impact of college on moral development or the impacts of instructional interventions on content acquisition, synthesizing the evidence is primarily a task of finding common threads among many small-sample, single-institution and/or single-course studies. Here the key problem, and one that is not always resolved, is the generalizability of the findings. In other areas of research, such as attitudes and values, psychosocial characteristics, educational attainment, economic returns, and personal health, we rely more on findings from secondary analyses of large, nationally representative samples. Such samples increase generalizability but may also come with a price. Secondary analysis often requires the construction of scales from items that may not have been intended for the purpose to which they are subsequently put in a particular investigation. Consequently, the items may end up having only a marginal or surface relationship with the construct they are purported to measure. In short, the price one often pays for the generalizability inherent in national samples is problematic measurement of salient variables.
A second and related problem is that a number of national data sets, which produce a substantial portion of the evidence on the impact of college on students, have become targets of opportunity for large numbers of social scientists. While this was clearly the original purpose of creating such nationally representative data bases, there is an additional price to be paid in terms of interpreting the findings from different studies. For example, the National Longitudinal Study of the High School Class of 1972 (NLS72) has been used by large numbers of economists, sociologists, and other social scientists to estimate the economic returns to postsecondary education or attendance at different kinds of postsecondary institutions. Much of this work appears to have been done in relative isolation with minimal communication between individual scholars and even less between scholars in different disciplines. The result is a body of findings based on the same data set and focusing on the same general outcomes but with different samples, different variables represented (or specified) in the prediction equations, different operational definitions of variables, different analytical procedures, and, quite frequently, different results. We do our best to resolve such conflicts in our synthesis, but it is not clear that complete resolution is always possible.
Third, researchers and lay readers alike should be wary of the potential in large national studies for identifying statistically significant (i.e., a low probability of being due to chance) differences or changes that may or may not have discernible educational, administrative, or policy significance. This potential is an artifact of the sensitivity of tests of statistical significance to large sample sizes: the larger the sample size the more likely one is to detect statistically significant associations between and among variables. To minimize the risk of inferring a mountain from a molehill, wherever possible we try to estimate the magnitude of the effects observed in an area of study.
Fourth, with a few exceptions, such as the research on moral reasoning, reflective judgment and cooperative learning, studies often differ substantially in their operational measurement of the same construct. For example, variables such as critical thinking, college “quality,” liberalism, formal education, career success, and subjective well-being, are measured in a number of different ways in different studies. Such multiple assessment versions of the same construct present a challenge not unlike that of the problematic measurement of constructs in secondary analysis. The challenge is essentially one of determining replicability of results. Is it possible to uncover consistent findings across studies that differ in the instruments used or the operational definitions of the same construct?
Fifth, in some areas of study, particularly those assessing change in cognitive development, psychosocial development, or attitudes and values, the evidence is sometimes derived from instruments that place a premium on stability of measurement. Consequently, some of these instruments may have a built-in bias against reflecting change due to education. Thus, evidence that suggests no shifts in certain student traits may not necessarily mean that no growth or development, in fact, occurred. There is at least the possibility that instruments emphasizing measurement stability may underestimate student growth during college.
Finally, it is important to place some parameters around the evidence we review in this synthesis. As with How College Affects Students, this book limits its focus to the various impacts of postsecondary education on the individual. We certainly acknowledge that education in general, and postsecondary education in particular, provide a wide range of benefits to society. For example, the positive impact of education on health may have important societal implications in areas such as employee productivity and health care costs. However, while such social benefits may be a potentially important outcome of postsecondary education, they fall outside the major focus of our synthesis. Similarly, as with its predecessor volume, this book focuses, in its entirety, on the impact of college or the impact of different college experiences on students. It is not intended to be a book about student cultures, or what college students do, or what it is like to be a college student. These items are certainly important areas for the best inquiry we can produce; but, they are, in and of themselves, beyond the scope of this synthesis. Consequently, many very important and newsworthy studies that describe the experience of college or trends in college student behaviors for different student groups or subcultures are not reviewed, or even cited, in this book. It is not that we think such investigations are unimportant. Rather, they simply do not focus on college impact and, therefore, fall outside what we consider to be the primary concern of our work in this book.
Adelman, C. (1998b). Institutional effects in an age of multi-institutional attendance. Paper presented at the annual meeting of the Association for the Study of Higher Education, Miami, FL.
Astin, A. (1973). Measurement and determinants of the outputs of higher education. In L. Solmon & P. Taubman (Eds.), Does college matter? Some evidence on the impacts of higher education. New York: Academic Press.
Astin, A. (1998). The changing American college student: Thirty-year trends, 1966-1996. Review of Higher Education, 21, 115-135.
Baird, L. (1992). Renovating the campus to enhance the intellectual life of the institution: Perspectives from research. Paper presented at the annual meeting of the Association for Institutional Research, Atlanta.
Baxter Magolda, M. (1993). The convergence of relational and impersonal knowing in young adult's epistemological development. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.
Baxter Magolda, M., & Buckley, J. (1997). Constructive-developmental pedagogy: Linking knowledge construction and students' espistemological development. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Bowen, H. (1977). Investment in learning. San Francisco: Jossey-Bass.
Brooks, J., & Brooks, M. (1993). In search of understanding: The case for constructivist classrooms. Alexandria, VA: Association for Supervision and Curriculum Development.
Buczynski, P. (1991). Longitudinal relations among intellectual development and identity during the first two years of college: A structural equation modeling analysis. Research in Higher Education, 32, 571-583.
Callan, P. (1997). Stewards of opportunity: America's public community colleges. Daedalus, 126, 95-112.
Chronicle of Higher Education: Almanac Issue. (1995, September 1). 42.
College affordability concerns of college freshmen greatest in 30 years. (1996, May). Postsecondary Education Opportunity, 1-6.
Cross, K. (1999). What do we know about students' learning, and how do we know it? Innovative Higher Education, 23, 255-270.
Davis, T., & Murrell, P. (1993). A structural model of perceived academic, personal, and vocational gains related to college student responsibility. Research in Higher Education, 34, 267-289.
Dykstra, D. (1996). Teaching introductory physics to college students. In C. Twomey Fosnot (Ed.), Constructivism: Theory, perspectives, and practice (pp. 182-204). New York: Teachers College Press.
Ehrmann, S. (1995). Asking the right questions: What does research tell us about technology and higher learning? Change, 27(2), 20-27.
Fassinger, P. (1995). Understanding classroom interaction. Students' and professors' contributions to students' silence. Journal of Higher Education, 66, 82-96.
Fassinger, P. (1996). Professors' and students' perceptions of why students participate in class. Teaching Sociology, 24, 25-33.
Felder, R. (1995). Longitudinal study of engineering performance and retention. IV. Instructional methods. Journal of Engineering Education, 84, 361-367.
Feldman, K., & Newcomb, T. (1969). The impact of college on students. San Francisco: Jossey-Bass.
Gardiner, L. (1994). Redesigning higher education: Producing dramatic gains in student learning. Washington, D.C.: Clearinghouse on Higher Education, The George Washington University.
Gardiner, L. (1998). Why we must change: The research evidence. Thought and Action, 14, 71-88.
Gladieux, L., & Swail, W. (1999). The virtual university and educational opportunity: Issues of equity and access for the next generation. Washington, DC: The College Board.
Green, K. (1996). The coming ubiquity of information technology. Change, 28(2), 24-28.
Gurin, G. (1971). The impact of the college experience. In S. Withey (Ed.), A degree and what else? Correlates and consequences of a college education. New York: McGraw-Hill.
Henderson, C. (1995). College freshmen with disabilities: A triennial statistical profile (Research report). Washington, D.C.: American Council on Education, HEATH Resource Center.
Institute for Higher Education Policy. (1999b). Distance learning in higher education. Washington, D.C.: Author.
Kuh, G. (1992a). In their own words: What students learn outside the classroom. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
Kuh, G., Schuh, J., Whitt, E., Andreas, R., Lyons, J., Strange, C., et al. (1991). Involving colleges: Successful approaches to fostering student learning and personal development outside the classroom. San Francisco: Jossey-Bass.
Kuh, G., & Vesper, N. (1999). Do computers enhance or detract from student learning? Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
Kuh, G., Vesper, N., & Krehbiel, L. (1994). Student learning at metropolitan universities. In J. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 10, pp. 1-44). New York: Agathon Press.
Levine, A., & Cureton, J. (1998). Collegiate life: An obituary. Change, 30(May/June), 12-17, 51.
Light, R., Singer, J., & Willett, J. (1990). By design: Planning research on higher education. Cambridge, MA: Harvard University Press.
Love, P., & Goodsell Love, A. (1995). Enhancing student learning: Intellectual, social, and emotional integration. Washington, D.C.: ASHE-ERIC Higher Education Report No. 4.
Mayer, R. (1999). Designing instruction for constructivist learning. In C. Reigeluth (Ed.), Instructional-design theories and models, Volume II (pp. 141-159). Mahwak, NJ: Lawrence Elbaum.
McPherson, M., & Shapiro, M. (1991). Keeping college affordable. Washington, DC: Brookings Institution.
Morganthan, T., & Nayar, S. (1996). Those scary college costs. Newsweek, 127, 52-58.
National Center for Education Statistics. (1996b). Working while in college (NCES Report No. 96-796). Washington, D.C.: U.S. Department of Education, Office of Educational Research and Improvement.
National Center for Education Statistics. (1998). Digest of education statistics 1997. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement.
Nucci, L., & Pascarella, E. (1987). The influence of college on moral development. In J. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 3). New York: Agathon.
O'Brien, E. (1993). Outside the classroom: Students as employees, volunteers, and interns. Research Briefs, 4(1), 1-12.
Pascarella, E. (1985). College environmental influences on learning and cognitive development: A critical review and synthesis. In J. Smart (Ed.), Higher education: Handbook of theory and research (pp. 1-64). New York: Agathon.
Pascarella, E. (1999). New studies track community college effects on students. Community College Journal, 69(June/July), 8-14.
Pascarella, E., & Terenzini, P. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass.
Pascarella, E., & Terenzini, P. (1998). Studying college in the 21st century: Meeting new challenges. Review of Higher Education, 21, 151-165.
Rhoads, R. (1995a). Gay student contraculture and resistance in a university community: Tension between solidarity/difference. Paper presented at the annual meeting of the Association for the Study of Higher Education, Orlando, FL.
Silverman, S., & Casazza, M. (1999). Learning and development: Making connections to enhance teaching. San Francisco: Jossey-Bass.
Stage, F. (1993). Chanting the names of the ancestors. Educational Researcher, 22(6), 22-24.
Stage, F., Muller, P., Kinzie, J., & Simmons, A. (1998). Creating learning centered classrooms: What does learning theory have to say? (ASHE-ERIC Higher Education Report Volume 26 No. No. 4). Washington, D.C.: George Washington University.
Terenzini, P. (1996). Rediscovering roots: Public policy and higher education research. Review of Higher Education, 26, 5-13.
Twomey Fosnet, C. (Ed.). (1996). Constructivism: Theory, perspectives, and practice. New York: Teachers College Press.
Upcraft, M., Terenzini, P., & Kruger, K. (1999). Looking beyond the horizon: Trends shaping sutdent affairs -- Technology. In C. Johnson & H. Cheatham (Eds.), Higher education trends for the next century: A research agenda for student success (pp. 30-35). Washington, DC: American College Personnel Association.
Walberg, H. (1985). Synthesis of research on teaching. In M. Wittrock (Ed.), Third handbook of research on teaching. Washington, DC: Amercian Educational Research Association.
West, T. (1996). Make way for the information age: Reconstructing the pillars of higher education. On the Horizon, 4(4), 1, 4-5.