CHAPTER 4
Cognitive Skills and Intellectual Growth
In Chapter Three we focused on the learning outcomes typically thought to be directly, or at least purposefully, related to the curriculum or academic program of a college, namely, the acquisition of subject matter knowledge and academic (usually verbal and quantitative) skills. In this chapter we turn to the acquisition of more general intellectual or cognitive competencies and skills which, if they are not so directly tied to a particular curriculum or course of study, are nevertheless thought to be salient outcomes of postsecondary education (Jones, 1994; Jones, Dougherty, Fantaske, & Hoffman, 1997; Jones et al., 1995). As we pointed out in our 1991 synthesis, these cognitive skills go by a number of different names (e.g., critical thinking, reflective judgment, epistemological development, and so on), and they differ somewhat in conceptual definition and the types of problems or issues they address. They do, however, have as a common theme the notion of applicability and utility across a wide range of different content areas. These cognitive competencies and skills represent the general intellectual outcomes of postsecondary education that permit individuals to:
process and utilize new information; communicate effectively; reason objectively and draw objective conclusions from various types of data; evaluate new ideas and techniques efficiently; become more objective about beliefs, attitudes, and values; evaluate arguments and claims critically; and make reasonable decisions in the face of imperfect information. These and related general cognitive skills are a particularly important resource for the individual in a society and world where factual knowledge is becoming obsolete at an accelerated rate . . . (Pascarella & Terenzini, 1991, pp. 114-115).
Change During College
Conclusions from
In our 1991 synthesis we concluded that, compared to freshmen, college seniors have better oral and written communication skills, are better abstract reasoners or critical thinkers, are more skilled at using reason and evidence to address ill-structured problems for which there are no verifiably correct answers, have greater intellectual flexibility in that they are better able to understand more than one side of a complex issue, and can develop more sophisticated abstract frameworks to deal with complexity. Our best estimates of the magnitudes of the gains, or the advantage of seniors over freshmen, were as follows: oral communication, .60 of a standard deviation (a 22 percentile point advantage); written communication, .50 of a standard deviation (a 19 percentile point advantage); Piagetian formal (abstract) reasoning, .33 of a standard deviation (a 13 percentile point advantage); critical thinking, 1 standard deviation (a 34 percentile point advantage); using reason and evidence to address ill-structured problems, 1 standard deviation (a 34 percentile point advantage); and ability to deal with conceptual complexity, 1.2 standard deviations (a 38 percentile point advantage). These estimates were based on a large and extensive body of studies.
Evidence from the 1990s
Compared to the literature base we reviewed for our 1991 synthesis, we found relatively few studies focusing on change or gains in general cognitive skills and intellectual growth during college. The literature we did find is focused largely in two broad areas that we term critical thinking and postformal reasoning.
There are many different definitions of, and ways of measuring, critical thinking (Ennis, 1985; Erwin, 1997; B. Moore & Parker, 1989; Paul, 1987, 1992; Sternberg, 1985; Tsui, 1998a). However, it would appear that most attempts to operationally define and measure critical thinking focus on an individual’s capability to do some or all of the following: identify central issues and assumptions in an argument, recognize important relationships, make correct references from the data, deduce conclusions from information or data provided, interpret whether conclusions are warranted based on given data, evaluate evidence or authority, make self corrections, and solve problems (Erwin, 1997). Typical in the measurement of these dimensions of critical thinking is the notion that some answers or solutions are more verifiably correct than others.
It comes as no surprise that the generally accepted dimensions of critical thinking identified above have a strong cognitive component. Recently, however, a number of scholars have argued that, in addition to the cognitive ability to use or apply critical thinking skills, there is also a motivational dimension to critical thinking. They term this dimension the disposition to think critically (Erwin, 1997; P. Facione, Facione, & Giancarlo, 1996; P. Facione, Sanchez, Facione, & Gainen, 1995; Jones, 1993, 1995; Taube, 1997). Thus, critical thinking as a broad concept involves both cognitive skills and the dispositional openness or willingness to apply those skills. Disposition to think critically involves, among other traits, such factors as the inclination to ask challenging questions and follow the reasons and evidence wherever they lead, tolerance for new ideas, willingness to use reason and evidence to solve problems, and willingness to see complexity in problems (P. Facione, Facione, & Giancarlo, 1994; P. Facione et al., 1995). In our synthesis we will review research on both critical thinking skills and critical thinking disposition.
The most extensive research in the 1990s on gains in critical
thinking skills during college appears to have been conducted by Facione (1997). His findings are based on an aggregate sample
of over 6000 students in nearly 150 undergraduate samples from 50 nursing
programs throughout the
Facione’s (1997) results are of course limited by the fact that they are based on samples of nursing students rather than on broader samples of undergraduates. Nevertheless, with some exceptions (Criner, 1992; Marr, 1995), the weight of evidence from other cross-sectional studies, using various standardized measures of critical thinking such as the Watson Glaser Critical Thinking Appraisal or the California Critical Thinking Test, is generally consistent with Facione’s findings in suggesting that the greater one’s exposure to postsecondary education the more advanced one’s level of critical thinking skills (Beck, Bennett, McLeod, & Molyneaux, 1992; Brooks & Shepard, 1990; Drouin, 1992; Hill, 1995; McDonough, 1997; Mines, King, Hood, & Wood, 1990; C. Pearson, 1991). Although not all studies provide requisite statistical data, our best estimate from this group of cross-sectional studies is that seniors have an advantage over freshmen of between .55 and .65 of a standard deviation (21 to 24 percentile points).
The longitudinal research on gains in critical thinking skills is less extensive than the cross-sectional research. Moreover, only one study, (Mentkowski et al., 1991) traces growth in critical thinking for more than three years. Mentkowski et al. administered the inference, assumptions, and deductions scales of the Watson Glaser Critical Thinking Appraisal to 135 undergraduates upon entrance to a small liberal arts college and again toward the end of their senior year. The gain across all three scales from entrance to college through the senior year averaged .25 of a standard deviation (10 percentile points). Facione (1997) and Hagedorn, Pascarella, Edison, Braxton, Nora and Terenzini (1999) both report critical thinking gains based on multi-institutional samples, but only for a three year period. Facione traced the growth of 625 nursing students at eight institutions from their sophomore through their senior year with the California Critical Thinking Skills Test and reported an average gain of .15 of a standard deviation (6 percentile points). Hagedorn et al. analyzed data from a sample of over 1000 students attending 18 four-year colleges located in 15 different states. From the time they entered college until the end of their junior year the students in the sample made an average gain of .37 of a standard deviation (14 percentile points) on the critical thinking module of the Collegiate Assessment of Academic Proficiency (CAAP). The CAAP critical thinking module is an objective, standardized test of critical thinking skills that correlates in the .75 range with the Watson Glaser Critical Thinking Appraisal (Pascarella, Edison, Nora, Hagedorn, & Braxton, 1995).
We uncovered only one additional study that looked at change in critical thinking during college using an objective, standardized instrument. Saucier (1995) administered the Watson-Glaser Critical Thinking Appraisal to several small, convenience samples of nursing students when they entered the program, either in their sophomore or junior year of college. The samples were then followed up with the same test one month before graduation. Saucier reports average gains in two of the three consecutive years covered by the study, but in only one year was the gain statistically significant. The failure to find statistically significant gains in the study is likely due to the very small sample sizes, ranging from 7 to 18. Moreover, since standard deviations were not reported we could not compute effect sizes for the critical thinking gains.
Although change during college is quite often an inaccurate estimate of the actual impact of college, the evidence from the 1990s nevertheless suggests that students are making gains in critical thinking skills during college that are appreciably smaller in magnitude than the gains we observed in our previous synthesis. It is not readily apparent from the evidence why this is the case. However, the more modest evidence of change in the 1990s does underscore an important point. Not all students develop as critical thinkers during college. For example, Keeley (1992) had samples of freshmen and seniors at a single institution read and evaluate two essays to identify the underlying assumptions of the argument put forth. While seniors were, on average, more proficient than freshmen in identifying assumptions, between 58 and 78% of seniors made the mistake of restating at least one premise as an assumption and between 21 and 58% of seniors made the mistake of restating at least three premises as assumptions.
Our synthesis uncovered four studies that estimate growth in the disposition to think critically during college. Three of these studies were cross-sectional (Bers, McGowan, & Rubin, 1996; N. Facione, 1997; Giancarlo & Facione, 1997) and one was longitudinal (Giancarlo & Facione, 1997). All four studies estimated growth in the disposition to think critically with the California Critical Thinking Dispositions Inventory (CCTDI). The CCTDI (P. Facione & Facione, 1992; P. Facione, Facione, Blohm, Howard, & Giancarlo, 1998) is a standardized measure that assesses the extent of one’s internal motivation to engage problems, seek answers to questions, and make decisions using critical thinking skills. It measures this disposition along seven dimensions or subscales: truthseeking, open-mindedness, analyticity, systematicity, critical thinking self-confidence, inquisitiveness, and maturity of judgment; and also reports a total score.
Using the same large multi-institutional sample of undergraduate nursing programs previously described, Facione (1997) administered the CCTDI to a sample of over 2600 undergraduates, consisting of freshmen, sophomores, juniors, and seniors. Using the descriptive statistics reported by Facione, we estimate that the seniors in their sample had an advantage in critical thinking disposition over freshmen of about .50 of a standard deviation (19 percentile points). However, most of the change appeared to have occurred by the second year of college. The estimated critical thinking disposition advantage of sophomores over freshmen was .46 of a standard deviation (18 percentile points), or about 90% as large as the corresponding advantage of seniors over freshmen.
Once again, because the Facione (1997) sample is limited to students in a single professional program there may be some question as to the generalizability of the results. However, two additional cross-sectional studies, one at a private liberal arts university and one at a community college, report findings generally consistent with those reported in Facione’s analyses with undergraduate nursing samples. Giancarlo and Facione (1997) administered the CCTDI to a sample of over 1100 students in all four class levels at a private university. Using an estimated freshman standard deviation on the CCTDI from a subsample of students attending the same institution, we computed a senior advantage over freshmen in critical thinking disposition of .45 of a standard deviation (17 percentile points). Once again, the sophomore advantage over freshmen was almost as large as the senior advantage, but since the sophomore average was not reported numerically we hesitated to compute an effect size.
Bers, McGowan, and Rubin (1996) administered the CCTDI to a sample of 224 community college students and found that disposition to think critically had a statistically significant positive relationship with exposure to postsecondary education, operationally defined as number of credit hours completed. Using the descriptive statistics reported by Bers et al., we estimated that the difference in overall critical thinking disposition between students who had completed 30 or more credits and those who had completed 12 or fewer credits was .68 of a standard deviation, a 25 percentile point advantage for the former group.
The longitudinal study of gains in critical thinking disposition was conducted by Giancarlo and Facione (1997) at the same private university where they conducted their cross-sectional investigation. A sample of 155 students took the CCTDI as beginning freshmen and then again as seniors. From the first to the second testing, the sample increased a statistically significant 7.45 points in critical thinking disposition. This translated into a gain from freshman to senior year of .28 of a standard deviation (11 percentile points).
Critical thinking (as it has typically been assessed) focuses to a great extent on an individual’s ability to solve intellectual puzzles or problems. These puzzles or problems come in many forms and require the application of complex reasoning and information processing; yet they typically have the common trait of a verifiably correct, or at least a more valid, answer. The critical thinking, information processing, and formal reasoning skills involved in puzzle or problem solving are an important acquisition in the development of intellectual resources. However, there are other kinds of real-world adult problems that require a somewhat different approach to reasoning than the intellectual skills typically included under the ruberic of critical thinking. These are “ill-structured” or “wicked” problems for which there is likely to be conflicting or incomplete information, unspecifiable problem parameters, and a number of plausible solutions, none of which may be verifiably correct (P. King & Kitchener, 1994). Examples of such problems are gun control, safety of food additives, waste disposal, objectivity of the press, reducing poverty, teaching evolution in the schools, and the like. In addressing these and similar problems, formal rules of logic may not suffice; and tentative answers or solutions typically need to be “constructed” rather than “discovered.” Descriptions of successful entrepreneurs, managers, researchers, and educators often include references to strong abilities for framing and resolving ill-structured problems (Lynch, 1996; Sherman, 1994). Several scholars and cognitive development theorists (Basseches, 1984; Perry, 970; Wood, 1997) have argued that constructing tentative solutions to such real-world problems requires a set of intellectual capabilities beyond those typically included in the concepts of critical thinking or formal reasoning (hence, the term postformal reasoning). These intellectual capabilities have been operationally defined in a number of ways but the most well known, and those which have dominated the measurement of postformal reasoning, are the Reflective Judgment Interview (P. King & Kitchener, 1994), the Measure of Epistemological Reflection (Baxter Magolda, 1990a; Taylor, 1983), and the Measure of Intellectual Development (Knefelkamp, 1974; W. Moore, 1991a; Widick, 1975).
The Reflective Judgment Interview (RJI) measures reasoning along a multilevel continuum of seven stages. Each level is characterized by a person’s view of knowledge and his or her concept of justification of belief. To paraphrase King and Kitchener (1994), at the lowest stages (pre-reflective thinking), knowledge is gained either by direct, personal observation or through the word of authority and is absolute and certain. Reasoning at the middle stages (quasi-reflective thinking) recognizes that knowledge claims about ill-structured problems contain elements of uncertainty. However, the quasi-reflective individual has difficulty successfully addressing ill-structured problems because of a lack of understanding of their inherent ambiguity. At the highest stages (reflective thinking), it is understood that knowledge useful in addressing ill-structured problems must be actively constructed. Beliefs are justifiable to the extent that they are based on a rational process involving appropriate forms of inquiry and use of evidence. The RJI consists of four dilemmas and a set of standardized probe questions designed to tap level of reasoning. Each dilemma is defined by two contradictory points of view and represents an ill-structured problem. Subjects are asked to state and justify their points of view about the issues in each dilemma. The four dilemmas represent different content domains: science, current events, religion, and history.
In our current synthesis we uncovered only one published study in the first half of the decade that estimated gains during college in reflective judgment. Employing a cross-sectional design Mines, King, Hood, & Wood (1990) tested 20 freshmen and 40 seniors at a large research university with the RJI. Their descriptive statistics indicate that seniors had an advantage over freshmen of about 2.08 standard deviations, which converts to a 48 percentile point advantage. Fortunately, our current synthesis of gains during college in reflective judgment was augmented by two comprehensive literature reviews of research using the RJI with postsecondary samples (P. King & Kitchener, 1994; Wood, 1997). It is apparent from these reviews that our 1991 synthesis missed several unpublished studies and that our current synthesis would also have overlooked several unpublished studies. Consequently, we used the results of the work of King, Kitchener, and Wood (P. King & Kitchener, 1994; Kitchener, Wood, & Jensen, 1999; Wood, 1997) to reestimate the overall relationship between exposure to postsecondary education and level of reflective judgment.
King and Kitchener (1994) report the individual results for 20 cross-sectional studies of 966 traditional age (e.g., 18 year old freshmen and 22 year old seniors) college students using the RJI. The studies were conducted between 1978 and 1993. The evidence suggests a strong positive relationship between exposure to postsecondary education and reflective thinking. Using the individual study data they report in an appendix, we estimated the average reflective thinking advantage of seniors over freshmen across all studies to be .68 of a standard deviation (25 percentile points). Similar summary data reported by Wood (1997) yielded a senior advantage over freshmen of .79 of a standard deviation (29 percentile points). King and Kitchener also report data from 5 cross-sectional studies of 135 nontraditional age college students (e.g., 22 year old freshmen and 26 year old seniors). Using the information they report, we estimate the senior advantage over freshmen across all studies to be .98 of a standard deviation in reflective thinking, or a 34 percentile point advantage.
In their 1994 summary, King and Kitchener also report the results of a smaller number of longitudinal investigations of change during college in reflective thinking. From the data they report, we weighted the studies by sample size and estimated the gain in reflective judgment from the freshman to the junior year to be .65 of a standard deviation (a 24 percentile point gain) and from the freshman to the senior year to be about 1.90 standard deviations (a 47 percentile point gain). These estimates have to be viewed with extreme caution, however, since they are based on only two studies with a combined sample of only 60 students.
More recent research on reflective judgment not included in either the King and Kitchener (1994) or the Wood (1997) summaries would nevertheless appear to be quite consistent with their conclusions. In what is probably the most comprehensive study of reflective judgment to date, Kitchener, Wood, and Jensen (1999) administered the Reflective Judgment Interview (RJI) and the Reasoning About Current Issues test (RCI, an objective paper and pencil measure of reflective thinking) to 1588 freshmen, 141 sophomores, 135 juniors and 265 seniors at four different institutions. Of the total sample, 34 freshmen and 29 juniors at two of the institutions completed the RJI while the remainder at all four institutions completed the RCI. On both measures there was a statistically significant class effect; and using the descriptive statistics reported we estimated the junior advantage over freshmen on the RJI to be .86 of a standard deviation (a 21 percentile point advantage) while the senior advantage over freshmen on the RCI was estimated at .71 of a standard deviation (a 16 percentile point advantage).
Perhaps more important than a quantitative estimate of the gains in reflective judgment that occur during college is an understanding of the qualitative change in thinking processes that appears to happen between the freshman and senior year. Results of both cross-sectional and longitudinal studies using the RJI are quite consistent in indicating that the typical change in reflective judgment between the freshman and senior year is a movement of about half a stage on the reflective judgment conceptual model from stage 3, the end of pre-reflective thinking, to stage 4, the beginning of quasi-reflective thinking. The modest size of this advance may be deceiving in that it represents a qualitative shift from a style of reasoning based on personal beliefs to one that explicitly uses reason and evidence in forming judgments. As such, it may represent an important prerequisite for the development of a reasoned approach to addressing ill-structured problems.
Another line of inquiry on change or growth in postformal reasoning during college has employed the Measure of Epistemological Reflection (MER). The MER was designed to measure development on the first five stages of Perry’s (1970) scheme of intellectual and ethical development. In the first five stages of the Perry scheme the individual moves from an understanding of knowledge that is presumed to be dualistic, absolute and obtainable only from authorities to an understanding of knowledge that assumes multiplicity, that is contextual and relative, and where analytical skills enable one to critique ideas—their own and others. The MER is a written instrument that assesses the respondent’s views in six domains of thinking related to learning and elicits specific justification for the respondent’s thinking. In a longitudinal study at a single institution, Baxter Magolda (1990a) employed the MER, along with a semistructured interview that also addressed the same six domains, to estimate growth during two years of college on the Perry scheme. On both instruments progression through college was associated with statistically significant gains on the Perry scheme. Using the descriptive statistics reported by Baxter Magolda, we estimated the change from the beginning of the freshman to the beginning of the junior year to be about 1.37 standard deviations (41 percentile points) on the MER and 1.73 standard deviations (46 percentile points) on the semistructured interview. Gains from the freshman to the sophomore year were also statistically significant and were about 70% and 35% as large as the two-year gains on the MER and the semistructured interview, respectively.
Similar results are reported in another single institution, longitudinal study by Kube and Thorndike (1991). They also employed the MER, but their study differs from Baxter Magolda’s (1990a) investigation in that they traced growth from the beginning of the freshman year through the end of the senior year. Using the descriptive statistics they report, we estimated the freshman to senior gain on the MER to be 2.54 standard deviations, which translates to an average gain of 49 percentile points.
A final line of research on change or growth in informal reasoning during college is one of several versions of the Measure of Intellectual Development (MID), or the Scale of Intellectual Development (SID). Like the Measure of Epistemological Reflection, the MID is a written instrument that attempts to capture development across the first five positions of the Perry scheme. The somewhat lesser used SID is a multiple choice measure. Results of research using these instruments have typically not reported change in terms of average scores. Rather, the evidence is typically reported in terms of differences in group (e.g., freshman versus senior) distributions across the different stages of the Perry model. With the exception of one study which reports mixed results (Durham, Hays, & Martinez, 1994), the weight of evidence from both longitudinal and cross-sectional investigations constituting this body of research is generally consistent in suggesting a statistically significant, positive relationship between extent of exposure to postsecondary education and level of intellectual development (Hart, Rickards, & Mentkowski, 1995; May, 1990; W. Moore, 1991b; F. Pearson & Rodgers, 1998; P. Thompson, 1991; Zhang & Richarde, 1998). Although there is some variation among studies, generally it would appear that freshmen are functioning intellectually in the transition from Perry stages 2 and 3, while seniors are functioning intellectually between stages 3 and 4. In short, the growth during college in intellectual development defined by the Perry scheme is largely a movement from a dualistic, right-wrong notion of knowledge to one embracing the potential legitimacy of multiple perspectives and the importance of context.[1]
In addition to the evidence based on standardized measures, there is also a small body of research that attempts to estimate growth in cognitive skills and intellectual development during college using student or alumni self reports (Bauer, 1992, 1995, 1996, 1998; Bauer, Mitchell, & Bauer, 1991; Dollar, 1991; Graham & Cockriel, 1989; Kelley, 1994). Consistent with the findings from studies employing standardized measures reviewed above, the weight of evidence from this body of research suggests that the majority of students and alumni perceive that they made moderate to substantial growth during college in such areas as “analytical thinking,” “synthesis of ideas,” “critical thinking,” and “independent learning.”
Net Effect of College
Conclusions from
In the 1991 synthesis, we made the following general conclusions with respect to the net impact of college on general cognitive skills and intellectual growth.
1. Seniors in college have significantly better written and oral communication skills than freshmen, even when controls are made for age and academic ability.
2. Graduates of community colleges score significantly higher on a measure of general intellectual and analytical skill development even in the presence of controls for age, verbal ability, and mathematical ability.
3. There is reasonably sound and consistent evidence to suggest that exposure to postsecondary education has a statistically significant positive effect on critical thinking even when controls are in effect for precollege level of critical thinking, academic aptitude, maturation, family socioeconomic origins, and aspirations. The positive net influence of postsecondary education, at least in the initial years of exposure, however, appears to be concentrated in the enhancement of one’s ability to determine the validity of data-based conclusions and to evaluate the strength or weakness of arguments.
4. Those who attend college make significantly greater gains in reflective judgment, one’s ability to use reason and evidence in making judgments about controversial issues, than those who do not attend college. This difference persists even when controls are made for differences in academic ability.
5. Net of age, intelligence, and academic ability, exposure to postsecondary education appears to have a statistically significant, positive influence on one’s intellectual flexibility (i.e., ability to comprehend and effectively argue both sides of a complex or controversial issue).
Evidence from the 1990s
Although we did not find studies in the current synthesis that spoke to the net effect of college on written or oral communication skills, or intellectual flexibility, the evidence we did uncover is quite consistent with our 1991 synthesis in suggesting that exposure to postsecondary education does, in fact, have a statistically significant, positive effect on both critical thinking skills and post formal reasoning. Moreover, the literature of the 1990s provides a somewhat greater opportunity for estimating the magnitude of the net effect of college on critical thinking and, to a lesser extent, postformal reasoning.
A number of cross-sectional and longitudinal investigations have attempted to estimate the net impact of differential exposure to postsecondary education on critical thinking. The majority of these studies employ objective, standardized measures. Rykiel (1995) administered the Watson-Glaser Critical Thinking Appraisal to independent samples of incoming freshmen and outgoing sophomores at a single community college. The sophomores had completed at least 43 of the 64 credits required for graduation. Introducing statistical controls for a measure of verbal aptitude, Rykiel found that sophomores scored significantly higher on the Watson-Glaser total score and on the evaluation of arguments subscale. Using the group standard deviations and adjusted group means reported by Rykiel, we estimated that sophomores in her study had a net advantage over incoming freshmen in total critical thinking of .34 of a standard deviation (13 percentile points). Since we could not find group standard deviations for the Watson-Glaser evaluation of arguments subscale we could not estimate an effect size for this group comparison. However, Rykiel’s finding on this subscale is partially consistent with the conclusion from our 1991 synthesis that the effect of the early years of college on critical thinking skills appears to be largely one of fostering the ability to determine the validity of data-based conclusions and to evaluate the strength or weakness of arguments.
Although Rykiel’s (1995) findings might be confounded by age or maturation, they are nevertheless also consistent with longitudinal research in which maturation is controlled. Analyzing longitudinal data from 13 four-year and 4 two-year colleges participating in the National Study of Student Learning (NSSL), Pascarella, Bohr, Nora, and Terenzini (1996) sought to determine the effects of exposure to postsecondary education, operationally defined as credit hours taken, on end-of-first-year critical thinking. Critical thinking was assessed with the critical thinking module from the Collegiate Assessment of Academic Proficiency (CAAP), which is a standardized test quite similar to the Watson-Glaser Critical Thinking Appraisal. In the presence of statistical controls for level of precollege critical thinking, race, gender, age, academic motivation, work responsibilities, the average critical thinking level of students at the institution attended, and type of coursework taken, four-year college students taking 24 or more credit hours during the first year of college had a statistically significant advantage in end-of-first-year critical thinking over students taking 6 or fewer credit hours of about .41 of a standard deviation. This number converts to an advantage of 16 percentile points. The corresponding advantage for two-year college students was also statistically significant and was estimated at .24 of a standard deviation, or 10 percentile points (see also Klassen, 2001).
The Pascarella et al. (1996) estimate of the net effect of postsecondary education on critical thinking for two-year colleges (.24 of a standard deviation) is somewhat smaller than the estimated effect from Rykiel’s (1995) two-year college study (.34 of a standard deviation). However, the smaller effect reported by Pascarella et al. is based on one year of exposure to postsecondary education while Rykiel’s study encompasses nearly two years. The effect on critical thinking of one year of exposure to postsecondary education reported by Pascarella et al. in their four-year college sample (.41 of a standard deviation) is quite consistent with the results of an earlier study (Pascarella, 1989) included in our 1991 synthesis. In that study, Pascarella matched samples of secondary school seniors who attended and did not attend college on the Watson-Glaser and followed them for an entire academic year. Net of such factors as initial level of critical thinking, academic ability, secondary school grades, and educational aspirations, those students with one year of exposure to college had an advantage of .44 of a standard deviation in critical thinking over those students who did not attend college during the first year after graduation from secondary school.
Other research has attempted to estimate the net impact of college on critical thinking beyond the first year. In the Mines, King, Hood, and Wood (1990) study reviewed in the previous section on change during college, samples of seniors and freshmen at a single institution were administered both the Watson-Glaser Critical Thinking Appraisal and the Cornell Critical Thinking Test. In an attempt to determine if the freshman-senior differences noted on these tests were confounded by academic ability, Mines et al. adjusted the group differences for composite ACT score. Net of this measure of academic ability, seniors still had a statistically significant advantage over freshmen on both measures of critical thinking. Since no adjusted means are reported, however, we could not estimate an effect size.
While the results of the Mines et al. study might be confounded by maturation, other research that controls for the differential effects of age, among other factors, reports similar results. Doyle, Edison, and Pascarella (1998) followed students from 18 four-year colleges and universities in the National Study of Student Learning through the end of their third year of college. With controls in effect for a battery of confounding influences such as precollege level of critical thinking, academic motivation, race, gender, work responsibilities, socioeconomic status, campus residence, coursework taken, and the quality of instruction received, the number of credit hours taken still had a statistically significant, positive effect on end-of-third-year scores on the critical thinking module of the Collegiate Assessment of Academic Proficiency. Using descriptive statistics available from the National Study of Student Learning (Hagedorn et al., 1999), we estimated that students who had completed 72 or more credit hours had a net advantage in critical thinking over students who had completed 18 or fewer hours of .55 of a standard deviation, or 20 percentile points. Since this estimate of the net effect of three years of college is actually larger in magnitude than our previous estimate of simple change during four years of college, we suspect the latter is estimate is overly conservative.
We uncovered four cross-sectional studies that attempt to estimate the net effects of college on postformal reasoning using the Reflective Judgment Interview. Because of the interview nature of the RJI each of the studies is based on relatively small samples, ranging from 38 to 63. Thus, they have only limited statistical power to detect significant reflective thinking differences between students with different levels of exposure to college. Nevertheless, the weight of evidence from these studies supports the conclusion that undergraduate education probably does have a net positive impact on the development of reflective thinking. Evans (1989) and Mines, King, Hood, and Wood (1990) each administered the RJI to small samples of freshmen and seniors at a single institution. Evans matched the two groups with respect to gender, college grades and academic aptitude (ACT composite scores) and found that seniors had a statistically significant advantage over freshmen of about .9 of a standard deviation (22 percentile points). Mines et al. statistically controlled for ACT composite and likewise found that seniors had a statistically significant advantage over freshmen in reflective thinking. A more recent study by Jensen, Kitchener, and Wood (1999) found that the senior advantage over freshmen on the RJI became statistically non-significant when the effects of verbal ability were controlled. However, analyzing data from what appears to be a different sample, Kitchener, Wood, and Jensen (1999) found that, even when statistical controls were made for ACT composite scores, juniors still had a statistically significant advantage over freshmen in reflective thinking. Since the last three studies (i.e., Mines et al.; Jensen et al.; and Kitchener et al.) do not report statistically-adjusted group means, we were unable to estimate net effect sizes from their results.
A fourth study attempting to estimate the net impact of college on reflective judgment employed the recently developed objective measure of reflective thinking called the Reasoning About Current Issues Test (RCI). In the Kitchener, Wood, and Jensen (1999) investigation reviewed in the previous section on “change during college,” the authors introduced a statistical control for ACT composite score. With this control in place, the senior advantage over freshmen in reflective thinking remained statistically significant, although we could not estimate a net effect size because ACT-adjusted means were not reported. Most recently, however, Wood (2000) presented additional evidence on the RCI that permitted estimation of a net senior-freshman effect size. Using an expanded sample of students from seven institutions, Wood had comparison groups of freshmen and seniors that were essentially matched on ACT composite score. Freshman ACT scores averaged 25, while the average ACT of his seniors was 24.72. Despite ACT score parity, however, the median performance of seniors was about 18 to 20 percentile points higher than the median performance of freshmen across the discrimination and endorsement of judgment sections of the RCI.
The weight of evidence from these studies suggests that freshman-senior differences in reflective thinking, whether measured by the RJI or the RCI, cannot typically be explained away by differences in academic ability. It is still possible of course that such net effects are confounded by age (P. King & Kitchener, 1994).
The small body of research that estimates the net impact of college on general cognitive skills and intellectual development employing student self-reports is generally consistent with the evidence based on objective, standardized measures. For example, Whitmire and Lawrence (1996) analyzed data from a sample of over 9,000 undergraduates in different types of institutions using a measure of student self-reported intellectual development consisting of estimates of gains in such areas as ability to think analytically, ability to put ideas together, ability to learn on one’s own, and quantitative thinking. With statistical controls in place for age, college grades, academic major, and measures of the college environment, year in college had a statistically significant, positive impact on self-reported gains in intellectual development, although it was not possible to compute an effect size. Similarly, in a single institution study by Grayson (1996), graduating seniors had statistically significant advantage in analytical and communication skills over entering students that persisted in the presence of controls for gender, race, and language spoken in the house. The net advantage of graduating seniors over entering freshmen was approximately .65 of a standard deviation, or 24 percentile points.
Between-College
Effects
Conclusions from How College Affects
Students
In our 1991 synthesis we found only very limited evidence to suggest that the development of general cognitive skills was influenced by institutional characteristics. When general cognitive growth was assessed by self-reports there was some evidence to suggest it is positively influenced by attendance at an institution with an academically selective undergraduate student body. However, this relationship generally disappeared in studies using objective, standardized measures of cognitive growth and/or controlling for important precollege characteristics. We concluded that student-body selectivity in and of itself may tell us little about institutional influences on general cognitive skills and intellectual growth. Rather, selectivity may only have a discernible impact on student general cognitive development if it is combined with other institutional factors such as small size and an institutional ethos that encourages a high level of student academic effort and involvement.
There was some cross-sectional evidence paired with statistical controls suggesting that African American students demonstrate greater development in measures of critical thinking and concept attainment at historically black institutions than at predominantly white ones. However, this finding was based on a single study.
The only area where we found replicated evidence with respect to between-college effects on general cognitive skills (and it was based on two studies conducted nearly thirty years apart) was in the influence of institutional curricular emphasis. Despite some methodological limitations, the findings suggested that students at institutions with a strong and balanced commitment to general education demonstrated particularly marked gains in measures of critical thinking and adult reasoning skills.
Evidence from the 1990s
Compared to our previous synthesis we uncovered a relatively larger body of evidence focusing on between-college effects on general cognitive skills and intellectual growth. This evidence falls generally into three categories: 1) institutional characteristics, 2) institutional type, and 3) institutional environments.
As with our synthesis of the evidence on the development of verbal, quantitative, and subject matter competence in the preceding chapter, undergraduate student-body selectivity was by far the dominant institutional characteristic considered in research on between-college effects on general cognitive skills and intellectual growth. The weight of evidence from this body of research is quite consistent with the conclusions from our 1991 synthesis. For example, a series of studies analyzing the National Study of Student Learning data base have attempted to determine if the average student-body academic ability (measured in various, but highly correlated ways) had a net impact on the development of individual students’ critical thinking (Doyle et al., 1998; Edison, Doyle, & Pascarella, 1998; Hagedorn et al., 1999; Prendergast, 1998). The sample consisted of five two-year colleges and 18 four-year colleges. While the sample was small, the four-year colleges ranged from essentially open-admission institutions to some of the most selective liberal arts colleges and research universities in the country. When statistical controls were made for such factors as individual precollege critical thinking, race, gender, family social origins, and age, the average critical thinking level of incoming students at each of the four-year institutions had only trivial and non-significant effects on scores on a standardized measure of critical thinking after three years of college (Hagedorn et al., 1999; Prendergast, 1998). When operationally defined as the average score on a composite of objective tests of precollege reading comprehension, mathematics, and critical thinking, institutional selectivity actually had a small, but statistically significant, negative effect on end-of-third year critical thinking. This negative impact persisted in the presence of controls for individual precollege critical thinking, socioeconomic origins, age, race, gender, and full- or part-time enrollment (Doyle et al., 1998; Edison et al., 1998).
Although based on only five institutions, there is some evidence in the Hagedorn et al. (1999) study to suggest that institutional selectivity positively influences the end-of-first-year critical thinking of two-year college students. However, this was not replicated by Whitt et al. (in press) employing a more extensive set of individual- and institution-level controls.
Results quite consistent with those yielded by the National Study of Student Learning data have been reported in analyses of a much more comprehensive sample of about 200 institutions in the Cooperative Institutional Research Program 1985-1989 data base. When controls were made for precollege academic ability and an extensive set of other potential confounding influences, student body selectivity (the average SAT or ACT score of the incoming students) had no statistically significant impact on the analytic score of the Graduate Record Examination (Anaya, 1992; Astin, 1993a; Dey, 1991).
When gains in general cognitive development are based on student self-reports rather than on objective, standardized measures, evidence concerning the impact of institutional selectivity is mixed and generally inconclusive. Most of the research in this area is based on various longitudinal iterations of the Cooperative Institutional Research Program data, and, with one exception (Volkwein, Valle, Parmely, Blose, & Zhou, 2000), each study estimates the net impact of selectivity after controlling for important precollege confounding influences. Kim (1995; 2002a), Tsui (1998b), and Volkwein et al. (2000) report small positive net effects of institutional selectivity on self-reported gains in such dimensions of cognitive development as critical thinking and analytic and problem-solving skills. However, analyses predicting self-reported gains on the same or very similar cognitive dimensions by Astin (1993b), Dey (1991), Franklin (1993), and Strauss and Volkwein (2001) report no statistically significant impact for institutional selectivity.
The effects on general cognitive skills of institutional characteristics other than student-body selectivity have been estimated. For example, Dey (1991) reported that institutional size negatively influenced at least two of three measures of gains in critical thinking skills; and he hypothesizes that this negative effect may be due to the inhibiting influence that institutional size has on interaction with peers and faculty. This finding, however, is based on a single sample and awaits replication.
The overall weight of evidence suggests that the net effects of institutional type on students’ general cognitive development are generally trivial in magnitude or inconsistent. For example, research based on two independent samples found that when student precollege characteristics and other potential confounding influences are taken into account, attendance at a two-year community college versus a four-year college or university has no statistically significant impact on end-of-first-year scores on a standardized, objective measure of critical thinking skills (Bohr et al., 1994; Pascarella, Bohr, Nora, & Terenzini, 1995a; Terenzini, Springer, Yaeger, Pascarella, & Nora, 1994). Similar results are reported with respect to the impact of attending an historically Black college (HBC) or predominantly White institution (PWI) on the development of critical thinking skills in African American students. When level of precollege critical thinking and other important confounding influences were controlled statistically, African American students attending HBCs had end-of-first- and end-of-third-year scores on a standardized, objective measure of critical thinking that were essentially indistinguishable from those of their counterparts attending PWIs. Evidence more supportive of the cognitive benefits of attendance at an HBC is reported by DeSousa & Kuh (1996), Kim (1995; 2002a), and Flowers (2002). All of these studies found that African-American students attending HBCs reported significantly greater progress on self-reported measures of gains in critical thinking or analytical skills than African American students at PWIs; although only Kim and Flowers introduced controls for potential confounding influences. Thus, when cognitive outcomes are assessed with self-report measures there is some support for the tentative conclusion from our 1991 synthesis that attendance at an HBC may foster increased levels of critical thinking in African American students. This was not the case, however, when outcomes are assessed with objective measures.
Other institutional typologies have been considered with respect to their impact on students’ general cognitive development. These typologies include: private versus public control (Franklin, 1993; Terenzini, Springer, et al., 1994); women’s versus coeducational institutions (Kim, 1995; 2002a); and Carnegie, or similar, classification (e.g., research university, doctoral university, comprehensive college, liberal arts college) (Franklin, 1993; Whitmire & Lawrence, 1996). The results of these investigations suggest either trivial and statistically non-significant or inconsistent effects. Kuh (1993b) did find that students at liberal arts colleges reported greater gains in cognitive complexity than their counterparts at comprehensive or metropolitan institutions; but without controls for student background characteristics, it is difficult to determine just how much of this difference is simply attributable to differential recruitment effects rather than to institutional impacts.
Institutional
Environments
As with our synthesis in the preceding chapter, research on the impact of institutional environments on the development of general cognitive skills and intellectual growth is characterized by a wide range of approaches and operational definitions of “institutional environment.” We believe the evidence, however, can be generally grouped into the following three general environmental emphases: scholarship and learning, close student-faculty relationships, and vocational/professional training.
A small body of research has focused on the extent to which an institution’s emphasis on scholarship or learning has a net influence on students’ intellectual growth and development of general cognitive skills. The clear weight of evidence from this research indicates that the stronger an institution’s scholarly emphasis the greater the growth in intellectual skills. This relationship holds irrespective of whether or not intellectual skills are measured with objective tests or student self-reports and even with controls for other institutional characteristics and/or environmental emphases. For example, Prendergast (1998), analyzing the National Study of Student Learning data, used a three-item measure estimating the extent to which students perceived the institutional environment as placing an emphasis on being critical, evaluative, and analytical, as well as emphasizing the development of scholarly and intellectual qualities and the development of expressive, aesthetic, and creative qualities. Controlling for an extensive battery of confounding influences, including student precollege critical thinking and institutional selectivity, this measure of an institution’s scholarly emphasis had a modest, but statistically significant, positive influence on end-of-third year scores on an objective measure of critical thinking. Analyzing the same data and employing a similar analytic design, Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) reported similar findings for end-of-first-year critical thinking; but in their analyses it was the institution’s emphasis on being critical, evaluative, and analytical that accounted for the influence. The findings of Prendergast and Terenzini et al. with an objective measure of critical thinking are essentially replicated in studies using the same environmental scale but student self-reports of their gains in intellectual and analytic competencies (Arnold, Kuh, Vesper, & Schuh, 1993; Kuh, Arnold, & Vesper, 1991; Whitmire & Lawrence, 1996).
Although based on studies that employ diverse measures, there is nevertheless a body of evidence to suggest that the extent to which an institution emphasizes close relationships and frequent interaction between faculty and students has implications for students’ general intellectual/cognitive development. For example, analyses by Terenzini et al. (1994) found that students’ end-of-first-year scores on an objective measure of critical thinking were positively and significantly (if modestly) influenced by the extent to which they perceived that the faculty at their institution were accessible to students and concerned about student development and teaching. This positive effect persisted even in the presence of controls for an extensive battery of potential confounding influences, including student level of precollege critical thinking, institutional control, and the scholarly emphasis of the institutional environment. Generally consistent findings are yielded when general cognitive growth is measured by student self-reports. After controlling for important confounding influences, Astin (1993b), Graham (1998), and Kuh, Arnold, and Vesper (1990) all provide evidence to suggest that student self-reported gains in critical thinking, analytical skills, or general intellectual development are significantly enhanced when the institution’s faculty are oriented toward students or when faculty are accessible to students and are concerned about student growth and development during college.[2]
Finally, there is a small body of literature which considers the impact on students’ general cognitive development of an institutional environment that emphasizes vocational preparation. The findings of this body of research are contradictory, and the direction of effects depends upon use of objective or student self-report measures of cognitive outcomes. Both Arnold et al. (1993) and Kuh et al. (1991) report that student self-reports of their gains on a measure of intellectual skills that includes analysis, synthesis, and learning on one’s own are positively influenced by an institutional environment that stresses professional/occupational competence and the practical value of one’s course of study. This relationship persisted in the presence of controls for students’ levels of social and academic involvement and other measures of the institutional environment. Conversely, using an objective measure of critical thinking after three years of college as the dependent variable, Prendergast (1998) found that the same scale used by Arnold et al. and Kuh et al. to assess an institution’s emphasis on vocational preparation had a statistically significant negative influence. This relationship also persisted when controls were made for such influences as level of precollege critical thinking, measures of academic and social involvement, institutional selectivity and other measures of the institutional environment.[3]
Within-College Effects
Conclusions from How College Affects Students
In our 1991 synthesis, we made the following general conclusions about within-college effects on general cognitive skills and intellectual growth.
1. One’s major course of study has a selective impact on the development of general cognitive skills. A student’s cognitive growth is greatest on measures where the content is most consistent with his or her academic major. On general measures of critical thinking (e.g, the Watson-Glaser Critical Thinking Appraisal) or postformal reasoning (e.g, reflective judgment), one’s academic major has little consistent relationship with gains.
2. The learning cycle-inquiry approach to instruction has been shown to enhance the development of formal (abstract) reasoning and conceptual complexity. This approach stresses an active, inductive learning process in which learner involvement in actual experiments or other concrete activities is used to introduce concepts and abstractions.
3. We found no single instructional or curricular approach that consistently and significantly facilitated the growth of critical thinking when critical thinking was measured by general instruments such as the Watson-Glaser Critical Thinking Appraisal. There was statistically non-significant evidence to suggest instruction that stresses student discussion at a relatively high level of cognitive activity and/or instruction that emphasizes problem-solving procedures and methods may enhance critical thinking.
4. There is evidence that a curriculum experience that requires the integration of ideas and themes across courses and discipline enhances critical thinking over simply taking a distribution of courses without an integrative rationale.
5. Specifically structured course interventions may enhance the development of postformal reasoning, specifically, stage movement on the Perry continuum. These interventions, which have been termed cognitive developmental instruction, focus on providing challenges to the individual’s initial cognitive and value structures paired with instructional supports appropriate for the individual’s initial level of cognitive development.
6. The estimated magnitude of instructional or curricular effects on measures of general cognitive skills tends to be smaller than that of the effect of the overall college experience. Put another way, a single instructional or curricular experience over a limited period may not provide the developmental impact of a cumulative set of mutually reinforcing experiences over an extended period of time.
7. Extent of growth in general cognitive skills during college appears to be a direct result of a student’s quality of effort or involvement in college. Involvement in intellectual and cultural activities may be more important to general cognitive development than other types of involvement (social, athletics, and so on). Yet it also appears that the nature and quality of social interactions with faculty members and student peers play a role of some consequence in one’s cognitive growth. These interactions are of particular salience if they focus on ideas or intellectual matters. The weight of evidence suggests that the more one’s social experience reflects and reinforces one’s academic experience, the greater will be the possibilities for intellectual development.
As with the evidence pertaining to within-college effects on the acquisition of verbal, quantitative, and subject matter competence, the corresponding evidence from the 1990s on more general cognitive skills and intellectual growth is extensive. Again, in an attempt to make organizational sense of this vast literature we have grouped the studies into the following six general clusters: 1) academic major, 2) coursework/curricular patterns, 3) general pedagogical approaches, 4) teacher behaviors, 5) academic effort/involvement, and 6) social and extracurricular effort/involvement.
Academic Major
Critical Thinking
It is difficult to form any firm conclusion concerning the impact of academic major on critical thinking. A substantial number of studies find no statistically significant differences in scores on standardized critical thinking measures among students from different academic fields of study (McDonough, 1997; Money, 1997; Sebrell & Erwin, 1998; Spaulding & Kleiner, 1992). In those studies that do find significant critical thinking differences among students in different academic majors (Gadzella & Masten, 1998; Gunn, 1993), it is difficult to determine if academic or other experiences in the major cause differences in critical thinking or if different academic fields of study simply tend to attract students with different levels of critical thinking ability to begin with (L. Tsui, 1999).
Studies that attempt to separate the recruitment effects from the socialization effects of different academic majors produce similarly inconsistent results. For example, Beckett (1996) found that health science majors made significantly greater gains than liberal arts or business majors on the Watson Glaser Critical Thinking Appraisal. Since the health science majors started out with higher scores this is just the opposite of what would happen if the differences in gains were simply the result of regression artifacts (Pascarella & Terenzini, 1991). Conversely, King, Wood, and Mines (1990) found that differences among seniors in different academic majors on the Watson Glaser Critical Thinking Appraisal and the Cornell Critical Thinking Test essentially disappeared when controls were introduced for academic aptitude. Similarly inconsistent results are reported when the dependent measure is self-reported gains in the ability to think critically rather than objective, standardized tests of critical thinking skills. Some studies find net differences due to academic major (Smart, Feldman, & Ethington, 1999; Whitmire & Lawrence, 1996) while others do not (Li, Long, & Simpson, 1998, 1999). Characteristic of this body of studies is the use of different operational definitions of academic major, which likely contributes to the inconsistent findings. Even with this taken into account, however, we find little consistent evidence to suggest that one’s major field of study, in and of itself, leads to different effects on general measures of critical thinking.
Reasoning Skills
We found very little research that focused on the impact of academic major on reasoning skills. The research that we did find, however, clearly underscores the notion that intellectual training in different fields of study leads to the development of different reasoning skills. For example, a longitudinal study by Lehman and Nisbett (1990) followed a sample of undergraduate students from the first semester of their initial year in college through the second semester of their senior year. The sample was divided into four groups, based on their eventual major: natural science, humanities, social science, and psychology. There were only chance differences among the groups in first-semester scores on measures of verbal reasoning, statistical/methodological reasoning, conditional reasoning, and academic ability. At the end of four years of college, however, there was a distinct pattern of differences in reasoning gains made that reflected one’s major field of study. Students majoring in the social sciences and psychology demonstrated substantially greater gains in the ability to solve real-world or scientific problems requiring statistical or methodological reasoning (average gain about 65%) than did natural science or humanities majors (average gain about 25%). Conversely, natural science and humanities majors made substantial and statistically significant gains in the ability to solve real-world or scientific problems requiring conditional reasoning (average gain about 63%), while their counterparts majoring in the social sciences or psychology made essentially no improvement in conditional reasoning during four years of college. With the exception of the impact of majoring in humanities on conditional reasoning, for which Lehman and Nesbit candidly admit they have no explanation, the differential impact of academic major on different reasoning skills would appear to reflect the type of inductive reasoning emphasized in different undergraduate academic fields (i.e., conditional reasoning in the natural sciences and statistical/methodological reasoning in the social sciences).
Other investigations have also found significant differences in reasoning or metacognitive skills among students in different academic majors (e.g., Zhang & Richarde, 1998). There is little to suggest, however, that anything but chance differences exist among students in these academic majors in the magnitude of the freshman-to-senior gains in reasoning skills.
Coursework/Curricular Patterns
One reason that major field of study may demonstrate little consistent impact on growth in critical thinking skills is that it represents too broad a representation of the formal academic experience one has during college. Thus, it is not particularly surprising that a number of investigations have focused on the impact on critical thinking of specific patterns of courses taken. Unfortunately, there is only limited consistency in the findings of these studies. A series of analyses of the National Study of Student Learning data have found that, even when such factors as precollege critical thinking, institutional characteristics, and other types of course exposure are taken into account, the number of science or engineering courses taken in the first year of college has a modest, but statistically significant, positive impact on end-of-first-year scores on the critical thinking test of the Collegiate Assessment of Academic Proficiency (Doyle et al., 1998; Terenzini, Springer et al., 1994). Similar results are reported by Olsen (1990a) in a single institution study using the same standardized measure of critical thinking and by Tsui (1999) in a large multi-institutional study using students’ self-reported growth in critical thinking. Olsen found that the number of science courses taken had a positive effect on critical thinking scores after two years of college, even when controls were made for academic aptitude, though not for other types of coursework taken. Tsui (1999) reported that number of science courses taken positively influenced seniors’ self-reported growth in critical thinking even in the presence of controls for level of exposure to other types of coursework and several instructional variables, though not for precollege academic aptitude.
There is also evidence, using objective, standardized measures to suggest that the number of arts and humanities courses taken may enhance end-of-first-year critical thinking (Terenzini, Springer et al., 1994), the number of literature courses taken may foster critical thinking after two years of college (Olsen, 1990a), and the number of social science courses taken may have positive effects on critical thinking measured after the third year of college (Edison et al., 1998). In contrast to the effects of natural science courses, however, we uncovered no replication of these findings on independent samples.
A different approach has been taken by Smith-Sanders and Twale (1997; 1998) in a single-institution study that sought to determine if degree of exposure to a required general education or core curriculum influenced scores on a standardized test of critical thinking. The general education or core curriculum included courses in the social sciences, natural sciences, mathematics, English composition, history, fine arts, philosophy, and the “Great Books.” The findings are somewhat difficult to interpret. With statistical controls for precollege academic ability, degree of exposure to the core curriculum had a significant curvilinear relationship with critical thinking skills. Generally, the greater one’s exposure to the institution’s core curriculum the higher one’s critical thinking skills; but, for reasons that were not entirely clear, students with the highest level of core curriculum hours completed had somewhat lower levels of critical thinking than students with less core curriculum hours completed. This finding, however, is based on a single sample and awaits replication.
Postformal Reasoning
A small number of studied have attempted to determine the impact of coursework or curricular organization on growth in various measures of postformal reasoning. At first glance it is difficult to find any basic commonality among the studies in terms of how they conceptualize or organize the notion of coursework or curricular experiences. However, there does appear to be a common thread running through investigations by Schilling (1991) and Wright (1989; 1992). Both studies report the impact on students’ growth in postformal reasoning of exposure to an interdisciplinary or integrated core curriculum that emphasized making explicit connections across courses and among ideas and disciplines. Courses were typically designed, often by faculty teams, to be integrative in content and to stress synthesis of relationships and connections among different academic disciplines.
In the Schilling (1991) study, a sample of seniors who enrolled in the integrated, or interdisciplinary, core curriculum was matched with a group of seniors who were not exposed to it on entering academic ability (ACT score), sex, and area of academic interest. Both groups took the Measure of Epistemological Reflection (recall that the MER measures reasoning development along the first five stages of Perry’s (1970) continuum of intellectual and ethical development). Although there was not sufficient statistical data reported to estimate an effect size, Schilling reported that seniors enrolled in the interdisciplinary core curriculum scored at a significantly more advanced stage on the Measure of Epistemological Reflection than did the matched seniors who were not enrolled. Generally consistent results are reported by Wright (1992) for growth along the Perry continuum during the first year of college. A sample of first-year students took the Learning Context Questionnaire (LCQ), another measure of development on the Perry continuum, when they first entered college and again at the end of their freshman year. With statistical controls introduced for initial LCQ scores and SAT mathematics scores, the number of interdisciplinary, general education courses completed had a statistically significant, positive association with development in reasoning, as measured by gain scores on the LCQ during the first year of college.
Both the Schilling (1991) and the Wright (1992) studies appear to be potentially confounded by the interaction of student self-selection and change. Moreover, in the Wright study, the interdisciplinary courses also introduced several non-traditional pedagogical approaches such as collaborative projects and dialogues. Consequently, it is not clear how much of the positive impact of interdisciplinary courses on informal reasoning was confounded by pedagogical practices. Despite these methodological problems, however, the Schilling and Wright investigations are potentially important in that they represent replicated evidence linking integrated, interdisciplinary general education coursework with development or growth in postformal reasoning.[4]
Other scholars have also sought to assess the impact of innovative coursework and/or curricula on informal reasoning. For example, Pavelich and colleagues (Pavelich, 1996; Pavelich & Moore, 1996; Pavelich, Olds, & Miller, 1995) have investigated changes in engineering students’ position on the Perry continuum during exposure to an innovative curriculum that stresses an experiential, problem-solving approach to engineering education. The centerpiece of the curriculum appears to be a series of courses that emphasize: real-world, open-ended problem solving; team building; and oral and written technical communication skills. The assumption is that these experiences would help students mature toward more complex thinking as well as foster increased capabilities for more effective decisions about ambiguous, real-world, engineering problems. Using an interview approach to measuring position on the Perry continuum, Pavelich and colleagues report that seniors exposed to the curriculum have a statistically significant advantage of about 2.3 standard deviations (49 percentile points) in reasoning development over freshmen with little or no exposure to the curriculum. In the absence of a control group, however, it is difficult to determine just how much of the senior advantage in reasoning is uniquely attributable to the innovative coursework and curriculum versus how much would occur simply by four years of postsecondary education without exposure to the innovation. [For example, recall from the earlier section of this chapter on change during college that Kube and Thorndike (1991) reported a simple freshman to senior gain on the Measure of Epistemological Reflection of 2.54 standard deviations.] In this regard, it is worth noting that a generally similar curricular innovation in nursing education was not significantly linked to students’ growth in reflective judgment (Nickerson, 1991).
There is substantially less evidence with respect to the impact
of general or broad-based pedagogical approaches on cognitive skills and
intellectual growth than there is with respect to their impact on verbal,
quantitative and subject-matter competence.
Nevertheless some research does exist and we have organized it into the
following three categories: 1) computers and information technology, 2)
collaborative/cooperative learning, and 3) interventions designed to increase
cognitive growth.
Computer and Information Technology
Interestingly, the most extensive research we uncovered on
computers and the development of general cognitive skills and intellectual
growth focused on the impact of learning a computer program language. A fairly large number of studies have
addressed this issue. Fortunately, Liao
& Bright (1991) have conducted a
meta-analysis of some 65 of these studies.
Their criteria for inclusion of a study were as follows: 1) it had to
assess the relation between computer programing and general cognitive skills
such as planning skills, thinking skills, reasoning skills, and metacognitive
skills (all cognitive skills were measured by standardized tests); 2) it had to
take place in an actual classroom setting; and 3) it had to have a control
group which did not require students to learn a computer language. Of the 65 studies in their meta-analysis, only
nine were conducted with postsecondary samples.
Those nine studies yielded 20 effect sizes. We took the raw statistical data provided by
Liao and Bright for the postsecondary studies and estimated that college
students required to learn a computer program language had an advantage of .35
of a standard deviation (14 percentile points) in various general cognitive
skills over their counterparts who did not learn a computer program
language. This effect size was statistically
significant at p <.05. Thus, it would
appear that the impact of learning a computer language may extend beyond the
specific computer language to the development of general cognitive
capabilities.
Although they do not focus specifically on learning a computer
program language, reasonably consistent results are reported in correlational
studies by Flowers, Pascarella, and Pierson (1999) and Kuh and Hu (2000). Analyzing the National Study of Student
Learning data, Flowers et al. sought to determine if different types of
computer use influenced critical thinking during the first year of college. To this end, they introduced controls for an
extensive series of confounding influences such as pre-college critical
thinking, academic motivation, full- or part-time enrollment, coursework taken,
and quality of instruction received. In
the presence of these controls, the extent to which coursework required students
to learn to use computers had a modest, but significant positive relationship
with end-of-first-year scores on a standardized critical thinking measure for
students in five community colleges. The
corresponding effect for students at four-year colleges was not
significant. In analysis of data from
over 70 four-year colleges, Kuh and Hu found that using computers for such
learning activities as searching the internet for course material, analyzing
data, and making visual displays each had small, positive effects on student
self-reported gains in intellectual development (e.g., synthesizing, thinking
analytically and logically). These
positive effects persisted even in the presence of statistical controls for such
factors as sex, race, socioeconomic background, grades and educational
aspirations, academic major, work responsibilities, and institutional
characteristics.
We also uncovered one study that focused on the use of electronic mail as a pedagogical tool for influencing aspects of general cognitive development. Marttunen (1997) conducted a study in which undergraduates practiced informal argumentation with each other using e-mail. Argumentation was based around ideas presented in two books, and level of argumentation was determined by the extent to which an argument was grounded in reason or evidence. Over a six-week period, the student groups did not have face-to-face meetings, but there was a significant increase in the level of their argumentation in e-mail messages. The absence of a control group of individuals who engaged in the same type of argumentation, but without using e-mail, makes it difficult to determine the unique effects of e-mail itself. Nevertheless, the results do suggest the possibility that electronic mail is a feasible tool in practicing and improving one’s level of argumentation.
Collaborative/Cooperative Learning
An interesting correlational study by Karabenick and Collins-Eaglin (1996) suggests why one might expect collaborative or cooperative learning approaches to foster general cognitive skills and intellectual growth. Using data from over 1000 students in 57 classes, they found that the greater the class emphasis on collaborative learning and the lower the emphasis on grades, the more likely students were to use higher-order learning strategies of elaboration, comprehension monitoring, and critical thinking. (Elaboration is the attempt to relate ideas in one’s class to ideas in other courses; comprehension monitoring is the attempt to try to figure out a point when one becomes confused; and critical thinking refers to consideration of alternatives to a conclusion or point made in class.) To the extent that use of higher-order learning strategies leads to the development of higher-order thinking skills, one might then expect collaborative and/or cooperative approaches to learning to facilitate the development of general cognitive skills and intellectual development during college. Although not unequivocal, there is a body of evidence to support this expectation.
Qin, Johnson, and Johnson (1995) conducted a meta-analysis of 43 experimental and quasi-experimental studies that considered the effects of cooperative versus individualistic or competitive learning approaches on general problem solving skills. Problem solving was operationally defined as “a process that required participants to form a cognitive representation of a task, plan a procedure for solving it, and execute the procedure and check the results” (Qin et al., 1995) (p. 131). Four types of problems were considered: 1) linguistic problems, which are primarily solved in written or oral languages; 2) nonlinguistic problems, which are primarily represented and solved in pictures, graphs, mazes, symbols, or formulas; 3) well-defined problems, which have well defined operational procedures and solutions (e.g., a chess problem); and 4) ill-defined problems, which have uncertainty with regard to operational procedures and ultimate solutions (e.g., real-world problems such as deciding which car to buy). We took the raw statistical data provided by Qin, Johnson, and Johnson for 20 of the 43 studies that were carried out with postsecondary samples and conducted our own (secondary) meta-analysis. We estimate that, compared to their counterparts not learning in a cooperative format, college students learning in cooperative groups had a statistically significant advantage in overall problem solving of .47 of a standard deviation (18 percentile points). The magnitude of this advantage was largely unchanged by differences in the methodological quality of studies. (Estimates of the methodological rigor of each study were provided in the Qin, Johnson, & Johnson meta-analysis). Similarly, it appeared that the advantage in problem solving accruing to students engaged in cooperative learning was essentially the same for both well-defined problems (.46 of a standard deviation) and ill-defined problems (.49 of a standard deviation).
Additional secondary analysis of the studies conducted with postsecondary samples provided in the Qin, Johnson, and Johnson (1995) meta-analysis suggested a substantial difference in the magnitude of the cooperative learning advantage in solving linguistic problems (.31 of a standard deviation) versus nonlinguistic problems (.96 of a standard deviation). This finding with college students paralleled Qin, Johnson, and Johnson’s results when studies were aggregated across all education levels. They suggest that one reason for this difference in problem-solving advantages accruing to cooperative learning is that there may be more ways to solve nonlinguistic problems than linguistic problems. Thus, cooperative group discussion, “which may result in a great number of strategies being suggested may give cooperators a greater advantage over competitors on nonlinguistic than on linguistic problems” (Qin et al., 1995) (p. 139).
Johnson, Johnson, and Smith (1996) conducted an additional meta-analysis of the effects of academic controversy on the development of cognitive reasoning skills. Recall from our more detailed description in Chapter 3 that academic controversy is a variation of cooperative learning in which a cooperative group of four students is divided into two pairs who are then assigned to opposing positions on a major topic of controversy in a discipline or field of study. As previously indicated in Chapter 3, their meta-analysis was based on 26 published studies, 75% of which were randomized true experiments and about 42% of which were conducted with postsecondary samples. No separation of effect sizes is made for postsecondary and non-postsecondary studies. Nevertheless, they found that, compared with approaches in which the student learns on his or her own, academic controversy produced an advantage of .90 of a standard deviation (22 percentile points) in measures of cognitive reasoning.
Results from the National Study of Student Learning pertaining to the net effects of collaborative/cooperative learning experiences on students’ self-reported gains in analytical skills (Cabrera, Nora, Bernal, Terenzini, & Pascarella, 1998) are quite consistent with the postsecondary-level findings derived from the Qin, Johnson, and Johnson (1995) and the Johnson, Johnson, and Smith (1996) meta-analyses. Evidence with respect to the impact of collaborative/cooperative learning on critical thinking, however, is less consistent, and depends largely on whether critical thinking is assessed with student self-reports or objective, standardized measures. For example, analyzing the 1985-1989 Cooperative Institutional Research Program Data, Tsui (1999) found that participating in group learning projects had a significant, positive influence on seniors’ self-reported growth in critical thinking. This effect persisted even in the presence of controls for coursework taken and other instructional variables. Conversely, a class-level quasi-experiment (Miller & Groccia, 1997), and a multi-institutional correlational study with statistical controls for precollege critical thinking and a battery of other confounding influences (Doyle et al., 1998) found that cooperative learning experiences had only small and chance effects on standardized measures of critical thinking skills.
A substantial body of research in the 1990s has sought to determine if purposefully designed instructional interventions during college can positively influence general cognitive skills and intellectual growth. The majority of this research has focused on critical thinking skills (Dale, Ballotti, Handa, & Zych, 1997; Gadzella, Ginther, & Bryant, 1996) or the disposition to think critically (Bers et al., 1996). Although some studies employ self-report measures of critical thinking strategies or critical thinking gains (Logan & Salisbury-Glennon, 1999; Peterson, 1996; Reiter, 1994), the preponderance of investigations employ objective, standardized tests of critical thinking, such as the Watson Glaser Critical Thinking Appraisal, the Cornell Critical Thinking Test, and the California Critical Thinking Skills Test (Dale et al., 1997; Forbes, 1997; Gadzella et al., 1996; Gadzella, Hartsoe, & Harper, 1989; Inlow & Chovan, 1993; Langer & Chiszar, 1993; West, 1994) or essay measures of critical thinking (MacPherson, 1999; Price, Wilmes, & Turmel, 1994; Weast, 1996). The clear majority of studies describe the experimental intervention as one in which students are explicitly or implicitly taught critical thinking or problem-solving skills, typically imbedded within a semester- or quarter-long course; but there does not appear to be a clear operational consensus as to just exactly what that means pedagogically. Put another way, it is not particularly clear that instruction or practice in critical thinking skills means anywhere near the same thing across studies. As a result, it is not surprising that the body of evidence from this research is far from unequivocal. [Tsui (1998a) reached a similar conclusion, though based on a substantially smaller sample of studies.]
All of the studies we uncovered using pretest-posttest designs without a control group report that students exposed to critical thinking skills interventions make statistically significant gains in measured critical thinking skills (Gadzella et al., 1996; Logan & Salisbury-Glennon, 1999; MacPherson, 1999; Peterson, 1996). The absence of a control group, however, makes it difficult, if not impossible, to determine what part of these gains is uniquely attributable to instruction in critical thinking skills and what part is due to confounding factors, such as maturation, taking the same test twice, or simple exposure to the academic content in which critical thinking instruction might be embedded. The results of experimental and quasi-experimental investigations which include a control group of students not receiving explicit instruction in critical thinking skills are more equivocal than studies without a control group. Some find that critical thinking interventions confer a statistically significant advantage in critical thinking skills on students exposed to them (Dale et al., 1997; P. Facione et al., 1998; Weast, 1996), some report mixed effects (Bers et al., 1996; Langer & Chiszar, 1993; Reiter, 1994; West, 1994), and some report no statistically significant advantage associated with exposure to critical thinking interventions (Forbes, 1997; Gadzella et al., 1989; Inlow & Chovan, 1993; Price et al., 1994). The methodological rigor of the studies does not appear to be associated with whether or not they find positive, mixed, or chance results. Not all of the studies provide the requisite statistical information to compute effect sizes. However, using the statistical information from those studies that do, we estimate that purposeful instruction in critical thinking skills, broadly defined, leads to an advantage in students’ measured critical thinking skills of approximately .23 of standard deviation (9 percentile points). We would caution, however, that this is a rough estimate.
Consistent with the earlier conclusions of Halpern (1993), we would conclude from the evidence we reviewed that development in critical thinking can be enhanced by purposeful instruction and practice in critical thinking and/or problem solving skills. We would amend that conclusion, however, by pointing out that the effects may be widely variable and, therefore, not particularly large. We suspect that, in large measure, this may be due to the general absence of a consensus across studies with respect to the operational definition of what constitutes instruction in critical thinking.
In addition to the research on critical thinking, a smaller body of studies has focused on the impact of purposefully designed course or instructional interventions on the development of postformal reasoning (Kronholm, 1996; Marra, Palmer, & Litzinger, 2000; McAdams & Foster, 1998; S. Thompson, 1995). Postformal reasoning in these studies is measured either in terms of reflective thinking (P. King & Kitchener, 1994) or the Perry (1970) continuum. Overall, this body of research is based on carefully conducted experiments or quasi-experiments with relatively sound internal validity. Moreover, there is a general, if not total, consistency of findings across studies. Students experiencing instructional interventions designed to increase their intellectual development and skills in addressing ill-structured problems tend to score at more advanced levels on measures of reflective thinking or intellectual development than their counterparts not exposed to the intervention.
For example, Kronholm (1996) conducted an experiment to determine if a one semester instructional intervention would positively influence students’ reflective thinking. Students in two general education science courses were assigned to one of three groups: an intervention group and two control groups. Students in all three groups were assessed with the Reflective Judgment Interview (RJI) at the beginning and end of the course. The intervention group was taught according to King and Kitchener’s (1994) reflective judgment-developmental instruction model, while the two control groups received course instruction without the reflective judgment-developmental instruction emphasis. Reflective judgment-developmental instruction begins with the introduction of an ill-structured problem or issue and progresses to an exploration of the problem or issue. This is accomplished by very focused questions and activities aimed at encouraging students to think about epistemological assumptions - or how they come to know (P. King & Kitchener, 1994). With statistical controls for initial RJI position, Kronholm found that students exposed to reflective judgment-developmental instruction made significantly greater average gains in reflective thinking than the average of the students in the two control groups.
Generally, if not totally, consistent results are reported by Thompson (1995) in a quasi-experimental assessment of the impact of a similar intervention emphasizing reflective judgment-developmental instruction on gains in reflective thinking. Students exposed to the intervention scored significantly higher at the end of the study on the Reflective Thinking Appraisal (a written measure of reflective thinking or judgment) than did their counterparts not receiving the intervention. (The two groups demonstrated only chance differences in ACT scores and RTA scores at the beginning of the study.) The difference between the two groups at the end of the study on the Reflective Judgment Interview were in the same direction, but not statistically significant.
Slightly different approaches were taken by Marra, Palmer, and Litzinger (2000) and by McAdams and Foster (1998). Marra et al. investigated the impact of a project-focused, active-learning, team problem-solving course in engineering design, while McAdams and Foster estimated the effects of Deliberate Psychological Education (DPE). DPE stresses active practice in problem solving related to actual role-taking experiences and augmented interactive exchanges with peers. With controls introduced for sex, academic aptitude, prior grades, class standing and honors program participation, Marra et al. found that students who had completed the engineering design course had significantly more advanced positions on the Perry (1970) continuum, as measured by a semi-structured interview, than students who were yet to take the course. McAdams and Foster reported that students randomly assigned to Deliberate Psychological Education instruction made positive, though not statistically significant, advances on the Perry continuum, while their counterparts not receiving DPE instruction actually retrogressed slightly. Similarly, the DPE students made a significant gain on a measure of conceptual complexity, while the control students did not.
The McAdams and Foster (1998) study did not report sufficient statistical information to compute an effect size. However, the other three studies did (Kronholm, 1996; Marra et al., 2000; S. Thompson, 1995). Using the information they provided, we estimated that interventions designed to improve intellectual development, in terms of postformal reasoning skills, provide an average advantage in the development of postformal reasoning of about .65 of a standard deviation (24 percentile points). We would caution, however, that because it is based on only three studies this estimate may not be particularly robust.
Finally, there is some evidence that constructivist-oriented pedagogy can facilitate students’ movement toward more sophisticated and complex epistemological beliefs. For example, recall from Chapter 3 the quasi-experimental study conducted by Hofer (1994; 1998-99) in which sections of a calculus course were randomly assigned to a constructivist-oriented approach called “new wave” and a traditional approach. The new wave sections emphasized active and collaborative learning activities in and out of class, use of graphing calculators, and a curriculum based on textual material that was largely a collection of complex work problems for which answers were not provided. The traditional sections used a standard calculus text that proceeded sequentially, and material was covered primarily through lecture and presentation by the instructor. The new wave and traditional students differed in only chance ways in terms of potentially confounding influences such as gender, ethnicity, high school calculus background and grades, and tested mathematics ability. At the end of the course, however, the new wave students were significantly more likely to hold more sophisticated epistemological beliefs about mathematics knowledge (e.g., math problems may have more than one right answer) than the traditionally taught students. The advantage accruing to the students in the constructivist-oriented, new wave sections was about .16 of a standard deviation (6 percentile points).
Generally consistent findings are reported by Baxter Magolda (1992b) in an intricately detailed qualitative investigation that analyzed the experiences of a sample of students during their four years of college. In the course of periodic interviews, students reported a number of factors that promoted their epistemological development. One of the important factors was teachers defining learning as mutually constructed meaning and, thereby, encouraging students to engage in thought processes much like critical thinking.
We uncovered only a smattering of evidence concerning the impact of teacher behaviors on general cognitive skills and intellectual development. The research that speaks to this issue most directly comes from analyses of the National Study of Student Learning data. Pascarella, Edison, Nora, Hagedorn and Braxton (1996) and Edison, Doyle, and Pascarella (1998) sought to determine if students perceptions of the extent of teacher organization and preparation (e.g., class time is used effectively) and teacher instructional skill and clarity (e.g., instructors give clear explanations) in the overall teaching they receive at their institution significantly influenced critical thinking. In both studies critical thinking was assessed with an objective, standardized measure; and each introduced statistical controls for an extensive set of confounding influences such as precollege critical thinking, academic motivation, race, sex, age, full- or part-time enrollment, patterns of coursework taken, and work responsibilities. In the presence of these controls, extent of teacher organization and preparation had modest, but statistically significant, positive effects on critical thinking at the end of the first- and third-years of college. The net effects on critical thinking of teacher instructional skill and clarity were nonsignificant. Thus, it would appear that a teacher behavior (i.e., organization and preparation) that, as was seen in Chapter 3, has been shown to positively influence course-level content mastery, may also have implications for students’ development of more general cognitive skills during college.
Somewhat more indirect evidence of the impact of teacher behaviors or orientations to teaching is suggested in a single-institution study conducted by Kember and Gow (1994). They sought to determine how two teaching orientations termed “knowledge transmission” and “learning facilitation” influenced the ways in which students approached learning. Teachers with a knowledge transmission orientation tended to think that disciplinary or subject-matter knowledge is most important, to be imparted by clear presentation of information to students. Conversely, teachers who were oriented toward a learning facilitation approach emphasized the development of problem-solving skills, critical thinking, and independent learning; they believed that good teaching should motivate students and be an interactive process. The findings suggested that departments with faculty who exhibited a greater use of learning facilitation were significantly more likely to have students who took a “deep approach” to learning. A deep approach characterizes learners who have an intrinsic interest in the subject and who search for personal meaning in learning activities. In contrast, where a knowledge transmission approach predominated, students tended not only to engage in surface learning, but also to change more in the direction of surface learning. Students who utilize a “surface approach” to learning are extrinsically motivated and therefore concentrate on memorizing content that might appear in examinations.[5] Generally consistent results are reported by Tsui (2000) in a four-institution study that uses self-reported gains in critical thinking as the dependent measure.
Academic Effort/Involvement
In the preceding chapter we saw that the development of verbal, quantitative, and subject matter competence was determined, not only by a student’s formal coursework and instructional experiences, but also by his or her individual level of academic effort or involvement. How much students learn is determined to a great extent by how much personal effort and time they are willing to invest in the process of learning itself. It should come as no great surprise that level of academic effort and involvement also weighs in as a substantial influence on the growth of general cognitive skills and intellectual development during college.
For example, a series of analyses of the National Study of Student Learning have attempted to determine the factors influencing critical thinking during college (Doyle et al., 1998; Terenzini, Springer, Pascarella, & Nora, 1995). Both of these studies assessed critical thinking with an objective, standardized measure (the critical thinking module of the Collegiate Assessment of Academic Proficiency), and both introduced statistical controls for an extensive battery of potential confounding influences, such as precollege critical thinking, academic motivation, student demographic variables, full- or part-time enrollment, work responsibilities, patterns of coursework taken, and measures of student social involvement. In the presence of such controls, factors such as hours studied per week, number of non-assigned books read, and an academic effort/involvement scale (e.g., “took detailed notes in class,” “did additional readings,” “participated in class discussions”) had statistically significant, positive effects on critical thinking skills at the end of the first year of college.
Similar results are reported for measures of postformal reasoning in studies by Kitchener, Wood, & Jensen (1999) and May (1990). Kitchener et al. found that first year growth in reflective thinking or judgment, as measured by the Reasoning About Current Issues Test (an objective test of reflective thinking), was significantly linked with the extent to which students were actively involved in learning experiences. This included such dimensions as involvement in writing experiences and engagement in course learning. The more students actively processed information and ideas acquired in class the greater their gains in reflective thinking. May reported that freshman to senior advances in intellectual development on the Perry (1970) continuum were significantly and positively correlated with a scholarly/intellectual involvement scale that assessed students’ effort in such areas as use of the library, writing experiences, and course learning.
In addition to those studies that measure general intellectual or cognitive development with standardized assessment instruments, there is also a substantial body of evidence to suggest that level of academic effort/involvement has a positive net influence on student self-reported gains in critical thinking and intellectual development during college. While the research designs of these studies are variable, nearly all attempt to statistically control for important confounding influences. Though not all factors are controlled in each study, these controls in different studies include such factors as precollege academic ability, race, sex, educational aspirations, family socioeconomic status, institutional characteristics, and extracurricular involvement. In the presence of such controls, measures of student self-reported gains in critical thinking/intellectual development appear to be positively influenced by such specific factors as amount studied (Astin, 1993b; Cabrera et al., 1998, Volkwein, Valle, Parmley, Blose, & Zhou, 2000); participation in honors programs (Kim, 1996); library use and involvement (Whitmire, 1998; Whitmire & Lawrence, 1996; Williams, 1996); engagement in course learning activities (Whitmire, 1998; Williams, 1996); and more global measures of academic effort and engagement that combine specific dimensions such as studying, reading for pleasure, involvement in library experiences, involvement in course learning, and involvement in writing experiences (Arnold et al., 1993; Franklin, 1995; Grayson, 1995; Kaufman & Creamer, 1991; Kuh et al., 1991; Li et al., 1999; Watson & Kuh, 1996).
As with the preceding chapter on the development of verbal, quantitative, and subject matter competence, there is a substantial amount of evidence pertaining to the ways in which student social and extracurricular involvement influence general cognitive skills and intellectual growth. To maintain a parallel organization with Chapter 3 we review that evidence with respect to: interactions with peers; interactions with faculty; Greek affiliation; intercollegiate athletic involvement; service involvement; diversity experiences; work responsibilities; and on- or off-campus residence.
Interactions With Peers
A modest, but relatively consistent, body of research indicates that students’ peers play a substantial role in their general cognitive growth and intellectual development in college. Indeed, some studies suggest that one’s peers may be an influence that is equal to, and in some cases perhaps even greater, than one’s formal classroom experience (Astin, 1993b; Terenzini et al., 1995; Terenzini, Springer et al., 1994). The vast majority of studies are correlational and typically attempt to determine the net impact of peer interactions on various measures of general cognitive development while statistically controlling for important confounding influences.
For example, analyzing data from the National Study of Student Learning, Whitt, Edison, Pascarella, Nora, & Terenzini (1999a), sought to determine the unique, or net, impact of out-of-class interactions with peers on an objective, standardized measure of critical thinking. To do this, they introduced statistical controls for an extensive set of potential confounding factors such as precollege critical thinking, academic motivation, student demographic characteristics, full- or part-time enrollment, hours spent studying, work responsibilities, patterns of coursework taken, and the average academic ability of the first-year students at the institution attended. In the presence of these controls, a scale measuring the nature and level of involvement in out-of-class-interactions with peers (e.g., discussions about art, music, and theater; discussions with students whose personal values, political beliefs, religious beliefs, or national origin were different from one’s own) had a modest, but statistically significant positive impact on critical thinking skills at the end of the first year of college. Analyzing essentially the same sample, and employing much the same set of statistical controls, Prendergast (1998) found that a somewhat expanded version of the out-of-class interactions scale used by Whitt et al. had a significant, positive net effect on critical thinking skills at the end of the third year of college. Generally consistent findings with independent samples have been reported for scores on the Analytical section of the Graduate Record Examination (Astin, 1993b); for critical thinking skills, as measured by the critical thinking module of the Collegiate Assessment of Academic Proficiency (Twale & Sanders, 1999); for women’s freshman to senior growth on the Perry (1970) continuum, as assessed by the Measure of Intellectual Development (May, 1990); and for first-year gains on the recognition dimension of reflective thinking or judgment, as measured by the Reasoning About Current Issues Test (Kitchener et al., 1999; Kitchener, Wood, & Jensen, 2000). Examples of items measuring students’ interactions with peers from this research included: socializing with peers; hours spent outside of class discussing current issues with peers; having serious discussions with peers whose interests, values, and philosophy of life were different from one’s own; and having conversations with peers that referred to knowledge acquired in classes or readings.
Although less extensive than evidence pertaining to the impact of student interactions with peers, there is related evidence to suggest that student extracurricular or cocurricular involvement may also have positive implications for cognitive development. Analyzing data from a subsample of six National Study of Student Learning institutions, in which approximately half the students commuted to college, Inman and Pascarella (1998) found that a measure of involvement in college clubs and organizations positively influenced end-of-first-year scores on a standardized measure of critical thinking skills. [The same net effect is also reported by Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) with the complete NSSL sample of 23 institutions.] This positive effect could not be explained away by differences in such confounding influences as precollege critical thinking level, academic motivation, student demographic characteristics, full- or part-time enrollment, work responsibilities, measures of academic involvement, and institutional selectivity. Similar findings are also reported by Baxter Magolda (1992a; 1992b) in her detailed qualitative investigation of growth in students’ epistemological sophistication and reasoning skills over four years of college. As suggested by one of the participants in her study: “I think some of my best experiences were outside the classroom, where I could take what I learned in the classroom and apply it” (Baxter Magolda, 1992b, p. 296).
In addition to those studies that employ objective, standardized measures, there is also a body of research that estimates the net impact of interactions with peers on student self-reports of their intellectual or cognitive growth. These studies are also largely correlational in design and tend to introduce statistical controls to adjust for the influence of potential confounding factors. The weight of evidence from this research is also consistent in suggesting that various measures of students’ interactions with peers outside of class have modest, but statistically significant, positive effects on self-reported gains during college in general cognitive, analytical, or intellectual competencies (Arnold et al., 1993; Franklin, 1995; Kaufman & Creamer, 1991; Kim, 2002a; Kuh et al., 1991; Li et al., 1999; Volkwein & Carbone, 1994; Watson & Kuh, 1996).
Interactions With Faculty
A modest body of research permits one to estimate the net impact of student interactions with faculty on general cognitive skills and intellectual development. Although there are exceptions (Doyle et al., 1998; Inman & Pascarella, 1998), the weight of evidence from this body of research suggests that student-faculty interactions that 1) tend to reinforce or extend the intellectual ethos of the classroom, or 2) focus on issues of student development, can have positive implications for general cognitive development during college (Astin, 1993b; Dey, 1991; Franklin, 1993; Frost, 1991; Kim, 1996; 2002a; Kitchener et al., 1999; Kuh, 1995; Terenzini, Springer et al., 1994; L. Tsui, 1999).
For example, Kitchener, Wood, and Jensen (1999) measured gains in reflective thinking during the first year of college with the Reasoning About Current Issues Test. They found that gain-scores on two dimensions of reflective thinking—recognition and discrimination—were both significantly and positively associated with a scale that measured students’ out-of-class-interactions with faculty focusing on such things as course issues, work on a faculty research project, career choice, and personal development. Similar findings are reported in multi-institution-sample, correlational studies that use standardized measures of cognitive development such as the Analytic score of the Graduate Record Examination (Dey, 1991) or that employ student self-reports of growth in critical thinking, intellectual development, or problem-solving skills as the dependent variable (Astin, 1993b; Franklin, 1995; Kim, 1995, 1996). In all of these studies, the effect of student-faculty interaction persists even in the presence of controls for confounding influences such as precollege academic ability, student demographic characteristics, and institutional selectivity.
There is also evidence to suggest that general cognitive skills may be enhanced when student-faculty interactions focus on, or emphasize, student development. In analyses of the National Study of Student Learning data, Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) found that a scale measuring the extent to which faculty were perceived as being concerned with student development and teaching had a significant, positive effect on end-of-first-year scores on a standardized measure of critical thinking skills. This positive effect persisted even in the presence of statistical controls for an extensive list of potential confounding influences that included precollege critical thinking scores, academic motivation, student demographic characteristics, institutional characteristics, coursework taken, full-or part-time enrollment, and measures of academic effort/involvement. The findings of the Terenzini et al. investigation are generally consistent with those of Frost (1991). Frost uncovered evidence in one of two institutional samples that the developmental emphasis of female students’ advising contacts with faculty had a significant, positive effect on both the recognition of assumptions scale and the deductions scale of the Watson-Glaser Critical Thinking Appraisal. These effects persisted even with statistical controls for prior level of critical thinking, SAT scores, high school grades, and family income. Developmental advising (Winston & Sandor, 1984) regards the advising process as developmental education, not just providing information. Consequently, it emphasizes activities designed to help students assume responsibilities for problem-solving and decision making.
We uncovered only two studies that investigate the impact of fraternity or sorority membership on the development of general cognitive skills during college (Pascarella, Edison, Whitt et al., 1996; Pascarella, Flowers, & Whitt, 1999). Both are based on analyses of the National Study of Student Learning data, and both employ a standardized test of critical thinking skills (the critical thinking module from the Collegiate Assessment of Academic Proficiency) as the dependent variable. Pascarella et al. (1996) considered the impact of Greek affiliation during the first year of college at 18 four-year institutions. Statistical controls were introduced for an extensive set of potential confounding influences, including precollege level of critical thinking, academic motivation, student demographic characteristics, full- or part-time enrollment, work responsibilities, patterns of coursework taken, and the academic selectivity of the institution attended. In the presence of these controls, Greek-affiliated men, as compared to their male counterparts who remained independent, had a statistically significant disadvantage of .27 of a standard deviation (11 percentile points) in end-of-first-year critical thinking. The corresponding critical thinking disadvantage for Greek-affiliated women was only about 40% as large as that found for Greek-affiliated men and was not statistically significant.
Pascarella, Flowers, and Whitt (1999) followed the same NSSL sample through the end of the third year of college to determine if the critical thinking disadvantage accruing to Greek-affiliated men persisted beyond the first year. Thus, critical thinking scores at the end of the third year of college were the dependent variable. With statistical controls introduced for the same set of confounding influences as the Pascarella et al. (1996) study, the negative effect of fraternity membership on critical thinking found after one year of college was reduced by about 50% and become statistically nonsignificant. Similarly, the small, nonsignificant negative effect of sorority membership on critical thinking after one year of college remained small and statistically nonsignificant through the end of the third year.
Taken together, the NSSL-based studies suggest that any substantive negative effects of fraternity membership on male critical thinking occur during the initial year of college. Thereafter, the impact of fraternity membership on critical thinking becomes small and statistically nonsignificant. There is little evidence that being a member of a sorority has anything but small and nonsignificant effects on critical thinking skills. These conclusions are based on analyses of a single sample, however, and await replication.[6]
Intercollegiate Athletic Involvement
A small body of research has addressed the impact of intercollegiate athletic participation on objective, standardized tests of critical thinking. Although limited by the small number of studies, this research suggests: 1) that intercollegiate athletes may not be as predisposed as their nonathlete counterparts to think critically, and 2) that male football and basketball players (i.e., those involved in revenue-producing sports) may not be making the same gains in critical thinking skills as either nonathletes or athletes in nonrevenue sports.
McBride and Reed (1998) administered the New Jersey Test of Reasoning Skills (a standardized test purporting to measure critical thinking) and the California Critical Thinking Dispositions Inventory (which assesses the disposition to actually employ critical thinking skills) to small samples of intercollegiate athletes and nonathletes attending a university with a national-level athletic program. Irrespective of gender, athletes scored significantly lower on both tests than did nonathletes. Thus, intercollegiate athletes not only had significantly lower critical thinking skills, they also demonstrated a significantly lower disposition to actually utilize critical thinking skills than nonathletes.
The design of the McBride and Reed (1998) study makes it difficult to determine if the differences in critical thinking between athletes and nonathletes are attributable to participation in intercollegiate athletics during college or simply reflect differences on these traits that the two groups bring to college. To address just this type of methodological issue a series of analyses were conducted with the National Study of Student Learning (NSSL) data (Pascarella, Bohr, Nora, & Terenzini, 1995b; Pascarella, Truckenmiller et al., 1999). These analyses were based on student samples from 18 four-year institutions and employed the standardized critical thinking test from the Collegiate Assessment of Academic Proficiency as the dependent variable. Separate effects of intercollegiate athletic participation were estimated for men and women, with male athletes being further divided into those involved in revenue sports (football and basketball) and those involved in nonrevenue sports (all others). In the first study (Pascarella et al., 1995b), statistical controls were made for precollege level of critical thinking, academic motivation, student demographic characteristics, full- or part-time enrollment, on- or off-campus residence, the academic selectivity of the institution attended, and whether the institution attended fielded an NCAA Division I or a non-Division I athletic program. In the presence of these controls, there were only trivial and nonsignificant differences in end-of-first-year critical thinking scores between women athletes and women nonathletes and among male athletes in revenue sports, male athletes in nonrevenue sports, and male nonathletes.
A follow-up study of the same NSSL sample was conducted to determine if the same parity among intercollegiate athletes and nonathletes in critical thinking extended beyond the first year of college (Pascarella, Truckenmiller et al., 1999). The dependent variable was critical thinking scores at the end of three years of college, and, in addition to controlling for the same variables as in the earlier study (Pascarella et al., 1995b), statistical controls were also introduced for work responsibilities, time spent studying, and patterns of coursework taken. In the presence of these controls there were still only trivial and statistically non-significant differences between women intercollegiate athletes and their non-athlete counterparts on critical thinking after three years of college. There were also only small and non-significant critical thinking differences between male athletes participating in nonrevenue sports and male nonathletes. However, male revenue athletes (i.e., those participating in intercollegiate football and basketball) had significantly lower end-of-third-year critical thinking scores than either male nonrevenue athletes or male nonathletes. Compared to male athletes participating in nonrevenue sports, male football and basketball players had a net critical thinking disadvantage after three years of college of about .35 of a standard deviation (14 percentile points). The corresponding third-year critical thinking skills disadvantage for football and basketball players relative to nonathletes was about .39 of a standard deviation (15 percentile points).
As with previously reviewed findings from the Pascarella et al. (1995b; 1999) studies, summarized in Chapter 3, the negative effects on end-of-third-year critical thinking of participation in intercollegiate football and basketball were similar in magnitude, irrespective of one’s levels of precollege critical thinking, academic motivation, socioeconomic status, or ethnicity. In short, no particular type of male student received greater critical thinking penalties from participating in revenue-producing sports than any other. Furthermore, no evidence was found to indicate that the disadvantage in third-year critical thinking skills accruing to male football and basketball players differed in magnitude at NCAA Division I or non-Division I institutions or at institutions differing in the academic selectivity of their undergraduate student bodies. It is worth noting, however, that all the findings pertaining to the impact of intercollegiate athletic participation on critical thinking are based on analyses of a single sample and await replication.
In Chapter 3 we reviewed evidence suggesting that service experiences, particularly when integrated with course activities through a reflective component (i.e., service learning), may foster increased acquisition of subject matter knowledge. There is also evidence to suggest that such service learning experiences can contribute in positive ways to the development of students’ general cognitive skills and intellectual growth. For example, Batchelder and Root (1994) conducted a quasi-experiment in which experimental sections of a course were taught with an integrated service-learning component and the control sections were taught without the integrated service-learning component. Both the experimental and control courses were similar in content and were taught by the same instructors, although it was not clear if students self-selected themselves into the service-learning sections. At the beginning and at the end of the course, students wrote a 30 minute essay on how they would respond to two situations (e.g., child abuse, alcoholism, initiating a community recycling program) that were designed to assess their dimensions of thinking about a social problem. With statistical controls for pretest scores, students in the service-learning sections demonstrated a significantly greater increase in use of complex and multidimensional perspectives in their essays than did students in the course sections without a service-learning component. We converted the partial correlations reported by Batchelder and Root to an effect size and estimate that the advantage in more complex and multidimensional perspectives accruing to the service-learning students was approximately .25 of a standard deviation (10 percentile points).
An important series of multi-institutional studies by Eyler and her colleagues corroborate and extend the Batchelder and Root (1994) findings (Eyler & Giles, 1999; Eyler, Giles, Lynch, & Gray, 1997; Eyler, Giles, Root, & Price, 1997). However, these basically quasi-experimental investigations take a somewhat more fine-grained approach to understanding the impact of service learning on different dimensions of cognitive development. Instead of simply comparing service-learning with a control group, Eyler and her colleagues hypothesized that student cognitive development is influenced by the degree to which service learning classes are well integrated and highly reflective. Integrated and reflective service-learning classes tend to: 1) integrate service experiences with course content; 2) provide for reflection about the service experience; and 3) permit one to apply subject matter learning to the service experience, and vice versa. The results of their program of research tend to support this hypothesis. The greater the extent to which service-learning courses met the criteria for being highly integrative and reflective, the stronger the positive effects of service-learning on such outcomes as reflective judgment (Eyler & Giles, 1999; Eyler, Giles, Lynch et al., 1997) and the use of complex and multiple perspectives in identifying the causes of and formulating the solutions to specific social problems (Eyler, Giles, Root et al., 1997). The simple addition of service experiences to a course typically had little impact on cognitive outcomes. To have an impact these experiences had to be highly integrated into course content and provisions made for continuous reflection. The effects reported by Eyler and colleagues tended to persist even in the presence of statistical controls for precourse measures of the specific cognitive outcomes. Since the studies do not report all the requisite statistical information, however, we could not estimate an effect size.
There is also quasi-experimental evidence from undergraduate samples at 42 institutions that students involved in four different types of service experiences (education, human services, environment, and public safety) indicated greater self-reported gains in critical thinking skills than their counterparts not involved in service experiences (Astin, 1996a; Astin & Sax, 1996). This effect persisted even when statistical controls were introduced for such factors as volunteering in high school, precollege leadership ability, commitment to participating in community action programs, tutoring other students in high school, gender, and monetary orientation. Consistent results are also reported in a more recent study by Vogelgesang and Astin (2000).
Diversity Experiences
In the preceding chapter we presented evidence that involvement in diversity experiences (e.g., taking ethnic or gender studies courses, discussing racial issues, attending a racial/cultural awareness workshop, socializing with someone from a different racial/ethnic group) can have a positive influence on students’ academic skill development and knowledge acquisition during college. There is also replicated evidence, based on both standardized measures and students’ self-reports, suggesting that involvement in diversity experiences can modestly enhance more general cognitive skills and intellectual development. For example, analyzing the multi-institutional 1985-89 Cooperative Institutional Research Program data, Dey (1991) sought to identify experiences that influenced three measures of cognitive development: the Analytical subtest score on the Graduate Record Examination, self-reported growth in critical thinking ability, and self-reported growth in analytical and problem solving skills. With statistical controls introduced for such factors as SAT verbal scores, secondary school experiences, intellectual self-esteem, self-rated mathematics ability, and institutional characteristics, discussing racial/ethnic issues during the past year had a statistically significant, positive effect on all three measures of cognitive development. Markedly consistent results are reported by Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) in their analyses of data from the 23 two- and four-year institutions participating in the National Study of Student Learning. Attending a racial/cultural awareness workshop had a small, but significant positive impact on end-of-first-year scores on a standardized test of critical thinking skills. This effect could not be explained away by differences in such factors as precollege critical thinking level, student demographic characteristics and aspirations, characteristics of the institution attended, full- or part-time enrollment, work responsibilities, patterns of coursework taken, or extracurricular involvement. Similarly, Kitchener, Wood, and Jensen (2000) found that growth on the Endorsement scale of the Reasoning About Current Issues Test, a measure of reflective thinking, was significantly and positively related to making friends and having discussions with students whose race was different from one’s own. The effect persisted even when initial scores on the RCI Endorsement scale were taken into account.
Analyses of a different iteration of the Cooperative Institutional Research Program data from that analyzed by Dey (1991) have also yielded evidence supporting the positive impact of involvement in diversity experiences on general cognitive development. Both Hurtado (1999) and Kim (1995; 1996; 2002a) analyzed data from the multi-institutional 1987-1991 CIRP sample, with Kim restricting the focus of her investigation to women. Both investigators introduced statistical controls for such factors as institutional selectivity, secondary school achievement, academic self-concept, study involvement, and leadership experiences during college. In the presence of such controls, measures of involvement in diversity experiences (e.g., studied with someone from a different racial/ethnic background, enrolled in an ethnic studies course, attended a racial/cultural awareness workshop) tended to have significant positive impacts on self-reported growth in critical thinking (Hurtado, 1999) and on a measure combining self-reported growth in both critical thinking and problem-solving skills (Kim, 1995, 1996, 2002a).
A more focused approach is reported by Gurin (1999) in two related single-institution studies. The first was essentially a quasi-experiment that compared students who took and did not take an introductory course in a program termed “Intergroup Relations, Community, and Conflict” (IRCC). The IRCC course covered the history of group experiences in the United States, a contemporary analysis of group inequalities in economic, political, and educational arenas, and an analysis of political issues and policies (e.g., immigration, bilingual education, affirmative action, sexual harassment, Middle East peace initiatives, etc.) that are contested by various groups in American society. In addition to lectures and discussion sections, students who took the IRCC course also took part in an ongoing dialogue group designed to: 1) examine between-group and within-group differences on a contested issue; 2) identify and negotiate group conflicts; and 3) find a basis for joint action on an issue. Students who took the IRCC course and the control group who did not were matched in such areas as first-year residence hall, race/ethnicity, gender, and in-state and out- of-state residency prior to college. At entrance to college and at the time of graduation, both groups completed a self-report index designed to measure complexity of thinking (e.g., prefer complex rather than simple explanations, enjoy discussions of causes of behaviors). With statistical controls for precollege level of complexity of thinking, students who completed the IRCC course had significantly higher complexity of thinking levels at the end of college than the control group students who did not. (An effect size could not be estimated from the results reported.) Similar results are reported by Adams and Zhou-McGovern (1994) with respect to a social and cultural diversity course that focused on such topics as racism, sexism, and social justice. Students exposed to the course made advances of about 1 standard deviation (34 percentile points) on the Perry (1970) continuum, but there was no control group.[7]
The second study conducted by Gurin (1999) attempted to determine what diversity experiences other than the IRCC course influenced the complexity of thinking index. A more comprehensive sample was employed, and it was assessed when it entered the institution (1990) and once again prior to graduation. Statistical controls were introduced for such factors as entering level of thinking complexity, SAT/ACT score, high school grades, gender, and racial composition of one’s high school and home neighborhood. With these factors taken into account, a number of diversity experiences had significant positive effects on complexity of thinking at the end of college for White students. These experiences included: 1) classroom diversity (extent to which students reported they had been exposed to diversity issues in classes); 2) personal interactions (extent to which interactions with other racial/ethnic groups involved “honest discussions about race” and “sharing of personal feelings and problems”); 3) number of multicultural events attended (e.g., Black History Month, Hispanic Heritage Celebration, etc.); and 4) participation in dialogue groups (an ongoing program of intergroup dialogue and conflict resolution at the institution). Diversity experiences had no significant effects on thinking complexity at the end of college for African-American students. This may not mean that African-American students receive smaller cognitive benefits from diversity experiences than White students, however. As far as we could tell, Gurin did not report statistically significant differences between regression coefficients for White and African-American students. Thus, the failure to find significant diversity experience effects on thinking complexity for African-American students may be attributable in large measure to the weak statistical power associated with the relatively small number of African-American students in the sample.
The body of evidence concerning the impact of work during college on the development of general cognitive skills and intellectual growth is inconsistent. Some analyses of the National Study of Student Learning data have found small negative effects of hours worked per week on a standardized measure of end-of-first-year critical thinking (Inman & Pascarella, 1998; Terenzini, Springer et al., 1994). Other analyses of the same data, introducing somewhat different statistical controls, have found that extent of on- or off-campus work is essentially unrelated to first- or third-year critical thinking (Pascarella, Bohr et al., 1996; Pascarella, Edison, Nora, Hagedorn, & Terenzini, 1998; Prendergast, 1998). Results of investigations on other samples tend to support the contention that work during college has little or no impact on general cognitive development; and, this finding holds irrespective of whether cognitive growth is measured with standardized tests (Criner, 1992) or student self-reports (Graham & Long, 1998; Kuh, 1995; Strauss & Volkwein, 2001).
We uncovered two studies suggesting the possibility that work can, at times, have a positive impact on critical thinking skills. However, the pattern of effects in the two studies was in marked contrast. Rykiel (1995) found that, when a measure of verbal aptitude was taken into account, community college students who worked 25 hours or more per week had scores on the Watson-Glaser Critical Thinking Appraisal that were about .27 of a standard deviation (11 percentile points) higher than their counterparts who worked 20 hours or less per week. Conversely, Pascarella, Edison, Nora, Hagedorn, and Terenzini (1998) found essentially the opposite with a sample of four-year college students. With controls for precollege critical thinking, demographic characteristics, institutional selectivity, coursework taken, and other factors, part-time on- or off-campus work (i.e., 15-20 hours per week) during college had positive effects on end-of-third-year scores on the critical thinking module of the Collegiate Assessment of Academic Proficiency. Working more than 15 to 20 hours per week, however, had a negative impact. It is not clear how the potentially different meaning of work for students in community colleges and four-year colleges might have shaped these contrasting findings.
We found little in the total body of evidence on work and cognitive development to indicate that the nature of the findings depended upon the methodological rigor of the study. Thus, given the general inconsistency of results across studies, we conclude that there is little compelling empirical evidence to suggest that on- or off-campus work in general has anything more than a trivial impact on cognitive or intellectual development during college. However, it may be, as suggested by Aper (1994; 1997), that work during college has a more salient impact on a student’s cognitive or intellectual development when it is related to his or her academic field of study or intended career. The literature on cognitive development during college is largely silent with respect to this type of inquiry. As we shall see in the chapter on career, however, work related to one’s major or career can have important implications for early career success.
As with the impact of work, the evidence pertaining to the direct effects on general cognitive growth of living on-campus (versus off-campus and commuting to college) is mixed and generally inconclusive. For example, a single-institution study by Pascarella, Bohr, Nora, Zusman, Inman, and Desler (1993) found that students living on-campus made significantly larger first-year gains in critical thinking skills (as measured by the critical thinking module of the Collegiate Assessment of Academic Proficiency) than their counterparts who commuted to college. The greater gains of resident students persisted even in the presence of controls for precollege critical thinking level, academic motivation, age, work responsibilities, and full- or part-time enrollment. There is little support, however, for a corresponding positive impact on critical thinking in investigations based on multi-institutional samples (Doyle et al., 1998; Inman & Pascarella, 1998; Whitt et al., in press). Each of these studies used a somewhat different subsample from the National Study of Student Learning data, and each introduces controls for somewhat different sets of confounding variables. At a minimum, however, each controlled for precollege critical thinking, the academic selectively of the institution attended, full- or part-time enrollment, academic motivation, and work responsibilities. Both the Inman and Pascarella (1998) and the Whitt et al. (in press) investigations uncovered no significant net effect of living on-campus on end-of-first-year scores on the critical thinking module of the Collegiate Assessment of Academic Proficiency. Doyle, Edison, & Pascarella (1998) actually found a significant negative effect, although this may be due in large measure to their combining four-year college samples with two-year college samples that had very few resident students.
Studies that follow students beyond the first year of college have generally found that living on-campus (versus off-campus and commuting to college) has little or no net impact on standardized measures of critical thinking (Prendergast, 1998; Whitt et al., in press). The only exception to this finding is reported by Flowers and Pascarella (1999) in a study that was limited to a relatively small sample of African-American students. With controls for such factors as precollege critical thinking, demographic characteristics, academic motivation, full- or part-time enrollment, patterns of coursework taken, and the academic selectivity and racial composition of the institution attended, living on-campus had a small, positive impact on African-American students’ critical thinking skills at the end of three years of college. This finding is based on a single study, however, and has yet to be replicated.
Overall, we found little consistent evidence that living on-campus (versus off-campus and commuting to college) directly influences general cognitive growth during college - at least in terms of critical thinking. However, consistent with our conclusions in Chapter 3, we suspect that it probably does exert an indirect positive influence on general cognitive growth and intellectual development by virtue of its enhancement of student academic and social/interpersonal engagement (Ballou, Reavill, & Schultz, 1995). Once again, however, we found no empirical test of this hypothesis.
Conditional Effects of College
In our 1991 synthesis we found little consistent evidence to indicate that the effects of college on general cognitive skills differed for different kinds of students. Similarly, little evidence existed to suggest that the characteristics of the college attended had a differential influence on general cognitive skills for different kinds of students. A small body of evidence suggested that learning cycle-inquiry instruction had its most pronounced influence on the development of Piagetian formal reasoning either for students who are initially at the concrete reasoning stage or for students grouped with others at the same initial Piagetian reasoning level.
Evidence from the 1990s
We uncovered substantially more research in the 1990s that estimates the conditional effects of college on general cognitive skills and intellectual development. As with the vast majority of research on conditional effects, however, nearly all of the findings are based on single samples, with little or no evidence of independent replication. This caveat having been stated, there is evidence of conditional effects that cluster into four general types: 1) student x net effects of college, 2) student x between-college effects, 3) student x within-college effects, and 4) instructional approaches.
Student x Net Effects of College
In contrast to our 1991 synthesis, evidence from the 1990s suggests that the net effects of college on critical thinking skills may vary in magnitude by student race and gender. All of this evidence comes from analyses of the National Study of Student Learning data. For example, with statistical controls in place for such factors as precollege critical thinking level, academic motivation, socioeconomic status, sex, institutional selectivity, total credit hours completed, study effort, patterns of coursework taken, work responsibilities, and extensive measures of academic and social involvement, African-American students made significantly smaller gains in critical thinking through the first- and third-years of college than did Euro-American (White) students (Flowers, 2000; Terenzini, Springer et al., 1994). The net disadvantage in critical thinking for African-American students was modest, about .16 of a standard deviation (6 percentile points) in the first year of college and about .18 of a standard deviation (7 percentile points) after three years of college (2000). Similarly, compared to their White counterparts, Latino students derived significantly smaller critical thinking benefits during the first year of college (Terenzini, Springer et al., 1994). The net disadvantage accruing to Latino students, relative to White students, was about .15 of a standard deviation (6 percentile points).
The conditional effects based on sex present a somewhat more complex picture in that there is evidence that the effects vary in direction at community colleges versus four-year colleges. Controlling for essentially the same variables as Flowers (2000), but with the addition of race, Whitt et al. (in press) found that in community colleges women had a statistically significant disadvantage in first-year critical thinking gains (relative to men) of about .16 of a standard deviation (6 percentile points). Conversely, while there was no significant conditional effect after one year, at the end of three years of college women in four-year institutions demonstrated a statistically significant advantage in critical thinking gains over men of .20 of a standard deviation (8 percentile points).
As with our 1991 synthesis we found little to suggest that institutional characteristics such as selectivity, size, or private versus public control had a differential impact on dimensions of general cognitive skills or intellectual development for different kinds of students (see, for example, Flowers, 2000; Whitt et al., in press). The one possible exception to this is reported in analyses of the community college sample of the National Study of Student Learning data by Hagedorn, Pascarella, Edison, Braxton, Nora, and Terenzini (1999). They found that the average precollege critical thinking level of students at a community college had a compensatory effect on individual students’ first-year gains in critical thinking skills. That is, the benefit of attending a community college where one’s peers had a relatively advanced level of critical thinking was somewhat more pronounced for a student who started postsecondary education with a relatively low level of critical thinking skills. For his or her counterpart who entered a community college with a relatively high level of critical thinking, the average critical thinking level of one’s peers was less important. This finding, however, is based on a sample of only five community colleges. Thus, it is suggestive at best. The same compensatory conditional effect was not uncovered in the NSSL four-year college sample.
Other student x between college conditional effects are likewise based on somewhat isolated findings, and they are not always easily explainable. For example, in analyses of the four-year college sample from the National Study of Student Learning data, Whitt et al. (in press) found that the average socioeconomic status of students at the institution attended was essentially unrelated to third-year critical thinking for men but had a significant negative impact for women. Other analyses of the NSSL data by Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) have suggested that, while an institution’s environmental emphasis on being critical, evaluative, and analytical has a positive net impact on first-year critical thinking for all students, the impact is most pronounced for Latino students. On the other hand, elements of “covert discrimination” in the college environment appear to adversely affect the growth of critical thinking skills in Latino students more than in other students. (Examples of covert discrimination were “people feeling uncomfortable around me” and “concerns/thinking about safety on campus.”)
There is also evidence to suggest that the impact of the institutional environment on self-reported gains in intellectual development and problem-solving skills may vary in magnitude for older versus traditional-age college students. Graham (1998) analyzed data from over 20,000 students at 154 colleges and universities throughout the country to determine the effects of “educational ethos” in the campus environment. Educational ethos in the environment included items measuring such things as faculty respect for students, the quality of instruction, the availability of faculty, informal contact with faculty, and sense of belonging on campus. The educational ethos measure was positively linked to self-reported gains in both intellectual development and problem-solving skills, but the impact was significantly stronger for students age 27 or older than it was for traditional-age students between 18 and 22. However, as Graham was not able to control for student background traits that might have predicted both self-reported gains and perceptions of educational ethos, this conditional effect may be a proxy for other influences not specified in his analytic model.
We uncovered a substantial number of student x within-college conditional effects. Most are based on findings from the National Study of Student Learning data (Pascarella, Edison, Nora, Hagedorn, & Terenzini, 1996b; Pascarella, Palmer, Moye, & Pierson, 2001; Terenzini, Springer et al., 1994; Whitt et al., in press). In these studies, a standardized measure of critical thinking skills is the dependent variable, and the studies each introduce statistical controls for an extensive array of potential confounding variables such as precollege critical thinking, academic motivation, student demographic characteristics, the academic selectivity of the institution attended, full- or part-time enrollment, work responsibilities, patterns of coursework taken, and other measures of academic and social involvement in college. A second sample of students from two institutions investigated growth on the Discrimination and Endorsement scales of the Reasoning About Current Issues Test (RCI). The RCI is an objective, paper and pencil measure of reflective thinking or judgment (Kitchener et al., 2000).
The significant student x within-college conditional effects uncovered from this series of studies include the following: 1) the number of courses taken in the natural sciences and engineering had a stronger positive influence on first-year critical thinking for White (versus African-American) students and for women (versus men) (Terenzini, Springer et al., 1994; Whitt et al., in press); 2) the number of courses taken in the humanities and fine arts had a stronger positive effect on first-year critical thinking for men than for women (Terenzini, Springer et al., 1994); 3) fraternity membership had a significant negative influence on the first-year critical thinking of White men but a modest positive influence for non-White men (e.g., African-American, Asian-American, Latino) (Pascarella, Edison, Whitt et al., 1996); 4) Greek-affiliation had a significantly stronger negative impact on first-year critical thinking gains for men than for women (Whitt et al., in press); 5) studying with peers had a net positive effect on first-year critical thinking for African-American students, but a negative effect for White students (Terenzini, Springer et al., 1994); 6) engaging in volunteer work had a significantly stronger positive effect on both first- and third-year critical thinking skills for men than for women (Whitt et al., in press); 7) full-time enrollment in college had a significantly stronger positive influence on first-year critical thinking for first-generation college students than for students whose parents attended college (Terenzini, Springer, Yaeger, Pascarella, & Nora, 1996); 8) involvement in diversity activities (e.g., attending a racial/cultural awareness workshop, making friends with someone of a different race) had significantly stronger effects on first- and third-year critical thinking for White than for non-White students; and 9) working off-campus was negatively related to growth in reflective thinking for men, but working on-campus, or cumulative hours worked either on- or off-campus was positively related to growth in reflective thinking for women (Kitchener et al., 2000).
There is also evidence from analyses by Whitt et al. (in press) to suggest that men in four-year colleges generally convert various dimensions of coursework and academic and social engagement into significantly stronger impacts on three-year gains in critical thinking than do women. For men, such factors as cumulative credit hours completed, number of social science and mathematics courses taken, computer use, social interactions with peers, and extent of involvement in volunteer work had significantly stronger positive impacts on gains in critical thinking through the third year of college than they did for women. On the other hand, living on-campus (versus living off-campus and commuting to college) had a stronger positive impact on end-of-third year critical thinking skills for women than it did for men.
Earlier in this book we reviewed evidence to suggest that cooperative learning may provide a general advantage over traditional, individualistic learning approaches in such areas as knowledge acquisition and the development of problem-solving skills. There is also evidence, however, to suggest that the positive effects of cooperative learning may be more pronounced for complex, versus less complex, levels of cognitive functioning. A well-conducted quasi-experiment by Garside (1996) randomly assigned intact classes in an introductory interpersonal communication course to two instructional approaches: lecture and small group cooperative discussion. Both conditions were taught by instructors with equal teaching experience. At the beginning and end of the experiment, students took a test of course content that assessed cognitive functioning at both lower-levels (knowledge, comprehension, and/or application) and higher-levels (analysis, synthesis, and/or evaluation) of skill development. The lecture condition produced gains in knowledge at the lower-level of cognitive functioning that were about 3 times as large as the gains for the small-group cooperative discussion condition. Conversely, the small-group cooperative discussion condition produced gains in knowledge at the higher-level of cognitive functioning that were about 1.7 times as large as the gains for the lecture condition.
Long-Term Effects of College
Conclusions from How College Affects
Students
Although the evidence (from national samples) on long-term effects of college was based largely on alumni self-reports, its clear weight indicates that the undergraduate experience is regarded as having had an important influence on general cognitive development and thinking skills. These perceptions of alumni did not appear to be a function of time elapsed since graduation. There was a slight trend for alumni from academically selective institutions to report greater impact than alumni from less selective institutions. However, the nature of the evidence made differential student recruitment by selective and less selective colleges as plausible an explanation for these findings as any differential socialization that might take place in colleges of varying selectivity.
There was also evidence to suggest that intellectually and socially stimulating work environments made an independent contribution to cognitive development in adult life. We suggested that a major, though indirect, long-term effect of college on cognitive development comes through its power to position graduates in intellectually demanding and socially stimulating employment.
Evidence from the 1990s
We uncovered only a small body of research that speaks, either directly or indirectly, to the long-term effects of college on general cognitive skills and intellectual development. Alumni self-reports, while not based on nationally representative samples, tend to reinforce the conclusions from our 1991 synthesis that the undergraduate experience has a substantial impact on general cognitive development and thinking skills (e.g., Kelley, 1994; Pettit, 1992). Moreover, there is also evidence to suggest that the undergraduate experience provides cognitive skills that increase one’s capacity for life-long learning and continuing intellectual growth. Analyzing data from the self-reports of over 17,000 seniors at 106 four-year institutions, Hayek and Kuh (1999) focused on the acquisition of skills during college that conceivably provide individuals with a repertoire of life-long learning competencies (e.g., learning on one’s own, analytical skills, synthesizing information, using computers). Out of four possible responses, “very little,” “some,” “quite a bit,” and “very much,” seniors in the sample indicated that during college they had made, on average, “quite a bit” of progress across all items constituting the scale of life-long learning competencies.
Consistent with the findings of Hayek and Kuh (1999) that students tend to acquire life-long learning skills during college, there is also evidence to suggest that the trends in intellectual growth shaped by the undergraduate experience tend to continue on the same trajectory after college. Perhaps the most comprehensive work in this area has been conducted by Mentkowski and Associates (2000) with samples of women from a small, single-sex, liberal arts college with a strong assessment tradition. A number of different measures of general intellectual development were administered to samples of women when they entered the institution, periodically during college, and five and a half years subsequent to graduation. The measures included the Measure of Intellectual Development (MID), which assesses epistemological development on the Perry (1970) scheme; the Test of Cognitive Development (TCD), which measures the development of abstract or formal operations reasoning; and the inference, recognition of assumptions, and deductions subscales of the Watson Glaser Critical Thinking Appraisal (WGCTA). The results with respect to growth during college on these measures is summarized in our previous (1991) synthesis, but Mentkowski and Associates (2000) report additional information on alumni performance. What is clear from their results is that general cognitive or intellectual development did not cease after college graduation. Across all these measures the estimated average gain occurring during the five years after college (.26 of a standard deviation) was nearly 80% as large as the estimated average gain occurring during college (.33 of a standard deviation). Thus, the intellectual growth put in motion during college appeared to continue for alumni subsequent to graduation.
Additional analyses of what appears to be the same data employed in the Mentkowski and Associates (2000) study, however, suggest some complexities in the overall pattern of intellectual growth during and subsequent to college. Hart, Richards, and Mentkowski (1995) examined periodic changes in essay scores on the Measure of Intellectual Development. Their analyses of the overall sample indicated a typical pattern of growth during college from the transition position between dualism and multiplicity to the transition between multiplicity and relativism. Five years after college, the alumnae as a group were well past multiplicity and into relativism. This general pattern, however, masked several different patterns of progress, and a small percentage of the women showed continuous decline over the period of the study. Such a finding suggests that, while the undergraduate experience may set in motion a general trajectory of intellectual growth, continual growth after college may depend in large measure on the degree of intellectual stimulation and challenge characterizing the post-college lives of alumni. We suspect that this synergistic combination of what happens during college and what happens in one’s post-college life underlies the finding by Glenn and Eklund (1991) that retired college professors have substantially higher levels of reflective thinking than do high school graduates of essentially the same age and tested intelligence. Similarly, Menkowski and Associates report that breadth of involvement after graduation in 12 areas (e.g., cultural activities, volunteer work, reading, discussing current issues and events, continuing education) had a significant, positive effect on five-year alumna scores on a composite measure of intellectual/epistemological and ego development. This effect persisted even in the presence of statistical controls for scores on the same composite measure at the time of graduation, level of critical thinking at graduation, and a measure of curricular preparation during college.
As we saw in the preceding chapter, there was replicated evidence to suggest an intergenerational impact of parental exposure to postsecondary education on sons’ and daughters’ learning in such areas as mathematics, science, and reading comprehension. Evidence pertaining to a parallel intergenerational impact on general cognitive skills is less compelling. In a single- institution pilot study for the National Study of Student Learning, Terenzini, Springer, Pascarella, and Nora (1995) found parents’ education had a significant positive impact on first-year gains on a standardized measure of critical thinking; and this effect persisted even in the presence of controls for precollege critical thinking, family income, race, sex, educational aspirations, measures of class and non-class experiences, and patterns of coursework taken. However, in analyses of the full, 23-institution sample from the National Study of Student Learning, Terenzini and colleagues (Terenzini et al., 1996) reported that, when controls were made for important confounding variables, the first-year gains in critical thinking made by students whose parents had attended college were essentially the same as those made by students whose parents had not attended college.
Summary
Change During College
Compared to the literature base we reviewed for our 1991 synthesis, we found relatively few studies on change or gains in general cognitive skills and intellectual growth during college. The literature we did uncover focused on the areas of critical thinking and postformal reasoning. Overall, our best estimate is that seniors had a critical thinking skills advantage over incoming students of about .50 of a standard deviation (19 percentile points). This number is substantially less than our 1991 estimate of 1 standard deviation (34 percentile points). The reason for this difference is unclear and may represent sample or instrument differences. However, the evidence of more modest change in the 1990s does underscore an important point. On an absolute standard, not all college graduates are proficient critical thinkers.
In addition to critical thinking skills, a new line of inquiry in the 1990s has estimated the extent to which students develop during college in the disposition to think critically. Across the studies we uncovered, the senior advantage over first-year students in critical thinking disposition was about .50 of a standard deviation (19 percentile points).
In the area of postformal reasoning, our conclusion is quite similar to that of our 1991 synthesis. Our best estimate is that seniors have an advantage over freshmen of about .90 of a standard deviation (22 percentile points) in reflective thinking, or the ability to use reason and evidence to address ill-structured problems, and about 2 standard deviations (48 percentile points) in epistemological sophistication or maturity.
Net Effects of College
Research on the net effects of college on general cognitive skills and intellectual growth focused largely on the areas of critical thinking and postformal reasoning. Consistent with our 1991 synthesis, the evidence suggests that a statistically significant part of the gain during college noted in both critical thinking and postformal reasoning is uniquely attributable to exposure to postsecondary education. Our best estimate is that the first three years of college provide an improvement in critical thinking skills of about .55 of a standard deviation or 20 percentile points. Since this estimate of the net effect of three years of college is actually larger than our previous estimate of simple change or growth during four years of college, we suspect that the latter estimate is overly conservative. Only one of the four studies estimating the net effect of college on postformal reasoning provided sufficient information to compute an effect size. In that study, seniors had an advantage in reflective thinking skills over freshmen of about .9 of standard deviation (22 percentile points). This rather tentative estimate of the net effect of college suggests that most, if not all, of the observed gain during college in reflective thinking is uniquely attributable to the college experience itself. At the same time, it should be noted that the research estimating the net effects of college on postformal reasoning, including reflective thinking, may be confounded by student age.
Between-College Effects
With the possible exception of research on institutional environments, we found little to suggest substantial or consistent between-college effects on students’ general cognitive development. In our 1991 synthesis, we concluded that student-body selectivity in and of itself may tell us little about institutional influences on general cognitive skills and intellectual growth. The clear weight of evidence from our present synthesis would support this conclusion once again. (Although there is some evidence from a single sample to suggest a positive effect of student-body selectivity on critical thinking after one year of college, this effect essentially disappeared by the end of the third year of college.) Similarly, little evidence indicated that single-sex institutions had a stronger impact on women’s general cognitive growth than coeducational institutions, and there was essential parity in the first-year critical thinking gains made by students attending community colleges and those attending four-year institutions. Some evidence suggests that institutional size may inhibit critical thinking skills, but this is based on a single-sample finding and awaits replication.
In our 1991 synthesis, we concluded that there was evidence from a single sample to suggest that African-American students demonstrate greater development in critical thinking and concept attainment at historically black institutions than at predominantly white ones. This is partially supported by evidence uncovered in our present synthesis. When standardized measures were the dependent variable, African-American students attending historically black colleges had gains in critical thinking that were indistinguishable from those demonstrated by their African-American counterparts enrolled in predominantly white colleges. However, when the dependent variable was self-reported gains in critical thinking or analytical skills, African-American students reported significantly greater progress when they attended historically black colleges.
The area of institutional environments provided the most consistent evidence we uncovered with respect to between-college effects on students’ general cognitive growth. There is replicated evidence to indicate that an institutional environment with a scholarly, esthetic, and critical/analytical emphasis has modest, positive net effects on student growth in critical thinking and analytical competencies. Moreover, this impact appears to occur independently of institutional selectivity. There is also replicated evidence to suggest that student critical thinking, analytical competencies, and general intellectual development are enhanced by an institutional environment that stresses close relationships and frequent interaction between faculty and students, and faculty concern about student growth and development during college. This environmental emphasis appeared to have an influence on general cognitive growth that operated independently of an institution’s scholarly environment.
Within-College Effects
As with the evidence pertaining to within-college effects on the acquisition of verbal, quantitative, and subject matter competence, the corresponding evidence from the 1990s on more general cognitive skills and intellectual growth is extensive. The following conclusions appear to be warranted:
1. Consistent with our 1991 conclusion, we found little consistent evidence to suggest that one’s major field of study, in and of itself, leads to different effects on general measures of critical thinking. We did find evidence that the intellectual training in different fields of study leads to the development of different reasoning skills, which is also consistent with our 1991 conclusion that academic major has a selective impact on cognitive growth.
2. There is replicated evidence that exposure to natural science courses may positively influence growth in critical thinking skills, but evidence with respect to other concentrations of coursework is either inconclusive or awaits replication.
3. In our 1991 synthesis, we concluded that critical thinking was enhanced by curricular experiences that require integration of ideas and themes across courses and disciplines. Consistent with this conclusion, we found replicated quasi-experimental evidence in our present synthesis that an interdisciplinary or integrated core curriculum that emphasized making explicit connections across courses, and among ideas and disciplines, positively influenced growth in measures of postformal reasoning.
4. Replicated experimental or quasi-experimental evidence suggests that learning a computer program language provided an advantage of .35 of a standard deviation (14 percentile points) in general cognitive skills such as planning, reasoning, and metacognition. Consistent correlational evidence also suggests that critical thinking and general reasoning skills may be enhanced by: 1) coursework requiring students to learn the use of computers; 2) using computers in such learning activities as analyzing data, making visual displays, and searching the internet for course material; and 3) using electronic mail to practice argumentation.
5. Replicated experimental and quasi-experimental evidence exists to suggest that students learning in cooperative groups gain an average advantage in problem-solving skills of .47 of a standard deviation (18 percentile points) over their counterparts not learning in a cooperative format. The positive effect of cooperative learning, however, was larger in the solution of nonlinguistic problems (.96 of a standard deviation) than in the solution of linguistic problems (.31 of a standard deviation).
6. Evidence suggests that critical thinking can be taught, although the average effect is based on a rough estimate and is quite modest in magnitude. Students who receive purposeful instruction and practice in critical thinking and/or problem solving skills appear, on average, to gain an advantage in critical thinking skills of .23 of a standard deviation (9 percentile points) over students not receiving such instruction. We hypothesize that the small effect is due in large part to the general absence of a consensus across studies with respect to the operational definition of what constitutes instruction in critical thinking.
7. Results of a small number of experimental or quasi-experimental studies suggest that postformal reasoning can be enhanced by three loosely-related innovative instructional approaches: reflective judgment-developmental instruction, active learning/team problem-solving instruction, and deliberate psychological instruction. We estimate that, on average, students exposed to these instructional approaches gain an advantage in postformal reasoning of about .65 of a standard deviation (24 percentile points) over students not receiving this type of instruction. This estimated effect size, however, is based on only three studies and may not be particularly robust.
8. We uncovered very little evidence pertaining to the impact of teaching behaviors on students’ general cognitive development. Though it awaits replication and experimental validation, there is correlational evidence, paired with extensive statistical controls, to suggest that extent of teacher organization and preparation in the overall instruction received at an institution positively influences growth in critical thinking skills through the first- and third-years of college.
9. Not surprisingly, quite apart from courses taken and instruction received, a student’s level of academic effort and involvement has an important net influence on his or her growth in general cognitive and intellectual development during college. This conclusion is quite consistent with that from our 1991 synthesis. Level of individual effort or engagement in such areas as hours studied per week, number of non-assigned books read, writing experiences, library use, and course learning activities appears to have unique positive impacts on standardized measures of critical thinking and reflective thinking, as well as on self-reported gains in critical thinking/intellectual development.
10. Also generally consistent with our 1991 synthesis, we uncovered a substantial body of evidence indicating that the nature of students' social and cocurricular involvement plays a unique role of some consequence in his or her general cognitive development during college. Interactions with peers that 1) extend and reinforce broad ideas introduced in one's academic experience, and 2) confront the individual with diverse interests, values, political beliefs, and cultural norms, appear the most salient in terms of positive impact on critical thinking, analytical skills, and postformal reasoning. Somewhat less extensive evidence also suggests that cocurricular activities such as involvement in clubs and organizations may foster critical thinking.
11. The weight of evidence suggests that student-faculty nonclassroom interactions that 1) tend to reinforce or extend the intellectual ethos of the classroom or formal academic experience, or 2) focus on issues of student development can have positive effects on dimensions of general cognitive development such as postformal reasoning, analytical ability, and critical thinking skills.
12. In addition to their positive effects on the acquisition of subject matter knowledge, there is also consistent evidence to suggest that involvement in diversity experiences and service learning experiences has a unique positive impact on dimensions of general cognitive development such as critical thinking, analytical competencies, and thinking complexity. The most salient diversity experiences appear to be informal interactions with racially and culturally diverse peers and involvement in more formal programs such as racial/cultural awareness workshops and coursework focusing on social/cultural diversity and intergroup relations. The most effective service learning experiences appear to be those that integrate service experiences with course content, provide for reflection about the service experience, and permit one to apply subject matter learning to the service experience, and vice versa.
13. Though it is based on single-sample findings that await replication, there is evidence to suggest that participation by men in revenue-producing intercollegiate sports (i.e., football and basketball) has a negative impact on the development of critical thinking during college. Similarly, joining a fraternity may inhibit critical thinking growth in first-year college men, although this negative impact appears to diminish in magnitude after the first year of college.
14. Given the general inconsistency of results across studies, we conclude that there is little compelling evidence to suggest on- or off-campus work in general has more than a trivial impact on cognitive or intellectual development during college. Similarly, we found little consistent evidence to suggest that living on-campus (versus off-campus and commuting to college) directly influences general cognitive growth during college—at least in terms of critical thinking. Consistent with our conclusions in Chapter 3, however, we suspect that it probably does exert an indirect positive influence on general cognitive growth and intellectual development through its enhancement of academic and social engagement. Once again, however, we found no empirical test of this hypothesis.
Conditional Effects
Compared to our 1991 synthesis, we uncovered substantially more research in the 1990s that estimates the conditional effects of college on general cognitive skills and intellectual development. As with the vast majority of research on conditional effects, however, nearly all of the findings are based on single samples, with little or no evidence of independent replication. In contrast to our 1991 synthesis, evidence from the 1990s suggests that the net effects of college on critical thinking skills may vary in magnitude by student race and gender. African-American students made smaller gains in critical thinking during the first and third years of college than did White students, while Latino students made smaller gains than White students in the first year. The conditional effects based on sex present a more complex picture in that the effects vary in direction at community colleges and four-year colleges. Women in community colleges appeared to make smaller gains in critical thinking skills during the first year of college than did men, while in four-year institutions women made larger critical thinking gains across three years of college than did men.
As with our 1991 synthesis, we found little consistent evidence to suggest that institutional characteristics such as selectivity, size, or private versus public control had a differential impact on dimensions of general cognitive skills or intellectual development for different kinds of students. There was some evidence, however, that institutional environments may play a stronger role in the growth of critical thinking skills for Latino students than for other students. An environmental emphasis on being critical, evaluative, and analytical had a stronger positive, net impact on growth in first-year critical thinking skills for Latino students than it did for other students, while covert discrimination had a stronger negative, net impact on critical thinking for Latinos than for other students.
There was also substantial evidence to indicate 1) that the effects on first-year growth in critical thinking skills of such factors as course-taking patterns, Greek affiliation, studying with peers, engaging in volunteer work, involvement in diversity activities, and full- versus part-time enrollment differed in magnitude for African-American (or non-White) versus White students, for men versus women, and for first-generation college students versus students whose parents attended college; and 2) that men in four-year colleges generally convert various dimensions of coursework and academic and social involvement into stronger impacts on three-year gains in critical thinking than do women. In the latter case, such factors as cumulative credit hours completed, number of social science and mathematics courses taken, computer use, social interactions with peers, and extent of involvement in volunteer work had stronger positive impacts on gains in critical thinking through the third year of college for men than they did for women. Conversely, living on-campus (versus off-campus and commuting to college) had a stronger positive impact on end-of-third year critical thinking skills for women than for men.
Finally, there is also evidence, based on a single sample, to suggest that the positive effects of cooperative learning may be more pronounced for higher or more complex levels of cognitive functioning (e.g., analysis, synthesis, and/or evaluation) than for lower or less complex levels of cognitive functioning (e.g., knowledge, comprehension, application).
Long-Term Effects of College
A major conclusion from our 1991 synthesis was that the undergraduate experience is regarded by alumni as having had an important influence on general cognitive development and thinking skills. Though based on less extensive evidence, the conclusion from our present synthesis would tend to reinforce this. Moreover, there is also evidence to suggest that the undergraduate experience provides cognitive skills that increase one’s capacity for life-long learning and continuing intellectual development. Overall, the trends in intellectual growth shaped by the undergraduate experience tend to continue on the same trajectory after college. However, continued intellectual and cognitive development after graduation also depends to a substantial extent on the degree of intellectual stimulation and challenge in the post-college lives of alumni. Although there was replicated evidence suggesting an intergenerational impact of parental exposure to postsecondary education on son’s and daughter’s knowledge acquisition in mathematics, science, and reading comprehension, the evidence for a similar intergenerational impact on general cognitive skills is not compelling.
Adams, M., & Zhou-McGovern, Y. (1994). The sociomoral development of undergraduates in a "social diversity" course: Developmental theory, research, and instructional applications. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.
Anaya, G. (1992). Cognitive development among college undergraduates. Unpublished doctoral dissertation, University of California, Los Angeles.
Aper, J. (1994). An investigation of the relationship between student work experience and student outcomes. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.
Aper, J. (1997). An investigation of the relationship between student work experience and student outcomes. Unpublished unpublished manuscript, University of Tennessee, Knoxville, TN.
Arnold, J., Kuh, G., Vesper, N., & Schuh, J. (1993). Student age and enrollment status as determinants of learning and personal development at metropolitan institutions. Journal of College Student Development, 34, 11-16.
Astin, A. (1993a). Diversity and multiculturalism on the campus: How are students affected? Change, 25, 44-49.
Astin, A. (1993b). What matters in college? Four critical years revisited. San Francisco: Jossey-Bass.
Astin, A. (1996a). "Involvement in learning" revisited: Lessons we have learned. Journal of College Student Development, 37, 123-134.
Astin, A., & Sax, L. (1996). How undergraduates are affected by service participation. Unpublished manuscript, University of California, Los Angeles.
Avens, C., & Zelley, R. (1992). QUANTA: An interdisciplinary learning community (four studies). Daytona Beach, FL: Daytona Beach Community College. (ERIC Document Reproduction Service No. ED 349 073).
Ballou, R., Reavill, L., & Schultz, B. (1995). Assessing the immediate and residual effects of the residence hall experience: Validating Pace's 1990 analysis of on-campus and off-campus students. Journal of College and University Student Housing, 25, 16-21.
Basseches, M. (1984). Dialectic thinking and adult development. Norwood, N.J.: Abelex.
Batchelder, T., & Root, S. (1994). Effects of an undergraduate program to integrate academic learning and service: Cognitive, prosocial cognitive, and identity outcomes. Journal of Adolescence, 17, 341-355.
Bauer, K. (1992). Self-reported gains in academic and social skills. Journal of College Student Development, 33, 492-498.
Bauer, K. (1995). Freshman to senior year gains reported on the college student experiences questionnaire. NASPA Journal, 32, 130-137.
Bauer, K. (1996). Year two of the longitudinal study of university undergraduates: Changes occurring from the freshman to sophomore year. Paper presented at the annual meeting of the Association for Institutional Research, Albuquerque, NM.
Bauer, K. (1998). Academic and social development of undergraduate students: Summary of findings for the UD longitudinal study, fall 1993-spring 1997. Paper presented at the annual forum of the Association for Institutional Research, Minneapolis, MN.
Bauer, K., Mitchell, F., & Bauer, P. (1991). Students' perceptions of selected academic and personal characteristics acquired at community colleges. College and University, 67, 65-71.
Baxter Magolda, M. (1990a). Gender differences in epistemological development. Journal of College Student Development, 31, 555-561.
Baxter Magolda, M. (1992a). Cocurricular influences on college students' intellectual development. Journal of College Student Development, 33, 203-213.
Baxter Magolda, M. (1992b). Knowing and reasoning in college: Gender-related patterns in students' intellectual development. San Francisco: Jossey-Bass.
Beck, S., Bennett, A., McLeod, R., & Molyneaux, D. (1992). Review of research on critical thinking in nursing education. Review of Research in Nursing Education, 5, 1-30.
Beckett, R. (1996). Critical thinking and self-efficacy in autodidactic learning: The effects of program type, self-esteem, and program characteristics. Dissertation Abstracts International, 16, 2551A.
Bers, T., McGowan, M., & Rubin, A. (1996). The disposition to think critically among community college students: The California Critical Thinking Disposition Inventory. Journal of General Education, 45, 197-223.
Bohr, L., Pascarella, E., Nora, A., Zusman, B., Jacobs, M., Desler, M., et al. (1994). Cognitive effects of 2-year and 4-year colleges: A preliminary study. Community College Review, 22, 4-11.
Brooks, K., & Shepard, J. (1990). The relationship between clinical decision making skills in nursing and general critical thinking abilities of senior nursing students in four types of nursing programs. Journal of Nursing Education, 2, 391-399.
Cabrera, A., Nora, A., Bernal, E., Terenzini, P., & Pascarella, E. (1998). Collaborative learning: Preferences, gains in cognitive and affective outcomes, and openness to diversity among college students. Paper presented at the annual meeting of the Association for the Study of Higher Education, Miami, FL.
Collins, A., White, B., & O'Brien, T. (1992). The relationship of cognitive style and selected characteristics of vocational education teachers. College Student Journal, 26, 167-173.
Criner, L. (1992). Teaching thinking and reasoning: A study of critical thinking in adults. Dissertation Abstracts International, 53, 2643A.
Dale, P., Ballotti, D., Handa, S., & Zych, T. (1997). An approach to teaching problem solving in the classroom. College Student Journal, 31, 76-79.
DeSousa, D., & Kuh, G. (1996). Does institutional racial composition make a difference in what black students gain from college? Journal of College Student Development, 37, 257-267.
Dey, E. (1991). Community service and critical thinking: An exploratory analysis of collegiate influences. Paper presented at the conference on Setting the Agenda for an Effective Research Strategy for Combining Service and Learning in the 1990s, Wingspread Conference Center, Racine, WI.
Dollar, R. (1991). College influences on analytical thinking and communication skills of part-time versus full-time students. College Student Journal, 25, 273-279.
Doyle, S., Edison, M., & Pascarella, E. (1998). The "seven principles of good practice in undergraduate education" as process indicators of cognitive development in college: A longitudinal study. Paper presented at the annual meeting of the Association for the Study of Higher Education, Miami, FL.
Drouin, L. (1992). An investigation of the critical thinking ability of engineering students seeking a bachelor of science degree. Dissertation Abstracts International, 53, 1070A.
Durham, R., Hays, J., & Martinez, R. (1994). Socio-cognitive development among Chicano and Anglo American college students. Journal of College Student Development, 35, 178-182.
Edison, M., Doyle, S., & Pascarella, E. (1998). Dimensions of teaching effectiveness and their impact on student cognitive development. Paper presented at the annual meeting of the Association for the Study of Higher Education, Miami, FL.
Ennis, R. (1985). A logical basis for measuring critical thinking skills. Educational Leadership, 43, 44-48.
Erwin, T. (1997). Definitions and assessment methods for critical thinking, problem solving, and writing. Harrisonburg, VA: James Madison University, Center for Assessment and Research Studies.
Eyler, J., & Giles, D. (1999). Where's the learning in service learning? San Francisco: Jossey-Bass.
Eyler, J., Giles, D., Lynch, C., & Gray, C. (1997). Service-learning and the development of reflective judgment. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Eyler, J., Giles, D., Root, S., & Price, J. (1997). Service-learning and the development of expert citizens. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Facione, N. (1997). Critical thinking assessment in nursing education programs: An aggregate data analysis. Millbrae, CA: The California Academic Press.
Facione, P. (1990a). The California Critical Thinking Skills Test-college level technical report 1: Experimental validation and content validity. Millbrae, CA: California Academic Press.
Facione, P. (1990b). The California Critical Thinking Skills Test-college level technical report 2: Factors predictive of critical thinking skills. Millbrae, CA: California Academic Press.
Facione, P. (1991). Testing college level critical thinking skills. Paper presented at the 11th Annual Conference on Critical Thinking and Educational Reform, Sonoma, CA.
Facione, P., & Facione, N. (1992). Test manual: The California Critical Thinking Dispositions Inventory. Millbrae, CA.
Facione, P., Facione, N., Blohm, S., Howard, K., & Giancarlo, C. (1998). The California Critical Thinking Skills Test: Test manual. Millbrae, CA: California Academic Press.
Facione, P., Facione, N., & Giancarlo, C. (1994). Critical thinking disposition as a measure of competent clinical judgment: The development of the California Critical Thinking Disposition Inventory. Journal of Nursing Education, 33, 345-350.
Facione, P., Facione, N., & Giancarlo, C. (1996). The motivation to think in working and learning. In New Directions for Higher Education (Vol. 96, pp. 67-79). San Francisco: Jossey-Bass.
Facione, P., Sanchez, C., Facione, N., & Gainen, J. (1995). The disposition toward critical thinking. Journal of General Education, 44, 1-25.
Flowers, L. (2000). Cognitive effects of college: Differences between African-American and Caucasion students. Unpublished doctoral dissertation, University of Iowa, Iowa City.
Flowers, L. (2002). The impact of college racial composition on African American students' academic and social gains: Additional evidence. Journal of College Student Development, 43, 403-410.
Flowers, L., & Pascarella, E. (1999). Cognitive effects of college racial composition on African-American students after three years of college. Journal of College Student Development, 40.
Flowers, L., Pascarella, E., & Pierson, C. (1999). Information technology use and cognitive outcomes in the first year of college. Unpublished manuscript, University of Iowa, Iowa City.
Forbes, C. (1997). Analyzing the growth of the critical thinking skills of college calculus students. Dissertation Abstracts International, 58, 2101A.
Franklin, M. (1993). The effects of differential college environments on academic learning and student perceptions of cognitive development. Unpublished doctoral dissertation, University of Nebraska, Lincoln.
Franklin, M. (1995). The effects of differential college environments on academic learning and student perceptions of cognitive development. Research in Higher Education, 36, 127-153.
Frost, S. (1991). Fostering the critical thinking of college women through academic advising and faculty contact. Journal of College Student Development, 32, 359-366.
Gadzella, B., Ginther, D., & Bryant, G. (1996). Teaching and learning critical thinking skills. Paper presented at the 26th International Congress of Psychology, Montreal, Canada.
Gadzella, B., Hartsoe, K., & Harper, J. (1989). Critical thinking and mental ability groups. Psychological Reports, 65, 1019-1026.
Gadzella, B., & Masten, W. (1998). Critical thinking and learning processes for students in two major fields. Journal of Instructional Psychology, 25, 256-261.
Garside, C. (1996). Look who's talking: A comparison of lecture and group discussion teaching strategies in developing critical thinking skills. Communication Education, 45, 212-227.
Giancarlo, C., & Facione, P. (1997). A longitudinal study of Santa Clara undergraduate students' disposition toward critical thinking. Unpublished manuscript, Santa Clara University, Santa Clara, CA.
Glenn, D., & Eklund, S. (1991). The relationship of graduate education and reflective judgment in older adults. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Graham, S. (1998). Adult growth in college: The effects of age and educational ethos. Journal of College Student Development, 39, 239-250.
Graham, S., & Cockriel, I. (1989). College outcome assessment factors: An empirical approach. College Student Journal, 23, 280-286.
Graham, S., & Gisi, S. (1999). The effects of instructional climate and student affairs on college student outcomes. Paper presented at the annual meeting of the Association for the Study of Higher Education, San Antonio, TX.
Graham, S., & Long, S. (1998). The role of college involvement for adult undergraduate students. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
Grayson, J. (1995). Does race matter? Outcomes of the first year experience in a Canadian University. Canadian Journal of Higher Education, 25(2), 79-109.
Grayson, J. (1996). Value added in generic skills between first and final year: A pilot project (Research report). Toronto, Canada: York University, Institute for Social Research.
Gunn, C. (1993). Assessing critical thinking: Development of a constructed response sheet. Dissertation Abstracts International, 54, 2267B.
Gurin, P. (1999). Expert report of Patricia Gurin. Available at: http://www.umich.edu/~newsinfo/admission/expert/gurinapb.html.
Hagedorn, L., Pascarella, E., Edison, M., Braxton, J., Nora, A., & Terenzini, P. (1999). Does institutional context influence the development of critical thinking? A research note. Review of Higher Education, 22, 247-263.
Halpern, D. (1993). Assessing the effectiveness of critical thinking instruction. Journal of General Education, 42, 238-254.
Hart, J., Rickards, W., & Mentkowski, M. (1995). Epistemological development during and after college: Longitudinal growth on the Perry scheme. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
Hayek, J., & Kuh, G. (1999). College activities and environmental factors associated with the development of life-long learning competencies of college seniors. Paper presented at the annual meeting of the Association for the Study of Higher Education, San Antonio, TX.
Hill, K. (1995). Critical thinking and its relation to academic, personal, and moral development in the college years. Dissertation Abstracts International, 56, 4603B.
Hofer, B. (1994). Epistemological beliefs and first-year college students: Motivation and cognition in different instructional contexts. Paper presented at the annual meeting of the American Psychological Association, Los Angeles.
Hofer, B. (1998-99). Instructional context in the college mathematics classroom: Epistemological beliefs and student motivation. Journal of Staff, Program, & Organization Development, 16, 73-82.
Hurtado, S. (Ed.). (1999). Linking diversity and educational purpose: How the diversity of the faculty and the student body impacts the classroom environment and student development. Cambridge, MA: Harvard Publishing Group.
Inlow, F., & Chovan, W. (1993). Another search for the effects of teaching thinking and problem solving skills on college students' performance. Journal of Instructional Psychology, 20, 215-223.
Inman, P., & Pascarella, E. (1998). The impact of college residence on the development of critical thinking skills in college freshmen. Journal of College Student Development, 39, 557-568.
Jehng, J., Johnson, S., & Anderson, R. (1993). Schooling and students' epistemological beliefs about learning. Contemporary Educational Psychology, 18, 23-35.
Jensen, L., Kitchener, K., & Wood, P. (1999). The role of need for cognition in the development of reflective judgment. Paper presented at the annual meeting of the American Psychological Association, Boston.
Johnson, D., Johnson, R., & Smith, K. (1996). Academic controversy: Enriching college instruction through intellectual conflict. Washington, D.C.: George Washington University, Graduate School of Education and Human Development.
Jones, E. (1993). Critical thinking literature review. University Park: Pennsylvania State University Press.
Jones, E. (1994). Defining important skills for college graduates to achieve. Paper presented at the Sixth International Conference on Thinking, Boston.
Jones, E. (1995). Defining essential critical thinking skills for college graduates. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
Jones, E., Dougherty, B., Fantaske, P., & Hoffman, S. (1997). Identifying college graduates' essential skills in reading and problem solving: Perspectives of faculty, employers, and policy makers. University Park: Pennsylvania State University.
Jones, E., Hoffman, S., Moore, L., Ratcliff, G., Tibbetts, S., & Click, B. (1995). National assessment of college student learning: Identifying college graduates essential skills in writing, speech and listening, and critical thinking (NCES No. No. 95-001). Washington, D.C.: U.S. Government Printing Office.
Karabenick, S., & Collins-Eaglin, J. (1996). Relation of perceived instructional goals and incentives to college students' use of learning strategies. Paper presented at the annual meeting of the American Educational Research Association, New York.
Kaufman, M., & Creamer, D. (1991). Influences of student goals for college on freshman-year quality of effort and growth. Journal of College Student Development, 32, 197-206.
Keeley, S. (1992). Are college students learning the critical thinking skill of finding assumptions? College Student Journal, 26, 316-322.
Kelley, L. (1994). Utilizing a graduate follow-up survey to assess institutional effectiveness in the small liberal arts college. Paper presented at the annual meeting of the Association for Institutional Research, New Orleans.
Kember, D., & Gow, L. (1994). Orientations to teaching and their effect on the quality of student learning. Journal of Higher Education, 65, 58-74.
Kim, M. (1995). Organizational effectiveness of women-only colleges: The impact of college environment on students' intellectual and ethical development. Unpublished doctoral dissertation, University of California, Los Angeles.
Kim, M. (1996). The effectiveness of women-only colleges for intellectual development for women students. Paper presented at the annual meeting of the American Educational Research Association, New York.
Kim, M. (2002a). Cultivating intellectual development: Comparing women-only colleges and coeducational colleges for educational effectiveness. Research in Higher Education, 43, 447-481.
King, A. (1992c). Facilitating elaborative learning through guided student-generated questioning. Educational Psychologist, 27, 111-126.
King, P., & Kitchener, K. (1994). Developing reflective judgment. San Francisco: Jossey-Bass.
King, P., Wood, P., & Mines, R. (1990). Critical thinking among college and graduate students. Review of Higher Education, 13, 167-186.
Kitchener, K., Wood, P., & Jensen, L. (1999). Curricular, co-curricular, and institutional influence on real-world problem-solving. Paper presented at the annual meeting of the American Psychological Association, Boston.
Kitchener, K., Wood, P., & Jensen, L. (2000). Promoting epistemic cognition and complex judgment in college students. Paper presented at the annual meeting of the American Psychological Association, Washington, DC.
Knefelkamp, L. (1974). Developmental instruction: Fostering intellectual and personal growth in college students. Unpublished doctoral dissertation, University of Minnesota, Minneapolis.
Kronholm, M. (1996). The impact of development instruction on reflective judgment. Review of Higher Education, 19, 199-225.
Kube, B., & Thorndike, R. (1991). Cognitive development during college: A longitudinal study measured on the Perry scheme. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Kuh, G. (1993b). In their own words: what students learn outside the classroom. American Educational Research Journal, 30, 277-304.
Kuh, G. (1995). The other curriculum: Out-of-class experiences associates with student learning and personal development. Journal of Higher Education, 66, 123-155.
Kuh, G., Arnold, J., & Vesper, N. (1990). Influence of collegiate environments on student learning. Paper presented at the annual meeting of the Association for the Study of Higher Education, Portland, OR.
Kuh, G., Arnold, J., & Vesper, N. (1991). The influence of student effort, college environments, and campus culture on undergraduate student learning and personal development. Paper presented at the annual meeting of the Association for the Study of Higher Education, Boston.
Kuh, G., & Hu, S. (2000). The effects of computer and information technology on student learning and other college experiences. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
Langer, P., & Chiszar, D. (1993). Assessment of critical thinking courses. Perceptual and Motor Skills, 77, 970.
Lehman, D., & Nisbett, R. (1990). A longitudinal study of the effects of undergraduate training on reasoning. Developmental Psychology, 26, 952-960.
Li, G., Long, S., & Simpson, M. (1998). Self-perceived gains in communication and critical thinking skills: Are there disciplinary differences? Paper presented at the annual meeting of the Association for Institutional Research, Minneapolis, MN.
Li, G., Long, S., & Simpson, M. (1999). Self-perceived gains in critical thinking and communication skills: Are there disciplinary differences? Research in Higher Education, 40, 43-60.
Liao, Y.-K., & Bright, G. (1991). Effects of computer-assisted instructor and computer programming on cognitive outcomes: A meta-analysis. Journal of Educational Computing Research, 7, 251-268.
Logan, C., & Salisbury-Glennon, J. (1999). The effects of a learner-centered undergraduate community of learners on motivation and cognitive learning strategies. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
Lynch, C. (1996). Facilitating and assessing unstructured problem solving. Journal of College Reading and Learning, 27, 16-27.
MacPherson, K. (1999). The development of critical thinking skills in undergraduate supervisory management units: Efficacy of student peer assessment. Assessment and Evaluation in Higher Education, 24, 273-284.
Mansfield, A., & Clinchy, B. (1990). Young women's ways of knowing: A longitudinal study of epistemological development in different domains of knowledge over the college years. Paper presented at the annual meeting of the American Educational Research Association, Boston.
Marr, D. (1995). An investigation of the construct validity of the long form of the Academic Profile. Princeton, NJ: Educational Testing Service.
Marra, R., Palmer, B., & Litzinger, T. (2000). The effects of a first-year engineering design course on student intellectual development as measured by the Perry scheme. Journal of Engineering Education, 89, 39-45.
Marttunen, M. (1997). Electronic mail as a pedagogical delivery system: An analysis of the learning of argumentation. Research in Higher Education, 38, 345-363.
May, D. (1990). Student development: A longitudinal investigation of the relationship between student effort and change in intellectual development. Unpublished doctoral dissertation, Memphis State University, Memphis, TN.
McAdams, C., & Foster, V. (1998). Promoting the development of high-risk college students through a deliberate psychological education-based freshman orientation course. Journal of the Freshman Year Experience, 10, 51-72.
McBride, R., & Reed, J. (1998). Thinking and college athletes-are they predisposed to critical thinking? College Student Journal, 32, 443-450.
McDonough, M. (1997). An assessment of critical thinking at the community college level. Dissertation Abstracts International, 58, 2561A.
Mentkowski, M., & Associates. (2000). Learning that lasts: Interpreting learning, development, and performance in college and beyond. San Francisco: Jossey-Bass.
Mentkowski, M., Rodgers, G., Deemer, D., Tamar, B., Reisetter, J., Rickards, W., et al. (1991). Understanding abilities, learning, and development through college outcome studies: What can we expect from higher education assessment? Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Miller, J., & Groccia, J. (1997). Are four heads better than one? A comparison of cooperation and traditional teaching formats in an introductory biology course. Innovative Higher Education, 21, 253-273.
Mines, R., King, P., Hood, A., & Wood, P. (1990). Stages of intellectual development and associated critical thinking skills in college students. Journal of College Student Development, 31, 538-547.
Money, S. (1997). The relationship between critical thinking scores, achievement scores, and grade point average in three different disciplines. Dissertation Abstracts International, 58, 3401A.
Moore, B., & Parker, R. (1989). Critical thinking: Evaluating claims and arguments in everyday life (2nd ed.). Mountain View, CA: Mayfield Publishing Company.
Moore, W. (1991a). The Perry scheme of intellectual and ethical development: An introduction to the model and major assessment approaches. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Moore, W. (1991b). The measure of intellectual development: An instrument manual. Farmville, VA: Center for the Study of Intellectual Development.
Nickerson, C. (1991). Curricular design and levels of reflective judgment. Dissertation Abstracts International, 52, 2040.
Olsen, S. (1990a). Examining the relationship between college core course areas and sophomore critical thinking test scores. Paper presented at the annual meeting of the American Evaluation Association, Washington, D.C.
Pascarella, E. (1989). The development of critical thinking: Does college make a difference. Journal of College Student Development, 30, 19-26.
Pascarella, E., Bohr, L., Nora, A., & Terenzini, P. (1995a). Cognitive effects of 2-year and 4-year colleges: New evidence. Educational Evaluation and Policy Analysis, 17, 83-96.
Pascarella, E., Bohr, L., Nora, A., & Terenzini, P. (1995b). Intercollegiate athletic participation and freshman-year cognitive outcomes. Journal of Higher Education, 66, 369-387.
Pascarella, E., Bohr, L., Nora, A., & Terenzini, P. (1996). Is differential exposure to college linked to the development of critical thinking? Research in Higher Education, 37, 159-174.
Pascarella, E., Bohr, L., Nora, A., Zusman, B., Inman, P., & Desler, M. (1993). Cognitive impacts of living on campus versus commuting to college. Journal of College Student Development, 34, 216-220.
Pascarella, E., Edison, M., Nora, A., Hagedorn, L., & Braxton, J. (1995). Effects of teacher organization /preparation and teacher skill/clarity on general cognitive skills in college. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
Pascarella, E., Edison, M., Nora, A., Hagedorn, L., & Braxton, J. (1996). Effects of teacher organization/preparation and teacher skill/clarity on general cognitive skills in college. Journal of College Student Development, 37, 7-19.
Pascarella, E., Edison, M., Nora, A., Hagedorn, L., & Terenzini, P. (1996b). Influences on students' openness to diversity and challenge in the first year of college. Journal of Higher Education, 67, 174-195.
Pascarella, E., Edison, M., Nora, A., Hagedorn, L., & Terenzini, P. (1998). Does work inhibit cognitive development during college? Education Evaluation and Policy Analysis, 20, 75-93.
Pascarella, E., Edison, M., Whitt, E., Nora, A., Hagedorn, L., & Terenzini, P. (1996). Cognitive effects of Greek affiliation during the first year of college. NASPA Journal, 33, 242-259.
Pascarella, E., Flowers, L., & Whitt, E. (1999). Cognitive effects of Greek affiliation in college: Addtional evidence. Unpublished manuscript, University of Iowa, Iowa City.
Pascarella, E., Palmer, B., Moye, M., & Pierson, C. (2001). Do diversity experiences influence the development of critical thinking? Journal of College Student Development, 42, 257-271.
Pascarella, E., & Terenzini, P. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass.
Pascarella, E., Truckenmiller, R., Nora, A., Terenzini, P., Edison, M., & Hagedorn, L. (1999). Cognitive impacts of intercollegiate athletic participation: Some further evidence. Journal of Higher Education, 70, 1-26.
Paul, R. (1987). Teaching thinking skills: Theory and practice. In J. Baron & R. Sternberg (Eds.), Teaching thinking skills: Theory and practice (pp. 127-148). New York: Freeman.
Paul, R. (1992). Critical thinking: What every person needs to survive in a rapidly changing world (2nd ed.). Santa Rosa, CA: Foundation for Critical Thinking.
Paulsen, M., & Wells, C. (1998). Domain differences in the epistemological beliefs of college students. Research in Higher Education, 39, 365-384.
Pavelich, M. (1996). Helping student develop higher-level thinking: Use of the Perry model. Paper presented at the Frontiers in Education Conference, Salt Lake City, UT.
Pavelich, M., & Moore, W. (1996). Measuring the effect of experiential education using the Perry model. Journal of Engineering Education, 85, 287-292.
Pavelich, M., Olds, B., & Miller, R. (1995). Real-world problem solving in freshman-sophomore engineering. In New Directions for Teaching and Learning (Vol. 61, pp. 45-54). San Francisco: Jossey-Bass.
Pearson, C. (1991). Barriers to success: Community college students' critical thinking skills. Washington, DC: U.S. Office of Education. (ERIC Document Reproduction Services No. ED 340 415).
Pearson, F., & Rodgers, R. (1998). Cognitive and identity development: Gender effects. Initiatives, 58, 17-33.
Perry, W. (1970). Forms of intellectual and ethical development in the college years. New York: Holt, Rinehart and Winston.
Peterson, M. (1996). A team-based approach to problem-based learning: An evaluation of structured team problem solving. Journal on Excellence in College Teaching, 7, 129-153.
Pettit, J. (1992). Listening to your alumni in assessing learning outcomes. Paper presented at the annual meeting of the Association for Institutional Research, Atlanta, GA.
Prendergast, D. (1998). Influences of college environments and the development of critical thinking skills in college juniors. Unpublished doctoral dissertation, University of Illinois, Chicago.
Price, W., Wilmes, D., & Turmel, M. (1994). General education assessment in introductory psychology. Journal on Excellence in College Teaching, 5, 121-133.
Qin, Z., Johnson, D., & Johnson, R. (1995). Cooperative versus competitive efforts and problem solving. Review of Educational Research, 65, 129-143.
Reiter, S. (1994). Teaching dialogically: Its relationship to critical thinking in college students. In P. Pintrich, D. Brown & C. Weinstein (Eds.), Student motivation, cognition, and learning (pp. 275-310). Hillsdale, NJ: Lawrence Erlbaum Associates.
Rykiel, J. (1995). The community college experience: Is there an effect on critical thinking and moral reasoning? Dissertation Abstracts International, 56, 3824A.
Saucier, B. (1995). Critical thinking skills of baccalaureate nursing students. Journal of Professional Nursing, 11, 351-357.
Schilling, K. (1991). Assessing models of liberal education: An empirical comparison. Washington, D.C.: Fund for the Improvement of Postsecondary Education (ERIC Document Reproduction Service, No. ED 359 864).
Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82, 498-504.
Schommer, M. (1993). Comparisons of beliefs about the nature of knowledge and learning among postsecondary students. Research in Higher Education, 34, 355-370.
Sebrell, K., & Erwin, T. (1998). Assessment of critical thinking: One performance method analyzed. Unpublished paper, James Madison University, Center for Assessment and Research Studies, Harrisonburg, VA.
Sherman, S. (1994, August 22). Leaders learn to heed the voice within. Fortune, 92-100.
Smart, J., Feldman, K., & Ethington, C. (1999). Academic discipline: Holland's theory and the study of college students and faculty. Nashville, TN: Vanderbuilt University Press.
Smith-Sanders, C., & Twale, D. (1997). Impact of number and type of core curriculum hours on critical thinking. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Smith-Sanders, C., & Twale, D. (1998). Further study of the impact of number and type of core curriculum hours on critical thinking. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
Spaulding, S., & Kleiner, K. (1992). The relationship of college and critical thinking: Are critical thinkers attracted or created by college disciplines. College Student Journal, 26, 162-166.
Sternberg, R. (1985). Teaching critical thinking, part 2: Possible solutions. Phi Delta Kappan, 67, 277-280.
Strauss, L., & Volkwein, J. (2001). Comparing student performance and growth in two and four year institutions. Paper presented at the annual forum of the Association for Institutional Research, Long Beach, CA.
Taube, K. (1997). Critical thinking ability and disposition as factors of performance on a written critical thinking test. Journal of General Education, 46, 129-164.
Taylor, M. (1983). The development of the measure of epistemological reflection. Dissertation Abstracts International, 44, 1065A.
Terenzini, P., Rendon, L., Upcraft, M., Millar, S., Allison, K., Gregg, P., et al. (1994). The transition to college: Diverse students, diverse stories. Research in Higher Education, 35, 57-73.
Terenzini, P., Springer, L., Pascarella, E., & Nora, A. (1995). Influences affecting the development of students' critical thinking skills. Research in Higher Education, 36, 23-29.
Terenzini, P., Springer, L., Yaeger, P., Pascarella, E., & Nora, A. (1994). The multiple influences on students' critical thinking skills. Paper presented at the annual meeting of the Association for the Study of Higher Education, Orlando, FL.
Terenzini, P., Springer, L., Yaeger, P., Pascarella, E., & Nora, A. (1996). First-generation college students: Characteristics, experiences, and cognitive development. Research in Higher Education, 37, 1-22.
Thompson, P. (1991). The effect of section size on student performance in a statistics course. College Student Journal, 25, 388-395.
Thompson, S. (1995). Techniques for assessing general education outcomes: The natural science core program at the University of Denver. Dissertation Abstracts International, 56, 2139A.
Tsui, L. (1998a). A review of research on critical thinking. Paper presented at the annual meeting of the Association for the Study of Higher Education, Miami, FL.
Tsui, L. (1998b). Fostering critical thinking in college students: A mixed-methods study of influences inside and outside of the classroom. Unpublished manuscript, University of California, Los Angeles.
Tsui, L. (1999). Courses and instruction affecting critical thinking. Research in Higher Education, 40, 185-200.
Tsui, L. (2000). Effects of campus culture on students' critical thinking. Review of Higher Education, 23, 421-444.
Tsui, L. (2001). Effects of campus culture on students' critical thinking. Review of Higher Education, 23, 421-442.
Twale, D., & Sanders, C. (1999). Impact of non-classroom experiences on critical thinking ability. NASPA Journal, 36, 133-146.
Vendley, G. (1998). Effect of a multi-ethnic, multicultural program on student participants. NASPA Journal, 35, 234-244.
Vogelgesang, L., & Astin, A. (2000). Comparing the effects of community service and service learning. Michigan Journal of Community Service Learning, 7, 25-34.
Volkwein, J., & Carbone, D. (1994). The impact of departmental research and teaching climates on undergraduate growth and satisfaction. Journal of Higher Education, 65, 147-167.
Volkwein, J., Valle, S., Parmely, K., Blose, G., & Zhou, Y. (2000). A multi-campus study of academic performance and cognitive growth among native freshmen, two-year transfers, and four-year transfers. Paper presented at the annual forum of the Association for Institutional Research, Cincinnati, OH.
Watson, L., & Kuh, G. (1996). The influence of dominant race environments on student involvement, perceptions, and educational gains: A look at historically black and predominantly white liberal arts institutions. Journal of College Student Development, 37, 415-424.
Weast, D. (1996). Alternative strategies: The case for critical thinking. Teaching Sociology, 24, 189-194.
West, K. (1994). Enhancing critical thinking in the political science curriculum. Dissertation Abstracts International, 55, 725A.
Whitmire, E. (1998). Development of critical thinking skills: An analysis of academic library experiences and other measures. College and Research Libraries, 59, 266-273.
Whitmire, E., & Lawrence, J. (1996). Undergraduate students' development of critical thinking skills: An institutional and disciplinary analysis and comparison with academic library use and other measures. Paper presented at the annual meeting of the Association for the Study of Higher Education, Memphis, TN.
Whitt, E., Edision, M., Pascarella, E., Nora, A., & Terenzini, P. (1999b). Interaction with peers and objective and self-reported cognitive outcomes across three years of college. Journal of College Student Development, 40, 61-78.
Whitt, E., Pascarella, E., Pierson, C., Elkins, B., & Marth, B. (in press). Sex and gender in college: Evidence of differences in experiences and outcomes. Nashville, TN: Vanderbilt University Press.
Widick, C. (1975). An evaluation of developmental instruction in a university setting. Dissertation Abstracts International, 36, 2041A.
Williams, R. (1996). Assessing students' gains from the college experience at East Tennessee State University. Dissertation Abstracts International, 57, 970A.
Winston, R., & Sandor, J. (1984). Academic advising inventory. Athens, GA: Student Development Associates.
Wood, P. (1997). A secondary analysis of claims regarding the reflective judgment interview: Internal consistency, sequentiality, and intra-divisional differences in ill-structured problem solving. In J. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 12, pp. 243-312). New York: Agathon.
Wood, P. (2000). Scaling and scoring of an objective measure of epistemic development. Paper presented at the annual meeting of the American Psychological Association, Washington, DC.
Wright, S. (1989). Selected correlates of intellectual development of college freshmen. Dissertation Abstracts International, 50, 3498A.
Wright, S. (1992). Fostering intellectual development of students in professional schools through interdisciplinary coursework. Innovative Higher Education, 16, 251-261.
Zhang, Z., & Richarde, R. (1998). Assessing college students' development: A repeated -measures analysis using a mixed model. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
[1] For additional studies providing evidence that level of exposure to postsecondary education is positively related to increased complexity or sophistication of epistemological development or beliefs about knowledge see Collins, White, and O’Brien (1992); Mansfiel & Clinchy, (1990); and Schommer (1990; 1993). Less convincing findings, though in the same direction, are reported by Jehng, Johnson, and Anderson (1993) and Paulsen and Wells (Paulsen & Wells, 1998). Zhang and D. Richarde (1998) also report related evidence to suggest that male students make gains in dimensions of metacognitive development (i.e., knowing what one knows and using consistent but adaptive strategies in problem solving during college).
[2] This body of evidence is perhaps further supported by Facione’s (1997) finding that nursing student’s critical thinking skill levels are positively associated with low student/faculty ratios in the programs in which they are enrolled. This association, however, is not adjusted for level of precollege critical thinking.
[3] There are additional studies that look at environmental influences on the growth of general cognitive skills. For example, Terenzini et al. (1994) found that perceived covert racism on campus positively influenced critical thinking, while Whitt et al. (in press) found that critical thinking in women was positively influenced by institutions that had a strong “chilly climate for women.” These somewhat counterintuitive findings, however, are based on a single sample and await replication. Tsui’s (2000) qualitative study of four campuses found that student self-reported gains in critical thinking were maximized by a campus culture that stressed active and collaborative learning experiences and fostered social and political awareness.
[4] In this regard, it is worth noting that Astin (1993b) found that taking interdisciplinary courses had a positive net effect on seniors’ self-reported gains in critical thinking skills. Similarly, although their findings may be confounded by sample mortality and absence of a control group, Avens and Zelley (1992) found that community college students participating in an interdisciplinary learning community made gains on the Perry (1970) continuum of intellectual and ethical development that were somewhat larger than national norms.
[5] See Graham and Gisi (1999) for evidence from a national sample that alumni retrospective reports of their satisfaction with the instruction they received in college are positively associated with self-reported cognitive gains (e.g., analyzing and drawing conclusions from various types of data). Similarly, a 20-college study by Volkwein, Valle, Parmley, Blose, and Zhou (2000) reports a strong, positive association between the perceived quality of the classroom experience and seniors’ self-reported intellectual growth (e.g., ideas, concepts, analytical reasoning).
[6] As this book was being completed, a study by Hayek, Carini, O’Day, and Kuh (2002) found that Greek-affiliated students reported greater general education gains during college than did their counterparts who were not Greek affiliated. However, there was no control for how students responded on the dependent measure when they entered college, so it is difficult to determine the internal validity of the results. What appears to be an “effect” of Greek affiliation could simply be the tendency for students who join fraternities and sororities to respond differently on self-reported gains instruments than other students (Pascarella, 2001).
[7] For evidence of growth in cognitive complexity and postformal reasoning associated with a multi-ethnic, multicultural immersion experience in a foreign country see Vendley (1998). See also MacPhee, Kreutzer, and Fritz (1994) with regard to the positive impact of diversity perspectives on critical thinking.