CHAPTER 4
Cognitive Skills and Intellectual Growth
In Chapter Three we focused on the learning outcomes typically thought to be directly, or at least purposefully, related to the curriculum or academic program of a college, namely, the acquisition of subject matter knowledge and academic (usually verbal and quantitative) skills. In this chapter we turn to the acquisition of more general intellectual or cognitive competencies and skills which, if they are not so directly tied to a particular curriculum or course of study, are nevertheless thought to be salient outcomes of postsecondary education (Jones, 1994; Jones, Dougherty, Fantaske, & Hoffman, 1997; Jones et al., 1995). As we pointed out in our 1991 synthesis, these cognitive skills go by a number of different names (e.g., critical thinking, reflective judgment, epistemological development, and so on), and they differ somewhat in conceptual definition and the types of problems or issues they address. They do, however, have as a common theme the notion of applicability and utility across a wide range of different content areas. These cognitive competencies and skills represent the general intellectual outcomes of postsecondary education that permit individuals to:
process and utilize new information; communicate effectively; reason objectively and draw objective conclusions from various types of data; evaluate new ideas and techniques efficiently; become more objective about beliefs, attitudes, and values; evaluate arguments and claims critically; and make reasonable decisions in the face of imperfect information. These and related general cognitive skills are a particularly important resource for the individual in a society and world where factual knowledge is becoming obsolete at an accelerated rate . . . (Pascarella & Terenzini, 1991, pp. 114-115).
Change During College
Conclusions from
In our 1991 synthesis we concluded that, compared to freshmen, college seniors have better oral and written communication skills, are better abstract reasoners or critical thinkers, are more skilled at using reason and evidence to address ill-structured problems for which there are no verifiably correct answers, have greater intellectual flexibility in that they are better able to understand more than one side of a complex issue, and can develop more sophisticated abstract frameworks to deal with complexity. Our best estimates of the magnitudes of the gains, or the advantage of seniors over freshmen, were as follows: oral communication, .60 of a standard deviation (a 22 percentile point advantage); written communication, .50 of a standard deviation (a 19 percentile point advantage); Piagetian formal (abstract) reasoning, .33 of a standard deviation (a 13 percentile point advantage); critical thinking, 1 standard deviation (a 34 percentile point advantage); using reason and evidence to address ill-structured problems, 1 standard deviation (a 34 percentile point advantage); and ability to deal with conceptual complexity, 1.2 standard deviations (a 38 percentile point advantage). These estimates were based on a large and extensive body of studies.
Evidence from the 1990s
Compared to the literature base we reviewed for our 1991 synthesis, we found relatively few studies focusing on change or gains in general cognitive skills and intellectual growth during college. The literature we did find is focused largely in two broad areas that we term critical thinking and postformal reasoning.
There are many different definitions of, and ways of measuring, critical thinking (Ennis, 1985; Erwin, 1997; B. Moore & Parker, 1989; Paul, 1987, 1992; Sternberg, 1985; Tsui, 1998a). However, it would appear that most attempts to operationally define and measure critical thinking focus on an individual’s capability to do some or all of the following: identify central issues and assumptions in an argument, recognize important relationships, make correct references from the data, deduce conclusions from information or data provided, interpret whether conclusions are warranted based on given data, evaluate evidence or authority, make self corrections, and solve problems (Erwin, 1997). Typical in the measurement of these dimensions of critical thinking is the notion that some answers or solutions are more verifiably correct than others.
It comes as no surprise that the generally accepted dimensions of critical thinking identified above have a strong cognitive component. Recently, however, a number of scholars have argued that, in addition to the cognitive ability to use or apply critical thinking skills, there is also a motivational dimension to critical thinking. They term this dimension the disposition to think critically (Erwin, 1997; P. Facione, Facione, & Giancarlo, 1996; P. Facione, Sanchez, Facione, & Gainen, 1995; Jones, 1993, 1995; Taube, 1997). Thus, critical thinking as a broad concept involves both cognitive skills and the dispositional openness or willingness to apply those skills. Disposition to think critically involves, among other traits, such factors as the inclination to ask challenging questions and follow the reasons and evidence wherever they lead, tolerance for new ideas, willingness to use reason and evidence to solve problems, and willingness to see complexity in problems (P. Facione, Facione, & Giancarlo, 1994; P. Facione et al., 1995). In our synthesis we will review research on both critical thinking skills and critical thinking disposition.
The most extensive research in the 1990s on gains in critical
thinking skills during college appears to have been conducted by Facione (1997). His findings are based on an aggregate sample
of over 6000 students in nearly 150 undergraduate samples from 50 nursing
programs throughout the
Facione’s (1997) results are of course limited by the fact that they are based on samples of nursing students rather than on broader samples of undergraduates. Nevertheless, with some exceptions (Criner, 1992; Marr, 1995), the weight of evidence from other cross-sectional studies, using various standardized measures of critical thinking such as the Watson Glaser Critical Thinking Appraisal or the California Critical Thinking Test, is generally consistent with Facione’s findings in suggesting that the greater one’s exposure to postsecondary education the more advanced one’s level of critical thinking skills (Beck, Bennett, McLeod, & Molyneaux, 1992; Brooks & Shepard, 1990; Drouin, 1992; Hill, 1995; McDonough, 1997; Mines, King, Hood, & Wood, 1990; C. Pearson, 1991). Although not all studies provide requisite statistical data, our best estimate from this group of cross-sectional studies is that seniors have an advantage over freshmen of between .55 and .65 of a standard deviation (21 to 24 percentile points).
The longitudinal research on gains in critical thinking skills is less extensive than the cross-sectional research. Moreover, only one study, (Mentkowski et al., 1991) traces growth in critical thinking for more than three years. Mentkowski et al. administered the inference, assumptions, and deductions scales of the Watson Glaser Critical Thinking Appraisal to 135 undergraduates upon entrance to a small liberal arts college and again toward the end of their senior year. The gain across all three scales from entrance to college through the senior year averaged .25 of a standard deviation (10 percentile points). Facione (1997) and Hagedorn, Pascarella, Edison, Braxton, Nora and Terenzini (1999) both report critical thinking gains based on multi-institutional samples, but only for a three year period. Facione traced the growth of 625 nursing students at eight institutions from their sophomore through their senior year with the California Critical Thinking Skills Test and reported an average gain of .15 of a standard deviation (6 percentile points). Hagedorn et al. analyzed data from a sample of over 1000 students attending 18 four-year colleges located in 15 different states. From the time they entered college until the end of their junior year the students in the sample made an average gain of .37 of a standard deviation (14 percentile points) on the critical thinking module of the Collegiate Assessment of Academic Proficiency (CAAP). The CAAP critical thinking module is an objective, standardized test of critical thinking skills that correlates in the .75 range with the Watson Glaser Critical Thinking Appraisal (Pascarella, Edison, Nora, Hagedorn, & Braxton, 1995).
We uncovered only one additional study that looked at change in critical thinking during college using an objective, standardized instrument. Saucier (1995) administered the Watson-Glaser Critical Thinking Appraisal to several small, convenience samples of nursing students when they entered the program, either in their sophomore or junior year of college. The samples were then followed up with the same test one month before graduation. Saucier reports average gains in two of the three consecutive years covered by the study, but in only one year was the gain statistically significant. The failure to find statistically significant gains in the study is likely due to the very small sample sizes, ranging from 7 to 18. Moreover, since standard deviations were not reported we could not compute effect sizes for the critical thinking gains.
Although change during college is quite often an inaccurate estimate of the actual impact of college, the evidence from the 1990s nevertheless suggests that students are making gains in critical thinking skills during college that are appreciably smaller in magnitude than the gains we observed in our previous synthesis. It is not readily apparent from the evidence why this is the case. However, the more modest evidence of change in the 1990s does underscore an important point. Not all students develop as critical thinkers during college. For example, Keeley (1992) had samples of freshmen and seniors at a single institution read and evaluate two essays to identify the underlying assumptions of the argument put forth. While seniors were, on average, more proficient than freshmen in identifying assumptions, between 58 and 78% of seniors made the mistake of restating at least one premise as an assumption and between 21 and 58% of seniors made the mistake of restating at least three premises as assumptions.
Our synthesis uncovered four studies that estimate growth in the disposition to think critically during college. Three of these studies were cross-sectional (Bers, McGowan, & Rubin, 1996; N. Facione, 1997; Giancarlo & Facione, 1997) and one was longitudinal (Giancarlo & Facione, 1997). All four studies estimated growth in the disposition to think critically with the California Critical Thinking Dispositions Inventory (CCTDI). The CCTDI (P. Facione & Facione, 1992; P. Facione, Facione, Blohm, Howard, & Giancarlo, 1998) is a standardized measure that assesses the extent of one’s internal motivation to engage problems, seek answers to questions, and make decisions using critical thinking skills. It measures this disposition along seven dimensions or subscales: truthseeking, open-mindedness, analyticity, systematicity, critical thinking self-confidence, inquisitiveness, and maturity of judgment; and also reports a total score.
Using the same large multi-institutional sample of undergraduate nursing programs previously described, Facione (1997) administered the CCTDI to a sample of over 2600 undergraduates, consisting of freshmen, sophomores, juniors, and seniors. Using the descriptive statistics reported by Facione, we estimate that the seniors in their sample had an advantage in critical thinking disposition over freshmen of about .50 of a standard deviation (19 percentile points). However, most of the change appeared to have occurred by the second year of college. The estimated critical thinking disposition advantage of sophomores over freshmen was .46 of a standard deviation (18 percentile points), or about 90% as large as the corresponding advantage of seniors over freshmen.
Once again, because the Facione (1997) sample is limited to students in a single professional program there may be some question as to the generalizability of the results. However, two additional cross-sectional studies, one at a private liberal arts university and one at a community college, report findings generally consistent with those reported in Facione’s analyses with undergraduate nursing samples. Giancarlo and Facione (1997) administered the CCTDI to a sample of over 1100 students in all four class levels at a private university. Using an estimated freshman standard deviation on the CCTDI from a subsample of students attending the same institution, we computed a senior advantage over freshmen in critical thinking disposition of .45 of a standard deviation (17 percentile points). Once again, the sophomore advantage over freshmen was almost as large as the senior advantage, but since the sophomore average was not reported numerically we hesitated to compute an effect size.
Bers, McGowan, and Rubin (1996) administered the CCTDI to a sample of 224 community college students and found that disposition to think critically had a statistically significant positive relationship with exposure to postsecondary education, operationally defined as number of credit hours completed. Using the descriptive statistics reported by Bers et al., we estimated that the difference in overall critical thinking disposition between students who had completed 30 or more credits and those who had completed 12 or fewer credits was .68 of a standard deviation, a 25 percentile point advantage for the former group.
The longitudinal study of gains in critical thinking disposition was conducted by Giancarlo and Facione (1997) at the same private university where they conducted their cross-sectional investigation. A sample of 155 students took the CCTDI as beginning freshmen and then again as seniors. From the first to the second testing, the sample increased a statistically significant 7.45 points in critical thinking disposition. This translated into a gain from freshman to senior year of .28 of a standard deviation (11 percentile points).
Critical thinking (as it has typically been assessed) focuses to a great extent on an individual’s ability to solve intellectual puzzles or problems. These puzzles or problems come in many forms and require the application of complex reasoning and information processing; yet they typically have the common trait of a verifiably correct, or at least a more valid, answer. The critical thinking, information processing, and formal reasoning skills involved in puzzle or problem solving are an important acquisition in the development of intellectual resources. However, there are other kinds of real-world adult problems that require a somewhat different approach to reasoning than the intellectual skills typically included under the ruberic of critical thinking. These are “ill-structured” or “wicked” problems for which there is likely to be conflicting or incomplete information, unspecifiable problem parameters, and a number of plausible solutions, none of which may be verifiably correct (P. King & Kitchener, 1994). Examples of such problems are gun control, safety of food additives, waste disposal, objectivity of the press, reducing poverty, teaching evolution in the schools, and the like. In addressing these and similar problems, formal rules of logic may not suffice; and tentative answers or solutions typically need to be “constructed” rather than “discovered.” Descriptions of successful entrepreneurs, managers, researchers, and educators often include references to strong abilities for framing and resolving ill-structured problems (Lynch, 1996; Sherman, 1994). Several scholars and cognitive development theorists (Basseches, 1984; Perry, 970; Wood, 1997) have argued that constructing tentative solutions to such real-world problems requires a set of intellectual capabilities beyond those typically included in the concepts of critical thinking or formal reasoning (hence, the term postformal reasoning). These intellectual capabilities have been operationally defined in a number of ways but the most well known, and those which have dominated the measurement of postformal reasoning, are the Reflective Judgment Interview (P. King & Kitchener, 1994), the Measure of Epistemological Reflection (Baxter Magolda, 1990a; Taylor, 1983), and the Measure of Intellectual Development (Knefelkamp, 1974; W. Moore, 1991a; Widick, 1975).
The Reflective Judgment Interview (RJI) measures reasoning along a multilevel continuum of seven stages. Each level is characterized by a person’s view of knowledge and his or her concept of justification of belief. To paraphrase King and Kitchener (1994), at the lowest stages (pre-reflective thinking), knowledge is gained either by direct, personal observation or through the word of authority and is absolute and certain. Reasoning at the middle stages (quasi-reflective thinking) recognizes that knowledge claims about ill-structured problems contain elements of uncertainty. However, the quasi-reflective individual has difficulty successfully addressing ill-structured problems because of a lack of understanding of their inherent ambiguity. At the highest stages (reflective thinking), it is understood that knowledge useful in addressing ill-structured problems must be actively constructed. Beliefs are justifiable to the extent that they are based on a rational process involving appropriate forms of inquiry and use of evidence. The RJI consists of four dilemmas and a set of standardized probe questions designed to tap level of reasoning. Each dilemma is defined by two contradictory points of view and represents an ill-structured problem. Subjects are asked to state and justify their points of view about the issues in each dilemma. The four dilemmas represent different content domains: science, current events, religion, and history.
In our current synthesis we uncovered only one published study in the first half of the decade that estimated gains during college in reflective judgment. Employing a cross-sectional design Mines, King, Hood, & Wood (1990) tested 20 freshmen and 40 seniors at a large research university with the RJI. Their descriptive statistics indicate that seniors had an advantage over freshmen of about 2.08 standard deviations, which converts to a 48 percentile point advantage. Fortunately, our current synthesis of gains during college in reflective judgment was augmented by two comprehensive literature reviews of research using the RJI with postsecondary samples (P. King & Kitchener, 1994; Wood, 1997). It is apparent from these reviews that our 1991 synthesis missed several unpublished studies and that our current synthesis would also have overlooked several unpublished studies. Consequently, we used the results of the work of King, Kitchener, and Wood (P. King & Kitchener, 1994; Kitchener, Wood, & Jensen, 1999; Wood, 1997) to reestimate the overall relationship between exposure to postsecondary education and level of reflective judgment.
King and Kitchener (1994) report the individual results for 20 cross-sectional studies of 966 traditional age (e.g., 18 year old freshmen and 22 year old seniors) college students using the RJI. The studies were conducted between 1978 and 1993. The evidence suggests a strong positive relationship between exposure to postsecondary education and reflective thinking. Using the individual study data they report in an appendix, we estimated the average reflective thinking advantage of seniors over freshmen across all studies to be .68 of a standard deviation (25 percentile points). Similar summary data reported by Wood (1997) yielded a senior advantage over freshmen of .79 of a standard deviation (29 percentile points). King and Kitchener also report data from 5 cross-sectional studies of 135 nontraditional age college students (e.g., 22 year old freshmen and 26 year old seniors). Using the information they report, we estimate the senior advantage over freshmen across all studies to be .98 of a standard deviation in reflective thinking, or a 34 percentile point advantage.
In their 1994 summary, King and Kitchener also report the results of a smaller number of longitudinal investigations of change during college in reflective thinking. From the data they report, we weighted the studies by sample size and estimated the gain in reflective judgment from the freshman to the junior year to be .65 of a standard deviation (a 24 percentile point gain) and from the freshman to the senior year to be about 1.90 standard deviations (a 47 percentile point gain). These estimates have to be viewed with extreme caution, however, since they are based on only two studies with a combined sample of only 60 students.
More recent research on reflective judgment not included in either the King and Kitchener (1994) or the Wood (1997) summaries would nevertheless appear to be quite consistent with their conclusions. In what is probably the most comprehensive study of reflective judgment to date, Kitchener, Wood, and Jensen (1999) administered the Reflective Judgment Interview (RJI) and the Reasoning About Current Issues test (RCI, an objective paper and pencil measure of reflective thinking) to 1588 freshmen, 141 sophomores, 135 juniors and 265 seniors at four different institutions. Of the total sample, 34 freshmen and 29 juniors at two of the institutions completed the RJI while the remainder at all four institutions completed the RCI. On both measures there was a statistically significant class effect; and using the descriptive statistics reported we estimated the junior advantage over freshmen on the RJI to be .86 of a standard deviation (a 21 percentile point advantage) while the senior advantage over freshmen on the RCI was estimated at .71 of a standard deviation (a 16 percentile point advantage).
Perhaps more important than a quantitative estimate of the gains in reflective judgment that occur during college is an understanding of the qualitative change in thinking processes that appears to happen between the freshman and senior year. Results of both cross-sectional and longitudinal studies using the RJI are quite consistent in indicating that the typical change in reflective judgment between the freshman and senior year is a movement of about half a stage on the reflective judgment conceptual model from stage 3, the end of pre-reflective thinking, to stage 4, the beginning of quasi-reflective thinking. The modest size of this advance may be deceiving in that it represents a qualitative shift from a style of reasoning based on personal beliefs to one that explicitly uses reason and evidence in forming judgments. As such, it may represent an important prerequisite for the development of a reasoned approach to addressing ill-structured problems.
Another line of inquiry on change or growth in postformal reasoning during college has employed the Measure of Epistemological Reflection (MER). The MER was designed to measure development on the first five stages of Perry’s (1970) scheme of intellectual and ethical development. In the first five stages of the Perry scheme the individual moves from an understanding of knowledge that is presumed to be dualistic, absolute and obtainable only from authorities to an understanding of knowledge that assumes multiplicity, that is contextual and relative, and where analytical skills enable one to critique ideas—their own and others. The MER is a written instrument that assesses the respondent’s views in six domains of thinking related to learning and elicits specific justification for the respondent’s thinking. In a longitudinal study at a single institution, Baxter Magolda (1990a) employed the MER, along with a semistructured interview that also addressed the same six domains, to estimate growth during two years of college on the Perry scheme. On both instruments progression through college was associated with statistically significant gains on the Perry scheme. Using the descriptive statistics reported by Baxter Magolda, we estimated the change from the beginning of the freshman to the beginning of the junior year to be about 1.37 standard deviations (41 percentile points) on the MER and 1.73 standard deviations (46 percentile points) on the semistructured interview. Gains from the freshman to the sophomore year were also statistically significant and were about 70% and 35% as large as the two-year gains on the MER and the semistructured interview, respectively.
Similar results are reported in another single institution, longitudinal study by Kube and Thorndike (1991). They also employed the MER, but their study differs from Baxter Magolda’s (1990a) investigation in that they traced growth from the beginning of the freshman year through the end of the senior year. Using the descriptive statistics they report, we estimated the freshman to senior gain on the MER to be 2.54 standard deviations, which translates to an average gain of 49 percentile points.
A final line of research on change or growth in informal reasoning during college is one of several versions of the Measure of Intellectual Development (MID), or the Scale of Intellectual Development (SID). Like the Measure of Epistemological Reflection, the MID is a written instrument that attempts to capture development across the first five positions of the Perry scheme. The somewhat lesser used SID is a multiple choice measure. Results of research using these instruments have typically not reported change in terms of average scores. Rather, the evidence is typically reported in terms of differences in group (e.g., freshman versus senior) distributions across the different stages of the Perry model. With the exception of one study which reports mixed results (Durham, Hays, & Martinez, 1994), the weight of evidence from both longitudinal and cross-sectional investigations constituting this body of research is generally consistent in suggesting a statistically significant, positive relationship between extent of exposure to postsecondary education and level of intellectual development (Hart, Rickards, & Mentkowski, 1995; May, 1990; W. Moore, 1991b; F. Pearson & Rodgers, 1998; P. Thompson, 1991; Zhang & Richarde, 1998). Although there is some variation among studies, generally it would appear that freshmen are functioning intellectually in the transition from Perry stages 2 and 3, while seniors are functioning intellectually between stages 3 and 4. In short, the growth during college in intellectual development defined by the Perry scheme is largely a movement from a dualistic, right-wrong notion of knowledge to one embracing the potential legitimacy of multiple perspectives and the importance of context.[1]
In addition to the evidence based on standardized measures, there is also a small body of research that attempts to estimate growth in cognitive skills and intellectual development during college using student or alumni self reports (Bauer, 1992, 1995, 1996, 1998; Bauer, Mitchell, & Bauer, 1991; Dollar, 1991; Graham & Cockriel, 1989; Kelley, 1994). Consistent with the findings from studies employing standardized measures reviewed above, the weight of evidence from this body of research suggests that the majority of students and alumni perceive that they made moderate to substantial growth during college in such areas as “analytical thinking,” “synthesis of ideas,” “critical thinking,” and “independent learning.”
Net Effect of College
Conclusions from
In the 1991 synthesis, we made the following general conclusions with respect to the net impact of college on general cognitive skills and intellectual growth.
1. Seniors in college have significantly better written and oral communication skills than freshmen, even when controls are made for age and academic ability.
2. Graduates of community colleges score significantly higher on a measure of general intellectual and analytical skill development even in the presence of controls for age, verbal ability, and mathematical ability.
3. There is reasonably sound and consistent evidence to suggest that exposure to postsecondary education has a statistically significant positive effect on critical thinking even when controls are in effect for precollege level of critical thinking, academic aptitude, maturation, family socioeconomic origins, and aspirations. The positive net influence of postsecondary education, at least in the initial years of exposure, however, appears to be concentrated in the enhancement of one’s ability to determine the validity of data-based conclusions and to evaluate the strength or weakness of arguments.
4. Those who attend college make significantly greater gains in reflective judgment, one’s ability to use reason and evidence in making judgments about controversial issues, than those who do not attend college. This difference persists even when controls are made for differences in academic ability.
5. Net of age, intelligence, and academic ability, exposure to postsecondary education appears to have a statistically significant, positive influence on one’s intellectual flexibility (i.e., ability to comprehend and effectively argue both sides of a complex or controversial issue).
Evidence from the 1990s
Although we did not find studies in the current synthesis that spoke to the net effect of college on written or oral communication skills, or intellectual flexibility, the evidence we did uncover is quite consistent with our 1991 synthesis in suggesting that exposure to postsecondary education does, in fact, have a statistically significant, positive effect on both critical thinking skills and post formal reasoning. Moreover, the literature of the 1990s provides a somewhat greater opportunity for estimating the magnitude of the net effect of college on critical thinking and, to a lesser extent, postformal reasoning.
A number of cross-sectional and longitudinal investigations have attempted to estimate the net impact of differential exposure to postsecondary education on critical thinking. The majority of these studies employ objective, standardized measures. Rykiel (1995) administered the Watson-Glaser Critical Thinking Appraisal to independent samples of incoming freshmen and outgoing sophomores at a single community college. The sophomores had completed at least 43 of the 64 credits required for graduation. Introducing statistical controls for a measure of verbal aptitude, Rykiel found that sophomores scored significantly higher on the Watson-Glaser total score and on the evaluation of arguments subscale. Using the group standard deviations and adjusted group means reported by Rykiel, we estimated that sophomores in her study had a net advantage over incoming freshmen in total critical thinking of .34 of a standard deviation (13 percentile points). Since we could not find group standard deviations for the Watson-Glaser evaluation of arguments subscale we could not estimate an effect size for this group comparison. However, Rykiel’s finding on this subscale is partially consistent with the conclusion from our 1991 synthesis that the effect of the early years of college on critical thinking skills appears to be largely one of fostering the ability to determine the validity of data-based conclusions and to evaluate the strength or weakness of arguments.
Although Rykiel’s (1995) findings might be confounded by age or maturation, they are nevertheless also consistent with longitudinal research in which maturation is controlled. Analyzing longitudinal data from 13 four-year and 4 two-year colleges participating in the National Study of Student Learning (NSSL), Pascarella, Bohr, Nora, and Terenzini (1996) sought to determine the effects of exposure to postsecondary education, operationally defined as credit hours taken, on end-of-first-year critical thinking. Critical thinking was assessed with the critical thinking module from the Collegiate Assessment of Academic Proficiency (CAAP), which is a standardized test quite similar to the Watson-Glaser Critical Thinking Appraisal. In the presence of statistical controls for level of precollege critical thinking, race, gender, age, academic motivation, work responsibilities, the average critical thinking level of students at the institution attended, and type of coursework taken, four-year college students taking 24 or more credit hours during the first year of college had a statistically significant advantage in end-of-first-year critical thinking over students taking 6 or fewer credit hours of about .41 of a standard deviation. This number converts to an advantage of 16 percentile points. The corresponding advantage for two-year college students was also statistically significant and was estimated at .24 of a standard deviation, or 10 percentile points (see also Klassen, 2001).
The Pascarella et al. (1996) estimate of the net effect of postsecondary education on critical thinking for two-year colleges (.24 of a standard deviation) is somewhat smaller than the estimated effect from Rykiel’s (1995) two-year college study (.34 of a standard deviation). However, the smaller effect reported by Pascarella et al. is based on one year of exposure to postsecondary education while Rykiel’s study encompasses nearly two years. The effect on critical thinking of one year of exposure to postsecondary education reported by Pascarella et al. in their four-year college sample (.41 of a standard deviation) is quite consistent with the results of an earlier study (Pascarella, 1989) included in our 1991 synthesis. In that study, Pascarella matched samples of secondary school seniors who attended and did not attend college on the Watson-Glaser and followed them for an entire academic year. Net of such factors as initial level of critical thinking, academic ability, secondary school grades, and educational aspirations, those students with one year of exposure to college had an advantage of .44 of a standard deviation in critical thinking over those students who did not attend college during the first year after graduation from secondary school.
Other research has attempted to estimate the net impact of college on critical thinking beyond the first year. In the Mines, King, Hood, and Wood (1990) study reviewed in the previous section on change during college, samples of seniors and freshmen at a single institution were administered both the Watson-Glaser Critical Thinking Appraisal and the Cornell Critical Thinking Test. In an attempt to determine if the freshman-senior differences noted on these tests were confounded by academic ability, Mines et al. adjusted the group differences for composite ACT score. Net of this measure of academic ability, seniors still had a statistically significant advantage over freshmen on both measures of critical thinking. Since no adjusted means are reported, however, we could not estimate an effect size.
While the results of the Mines et al. study might be confounded by maturation, other research that controls for the differential effects of age, among other factors, reports similar results. Doyle, Edison, and Pascarella (1998) followed students from 18 four-year colleges and universities in the National Study of Student Learning through the end of their third year of college. With controls in effect for a battery of confounding influences such as precollege level of critical thinking, academic motivation, race, gender, work responsibilities, socioeconomic status, campus residence, coursework taken, and the quality of instruction received, the number of credit hours taken still had a statistically significant, positive effect on end-of-third-year scores on the critical thinking module of the Collegiate Assessment of Academic Proficiency. Using descriptive statistics available from the National Study of Student Learning (Hagedorn et al., 1999), we estimated that students who had completed 72 or more credit hours had a net advantage in critical thinking over students who had completed 18 or fewer hours of .55 of a standard deviation, or 20 percentile points. Since this estimate of the net effect of three years of college is actually larger in magnitude than our previous estimate of simple change during four years of college, we suspect the latter is estimate is overly conservative.
We uncovered four cross-sectional studies that attempt to estimate the net effects of college on postformal reasoning using the Reflective Judgment Interview. Because of the interview nature of the RJI each of the studies is based on relatively small samples, ranging from 38 to 63. Thus, they have only limited statistical power to detect significant reflective thinking differences between students with different levels of exposure to college. Nevertheless, the weight of evidence from these studies supports the conclusion that undergraduate education probably does have a net positive impact on the development of reflective thinking. Evans (1989) and Mines, King, Hood, and Wood (1990) each administered the RJI to small samples of freshmen and seniors at a single institution. Evans matched the two groups with respect to gender, college grades and academic aptitude (ACT composite scores) and found that seniors had a statistically significant advantage over freshmen of about .9 of a standard deviation (22 percentile points). Mines et al. statistically controlled for ACT composite and likewise found that seniors had a statistically significant advantage over freshmen in reflective thinking. A more recent study by Jensen, Kitchener, and Wood (1999) found that the senior advantage over freshmen on the RJI became statistically non-significant when the effects of verbal ability were controlled. However, analyzing data from what appears to be a different sample, Kitchener, Wood, and Jensen (1999) found that, even when statistical controls were made for ACT composite scores, juniors still had a statistically significant advantage over freshmen in reflective thinking. Since the last three studies (i.e., Mines et al.; Jensen et al.; and Kitchener et al.) do not report statistically-adjusted group means, we were unable to estimate net effect sizes from their results.
A fourth study attempting to estimate the net impact of college on reflective judgment employed the recently developed objective measure of reflective thinking called the Reasoning About Current Issues Test (RCI). In the Kitchener, Wood, and Jensen (1999) investigation reviewed in the previous section on “change during college,” the authors introduced a statistical control for ACT composite score. With this control in place, the senior advantage over freshmen in reflective thinking remained statistically significant, although we could not estimate a net effect size because ACT-adjusted means were not reported. Most recently, however, Wood (2000) presented additional evidence on the RCI that permitted estimation of a net senior-freshman effect size. Using an expanded sample of students from seven institutions, Wood had comparison groups of freshmen and seniors that were essentially matched on ACT composite score. Freshman ACT scores averaged 25, while the average ACT of his seniors was 24.72. Despite ACT score parity, however, the median performance of seniors was about 18 to 20 percentile points higher than the median performance of freshmen across the discrimination and endorsement of judgment sections of the RCI.
The weight of evidence from these studies suggests that freshman-senior differences in reflective thinking, whether measured by the RJI or the RCI, cannot typically be explained away by differences in academic ability. It is still possible of course that such net effects are confounded by age (P. King & Kitchener, 1994).
The small body of research that estimates the net impact of college on general cognitive skills and intellectual development employing student self-reports is generally consistent with the evidence based on objective, standardized measures. For example, Whitmire and Lawrence (1996) analyzed data from a sample of over 9,000 undergraduates in different types of institutions using a measure of student self-reported intellectual development consisting of estimates of gains in such areas as ability to think analytically, ability to put ideas together, ability to learn on one’s own, and quantitative thinking. With statistical controls in place for age, college grades, academic major, and measures of the college environment, year in college had a statistically significant, positive impact on self-reported gains in intellectual development, although it was not possible to compute an effect size. Similarly, in a single institution study by Grayson (1996), graduating seniors had statistically significant advantage in analytical and communication skills over entering students that persisted in the presence of controls for gender, race, and language spoken in the house. The net advantage of graduating seniors over entering freshmen was approximately .65 of a standard deviation, or 24 percentile points.
Between-College
Effects
Conclusions from How College Affects
Students
In our 1991 synthesis we found only very limited evidence to suggest that the development of general cognitive skills was influenced by institutional characteristics. When general cognitive growth was assessed by self-reports there was some evidence to suggest it is positively influenced by attendance at an institution with an academically selective undergraduate student body. However, this relationship generally disappeared in studies using objective, standardized measures of cognitive growth and/or controlling for important precollege characteristics. We concluded that student-body selectivity in and of itself may tell us little about institutional influences on general cognitive skills and intellectual growth. Rather, selectivity may only have a discernible impact on student general cognitive development if it is combined with other institutional factors such as small size and an institutional ethos that encourages a high level of student academic effort and involvement.
There was some cross-sectional evidence paired with statistical controls suggesting that African American students demonstrate greater development in measures of critical thinking and concept attainment at historically black institutions than at predominantly white ones. However, this finding was based on a single study.
The only area where we found replicated evidence with respect to between-college effects on general cognitive skills (and it was based on two studies conducted nearly thirty years apart) was in the influence of institutional curricular emphasis. Despite some methodological limitations, the findings suggested that students at institutions with a strong and balanced commitment to general education demonstrated particularly marked gains in measures of critical thinking and adult reasoning skills.
Evidence from the 1990s
Compared to our previous synthesis we uncovered a relatively larger body of evidence focusing on between-college effects on general cognitive skills and intellectual growth. This evidence falls generally into three categories: 1) institutional characteristics, 2) institutional type, and 3) institutional environments.
As with our synthesis of the evidence on the development of verbal, quantitative, and subject matter competence in the preceding chapter, undergraduate student-body selectivity was by far the dominant institutional characteristic considered in research on between-college effects on general cognitive skills and intellectual growth. The weight of evidence from this body of research is quite consistent with the conclusions from our 1991 synthesis. For example, a series of studies analyzing the National Study of Student Learning data base have attempted to determine if the average student-body academic ability (measured in various, but highly correlated ways) had a net impact on the development of individual students’ critical thinking (Doyle et al., 1998; Edison, Doyle, & Pascarella, 1998; Hagedorn et al., 1999; Prendergast, 1998). The sample consisted of five two-year colleges and 18 four-year colleges. While the sample was small, the four-year colleges ranged from essentially open-admission institutions to some of the most selective liberal arts colleges and research universities in the country. When statistical controls were made for such factors as individual precollege critical thinking, race, gender, family social origins, and age, the average critical thinking level of incoming students at each of the four-year institutions had only trivial and non-significant effects on scores on a standardized measure of critical thinking after three years of college (Hagedorn et al., 1999; Prendergast, 1998). When operationally defined as the average score on a composite of objective tests of precollege reading comprehension, mathematics, and critical thinking, institutional selectivity actually had a small, but statistically significant, negative effect on end-of-third year critical thinking. This negative impact persisted in the presence of controls for individual precollege critical thinking, socioeconomic origins, age, race, gender, and full- or part-time enrollment (Doyle et al., 1998; Edison et al., 1998).
Although based on only five institutions, there is some evidence in the Hagedorn et al. (1999) study to suggest that institutional selectivity positively influences the end-of-first-year critical thinking of two-year college students. However, this was not replicated by Whitt et al. (in press) employing a more extensive set of individual- and institution-level controls.
Results quite consistent with those yielded by the National Study of Student Learning data have been reported in analyses of a much more comprehensive sample of about 200 institutions in the Cooperative Institutional Research Program 1985-1989 data base. When controls were made for precollege academic ability and an extensive set of other potential confounding influences, student body selectivity (the average SAT or ACT score of the incoming students) had no statistically significant impact on the analytic score of the Graduate Record Examination (Anaya, 1992; Astin, 1993a; Dey, 1991).
When gains in general cognitive development are based on student self-reports rather than on objective, standardized measures, evidence concerning the impact of institutional selectivity is mixed and generally inconclusive. Most of the research in this area is based on various longitudinal iterations of the Cooperative Institutional Research Program data, and, with one exception (Volkwein, Valle, Parmely, Blose, & Zhou, 2000), each study estimates the net impact of selectivity after controlling for important precollege confounding influences. Kim (1995; 2002a), Tsui (1998b), and Volkwein et al. (2000) report small positive net effects of institutional selectivity on self-reported gains in such dimensions of cognitive development as critical thinking and analytic and problem-solving skills. However, analyses predicting self-reported gains on the same or very similar cognitive dimensions by Astin (1993b), Dey (1991), Franklin (1993), and Strauss and Volkwein (2001) report no statistically significant impact for institutional selectivity.
The effects on general cognitive skills of institutional characteristics other than student-body selectivity have been estimated. For example, Dey (1991) reported that institutional size negatively influenced at least two of three measures of gains in critical thinking skills; and he hypothesizes that this negative effect may be due to the inhibiting influence that institutional size has on interaction with peers and faculty. This finding, however, is based on a single sample and awaits replication.
The overall weight of evidence suggests that the net effects of institutional type on students’ general cognitive development are generally trivial in magnitude or inconsistent. For example, research based on two independent samples found that when student precollege characteristics and other potential confounding influences are taken into account, attendance at a two-year community college versus a four-year college or university has no statistically significant impact on end-of-first-year scores on a standardized, objective measure of critical thinking skills (Bohr et al., 1994; Pascarella, Bohr, Nora, & Terenzini, 1995a; Terenzini, Springer, Yaeger, Pascarella, & Nora, 1994). Similar results are reported with respect to the impact of attending an historically Black college (HBC) or predominantly White institution (PWI) on the development of critical thinking skills in African American students. When level of precollege critical thinking and other important confounding influences were controlled statistically, African American students attending HBCs had end-of-first- and end-of-third-year scores on a standardized, objective measure of critical thinking that were essentially indistinguishable from those of their counterparts attending PWIs. Evidence more supportive of the cognitive benefits of attendance at an HBC is reported by DeSousa & Kuh (1996), Kim (1995; 2002a), and Flowers (2002). All of these studies found that African-American students attending HBCs reported significantly greater progress on self-reported measures of gains in critical thinking or analytical skills than African American students at PWIs; although only Kim and Flowers introduced controls for potential confounding influences. Thus, when cognitive outcomes are assessed with self-report measures there is some support for the tentative conclusion from our 1991 synthesis that attendance at an HBC may foster increased levels of critical thinking in African American students. This was not the case, however, when outcomes are assessed with objective measures.
Other institutional typologies have been considered with respect to their impact on students’ general cognitive development. These typologies include: private versus public control (Franklin, 1993; Terenzini, Springer, et al., 1994); women’s versus coeducational institutions (Kim, 1995; 2002a); and Carnegie, or similar, classification (e.g., research university, doctoral university, comprehensive college, liberal arts college) (Franklin, 1993; Whitmire & Lawrence, 1996). The results of these investigations suggest either trivial and statistically non-significant or inconsistent effects. Kuh (1993b) did find that students at liberal arts colleges reported greater gains in cognitive complexity than their counterparts at comprehensive or metropolitan institutions; but without controls for student background characteristics, it is difficult to determine just how much of this difference is simply attributable to differential recruitment effects rather than to institutional impacts.
Institutional
Environments
As with our synthesis in the preceding chapter, research on the impact of institutional environments on the development of general cognitive skills and intellectual growth is characterized by a wide range of approaches and operational definitions of “institutional environment.” We believe the evidence, however, can be generally grouped into the following three general environmental emphases: scholarship and learning, close student-faculty relationships, and vocational/professional training.
A small body of research has focused on the extent to which an institution’s emphasis on scholarship or learning has a net influence on students’ intellectual growth and development of general cognitive skills. The clear weight of evidence from this research indicates that the stronger an institution’s scholarly emphasis the greater the growth in intellectual skills. This relationship holds irrespective of whether or not intellectual skills are measured with objective tests or student self-reports and even with controls for other institutional characteristics and/or environmental emphases. For example, Prendergast (1998), analyzing the National Study of Student Learning data, used a three-item measure estimating the extent to which students perceived the institutional environment as placing an emphasis on being critical, evaluative, and analytical, as well as emphasizing the development of scholarly and intellectual qualities and the development of expressive, aesthetic, and creative qualities. Controlling for an extensive battery of confounding influences, including student precollege critical thinking and institutional selectivity, this measure of an institution’s scholarly emphasis had a modest, but statistically significant, positive influence on end-of-third year scores on an objective measure of critical thinking. Analyzing the same data and employing a similar analytic design, Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) reported similar findings for end-of-first-year critical thinking; but in their analyses it was the institution’s emphasis on being critical, evaluative, and analytical that accounted for the influence. The findings of Prendergast and Terenzini et al. with an objective measure of critical thinking are essentially replicated in studies using the same environmental scale but student self-reports of their gains in intellectual and analytic competencies (Arnold, Kuh, Vesper, & Schuh, 1993; Kuh, Arnold, & Vesper, 1991; Whitmire & Lawrence, 1996).
Although based on studies that employ diverse measures, there is nevertheless a body of evidence to suggest that the extent to which an institution emphasizes close relationships and frequent interaction between faculty and students has implications for students’ general intellectual/cognitive development. For example, analyses by Terenzini et al. (1994) found that students’ end-of-first-year scores on an objective measure of critical thinking were positively and significantly (if modestly) influenced by the extent to which they perceived that the faculty at their institution were accessible to students and concerned about student development and teaching. This positive effect persisted even in the presence of controls for an extensive battery of potential confounding influences, including student level of precollege critical thinking, institutional control, and the scholarly emphasis of the institutional environment. Generally consistent findings are yielded when general cognitive growth is measured by student self-reports. After controlling for important confounding influences, Astin (1993b), Graham (1998), and Kuh, Arnold, and Vesper (1990) all provide evidence to suggest that student self-reported gains in critical thinking, analytical skills, or general intellectual development are significantly enhanced when the institution’s faculty are oriented toward students or when faculty are accessible to students and are concerned about student growth and development during college.[2]
Finally, there is a small body of literature which considers the impact on students’ general cognitive development of an institutional environment that emphasizes vocational preparation. The findings of this body of research are contradictory, and the direction of effects depends upon use of objective or student self-report measures of cognitive outcomes. Both Arnold et al. (1993) and Kuh et al. (1991) report that student self-reports of their gains on a measure of intellectual skills that includes analysis, synthesis, and learning on one’s own are positively influenced by an institutional environment that stresses professional/occupational competence and the practical value of one’s course of study. This relationship persisted in the presence of controls for students’ levels of social and academic involvement and other measures of the institutional environment. Conversely, using an objective measure of critical thinking after three years of college as the dependent variable, Prendergast (1998) found that the same scale used by Arnold et al. and Kuh et al. to assess an institution’s emphasis on vocational preparation had a statistically significant negative influence. This relationship also persisted when controls were made for such influences as level of precollege critical thinking, measures of academic and social involvement, institutional selectivity and other measures of the institutional environment.[3]
Within-College Effects
Conclusions from How College Affects Students
In our 1991 synthesis, we made the following general conclusions about within-college effects on general cognitive skills and intellectual growth.
1. One’s major course of study has a selective impact on the development of general cognitive skills. A student’s cognitive growth is greatest on measures where the content is most consistent with his or her academic major. On general measures of critical thinking (e.g, the Watson-Glaser Critical Thinking Appraisal) or postformal reasoning (e.g, reflective judgment), one’s academic major has little consistent relationship with gains.
2. The learning cycle-inquiry approach to instruction has been shown to enhance the development of formal (abstract) reasoning and conceptual complexity. This approach stresses an active, inductive learning process in which learner involvement in actual experiments or other concrete activities is used to introduce concepts and abstractions.
3. We found no single instructional or curricular approach that consistently and significantly facilitated the growth of critical thinking when critical thinking was measured by general instruments such as the Watson-Glaser Critical Thinking Appraisal. There was statistically non-significant evidence to suggest instruction that stresses student discussion at a relatively high level of cognitive activity and/or instruction that emphasizes problem-solving procedures and methods may enhance critical thinking.
4. There is evidence that a curriculum experience that requires the integration of ideas and themes across courses and discipline enhances critical thinking over simply taking a distribution of courses without an integrative rationale.
5. Specifically structured course interventions may enhance the development of postformal reasoning, specifically, stage movement on the Perry continuum. These interventions, which have been termed cognitive developmental instruction, focus on providing challenges to the individual’s initial cognitive and value structures paired with instructional supports appropriate for the individual’s initial level of cognitive development.
6. The estimated magnitude of instructional or curricular effects on measures of general cognitive skills tends to be smaller than that of the effect of the overall college experience. Put another way, a single instructional or curricular experience over a limited period may not provide the developmental impact of a cumulative set of mutually reinforcing experiences over an extended period of time.
7. Extent of growth in general cognitive skills during college appears to be a direct result of a student’s quality of effort or involvement in college. Involvement in intellectual and cultural activities may be more important to general cognitive development than other types of involvement (social, athletics, and so on). Yet it also appears that the nature and quality of social interactions with faculty members and student peers play a role of some consequence in one’s cognitive growth. These interactions are of particular salience if they focus on ideas or intellectual matters. The weight of evidence suggests that the more one’s social experience reflects and reinforces one’s academic experience, the greater will be the possibilities for intellectual development.
As with the evidence pertaining to within-college effects on the acquisition of verbal, quantitative, and subject matter competence, the corresponding evidence from the 1990s on more general cognitive skills and intellectual growth is extensive. Again, in an attempt to make organizational sense of this vast literature we have grouped the studies into the following six general clusters: 1) academic major, 2) coursework/curricular patterns, 3) general pedagogical approaches, 4) teacher behaviors, 5) academic effort/involvement, and 6) social and extracurricular effort/involvement.
Academic Major
Critical Thinking
It is difficult to form any firm conclusion concerning the impact of academic major on critical thinking. A substantial number of studies find no statistically significant differences in scores on standardized critical thinking measures among students from different academic fields of study (McDonough, 1997; Money, 1997; Sebrell & Erwin, 1998; Spaulding & Kleiner, 1992). In those studies that do find significant critical thinking differences among students in different academic majors (Gadzella & Masten, 1998; Gunn, 1993), it is difficult to determine if academic or other experiences in the major cause differences in critical thinking or if different academic fields of study simply tend to attract students with different levels of critical thinking ability to begin with (L. Tsui, 1999).
Studies that attempt to separate the recruitment effects from the socialization effects of different academic majors produce similarly inconsistent results. For example, Beckett (1996) found that health science majors made significantly greater gains than liberal arts or business majors on the Watson Glaser Critical Thinking Appraisal. Since the health science majors started out with higher scores this is just the opposite of what would happen if the differences in gains were simply the result of regression artifacts (Pascarella & Terenzini, 1991). Conversely, King, Wood, and Mines (1990) found that differences among seniors in different academic majors on the Watson Glaser Critical Thinking Appraisal and the Cornell Critical Thinking Test essentially disappeared when controls were introduced for academic aptitude. Similarly inconsistent results are reported when the dependent measure is self-reported gains in the ability to think critically rather than objective, standardized tests of critical thinking skills. Some studies find net differences due to academic major (Smart, Feldman, & Ethington, 1999; Whitmire & Lawrence, 1996) while others do not (Li, Long, & Simpson, 1998, 1999). Characteristic of this body of studies is the use of different operational definitions of academic major, which likely contributes to the inconsistent findings. Even with this taken into account, however, we find little consistent evidence to suggest that one’s major field of study, in and of itself, leads to different effects on general measures of critical thinking.
Reasoning Skills
We found very little research that focused on the impact of academic major on reasoning skills. The research that we did find, however, clearly underscores the notion that intellectual training in different fields of study leads to the development of different reasoning skills. For example, a longitudinal study by Lehman and Nisbett (1990) followed a sample of undergraduate students from the first semester of their initial year in college through the second semester of their senior year. The sample was divided into four groups, based on their eventual major: natural science, humanities, social science, and psychology. There were only chance differences among the groups in first-semester scores on measures of verbal reasoning, statistical/methodological reasoning, conditional reasoning, and academic ability. At the end of four years of college, however, there was a distinct pattern of differences in reasoning gains made that reflected one’s major field of study. Students majoring in the social sciences and psychology demonstrated substantially greater gains in the ability to solve real-world or scientific problems requiring statistical or methodological reasoning (average gain about 65%) than did natural science or humanities majors (average gain about 25%). Conversely, natural science and humanities majors made substantial and statistically significant gains in the ability to solve real-world or scientific problems requiring conditional reasoning (average gain about 63%), while their counterparts majoring in the social sciences or psychology made essentially no improvement in conditional reasoning during four years of college. With the exception of the impact of majoring in humanities on conditional reasoning, for which Lehman and Nesbit candidly admit they have no explanation, the differential impact of academic major on different reasoning skills would appear to reflect the type of inductive reasoning emphasized in different undergraduate academic fields (i.e., conditional reasoning in the natural sciences and statistical/methodological reasoning in the social sciences).
Other investigations have also found significant differences in reasoning or metacognitive skills among students in different academic majors (e.g., Zhang & Richarde, 1998). There is little to suggest, however, that anything but chance differences exist among students in these academic majors in the magnitude of the freshman-to-senior gains in reasoning skills.
Coursework/Curricular Patterns
One reason that major field of study may demonstrate little consistent impact on growth in critical thinking skills is that it represents too broad a representation of the formal academic experience one has during college. Thus, it is not particularly surprising that a number of investigations have focused on the impact on critical thinking of specific patterns of courses taken. Unfortunately, there is only limited consistency in the findings of these studies. A series of analyses of the National Study of Student Learning data have found that, even when such factors as precollege critical thinking, institutional characteristics, and other types of course exposure are taken into account, the number of science or engineering courses taken in the first year of college has a modest, but statistically significant, positive impact on end-of-first-year scores on the critical thinking test of the Collegiate Assessment of Academic Proficiency (Doyle et al., 1998; Terenzini, Springer et al., 1994). Similar results are reported by Olsen (1990a) in a single institution study using the same standardized measure of critical thinking and by Tsui (1999) in a large multi-institutional study using students’ self-reported growth in critical thinking. Olsen found that the number of science courses taken had a positive effect on critical thinking scores after two years of college, even when controls were made for academic aptitude, though not for other types of coursework taken. Tsui (1999) reported that number of science courses taken positively influenced seniors’ self-reported growth in critical thinking even in the presence of controls for level of exposure to other types of coursework and several instructional variables, though not for precollege academic aptitude.
There is also evidence, using objective, standardized measures to suggest that the number of arts and humanities courses taken may enhance end-of-first-year critical thinking (Terenzini, Springer et al., 1994), the number of literature courses taken may foster critical thinking after two years of college (Olsen, 1990a), and the number of social science courses taken may have positive effects on critical thinking measured after the third year of college (Edison et al., 1998). In contrast to the effects of natural science courses, however, we uncovered no replication of these findings on independent samples.
A different approach has been taken by Smith-Sanders and Twale (1997; 1998) in a single-institution study that sought to determine if degree of exposure to a required general education or core curriculum influenced scores on a standardized test of critical thinking. The general education or core curriculum included courses in the social sciences, natural sciences, mathematics, English composition, history, fine arts, philosophy, and the “Great Books.” The findings are somewhat difficult to interpret. With statistical controls for precollege academic ability, degree of exposure to the core curriculum had a significant curvilinear relationship with critical thinking skills. Generally, the greater one’s exposure to the institution’s core curriculum the higher one’s critical thinking skills; but, for reasons that were not entirely clear, students with the highest level of core curriculum hours completed had somewhat lower levels of critical thinking than students with less core curriculum hours completed. This finding, however, is based on a single sample and awaits replication.
Postformal Reasoning
A small number of studied have attempted to determine the impact of coursework or curricular organization on growth in various measures of postformal reasoning. At first glance it is difficult to find any basic commonality among the studies in terms of how they conceptualize or organize the notion of coursework or curricular experiences. However, there does appear to be a common thread running through investigations by Schilling (1991) and Wright (1989; 1992). Both studies report the impact on students’ growth in postformal reasoning of exposure to an interdisciplinary or integrated core curriculum that emphasized making explicit connections across courses and among ideas and disciplines. Courses were typically designed, often by faculty teams, to be integrative in content and to stress synthesis of relationships and connections among different academic disciplines.
In the Schilling (1991) study, a sample of seniors who enrolled in the integrated, or interdisciplinary, core curriculum was matched with a group of seniors who were not exposed to it on entering academic ability (ACT score), sex, and area of academic interest. Both groups took the Measure of Epistemological Reflection (recall that the MER measures reasoning development along the first five stages of Perry’s (1970) continuum of intellectual and ethical development). Although there was not sufficient statistical data reported to estimate an effect size, Schilling reported that seniors enrolled in the interdisciplinary core curriculum scored at a significantly more advanced stage on the Measure of Epistemological Reflection than did the matched seniors who were not enrolled. Generally consistent results are reported by Wright (1992) for growth along the Perry continuum during the first year of college. A sample of first-year students took the Learning Context Questionnaire (LCQ), another measure of development on the Perry continuum, when they first entered college and again at the end of their freshman year. With statistical controls introduced for initial LCQ scores and SAT mathematics scores, the number of interdisciplinary, general education courses completed had a statistically significant, positive association with development in reasoning, as measured by gain scores on the LCQ during the first year of college.
Both the Schilling (1991) and the Wright (1992) studies appear to be potentially confounded by the interaction of student self-selection and change. Moreover, in the Wright study, the interdisciplinary courses also introduced several non-traditional pedagogical approaches such as collaborative projects and dialogues. Consequently, it is not clear how much of the positive impact of interdisciplinary courses on informal reasoning was confounded by pedagogical practices. Despite these methodological problems, however, the Schilling and Wright investigations are potentially important in that they represent replicated evidence linking integrated, interdisciplinary general education coursework with development or growth in postformal reasoning.[4]
Other scholars have also sought to assess the impact of innovative coursework and/or curricula on informal reasoning. For example, Pavelich and colleagues (Pavelich, 1996; Pavelich & Moore, 1996; Pavelich, Olds, & Miller, 1995) have investigated changes in engineering students’ position on the Perry continuum during exposure to an innovative curriculum that stresses an experiential, problem-solving approach to engineering education. The centerpiece of the curriculum appears to be a series of courses that emphasize: real-world, open-ended problem solving; team building; and oral and written technical communication skills. The assumption is that these experiences would help students mature toward more complex thinking as well as foster increased capabilities for more effective decisions about ambiguous, real-world, engineering problems. Using an interview approach to measuring position on the Perry continuum, Pavelich and colleagues report that seniors exposed to the curriculum have a statistically significant advantage of about 2.3 standard deviations (49 percentile points) in reasoning development over freshmen with little or no exposure to the curriculum. In the absence of a control group, however, it is difficult to determine just how much of the senior advantage in reasoning is uniquely attributable to the innovative coursework and curriculum versus how much would occur simply by four years of postsecondary education without exposure to the innovation. [For example, recall from the earlier section of this chapter on change during college that Kube and Thorndike (1991) reported a simple freshman to senior gain on the Measure of Epistemological Reflection of 2.54 standard deviations.] In this regard, it is worth noting that a generally similar curricular innovation in nursing education was not significantly linked to students’ growth in reflective judgment (Nickerson, 1991).
There is substantially less evidence with respect to the impact
of general or broad-based pedagogical approaches on cognitive skills and
intellectual growth than there is with respect to their impact on verbal,
quantitative and subject-matter competence.
Nevertheless some research does exist and we have organized it into the
following three categories: 1) computers and information technology, 2)
collaborative/cooperative learning, and 3) interventions designed to increase
cognitive growth.
Computer and Information Technology
Interestingly, the most extensive research we uncovered on
computers and the development of general cognitive skills and intellectual
growth focused on the impact of learning a computer program language. A fairly large number of studies have
addressed this issue. Fortunately, Liao
& Bright (1991) have conducted a
meta-analysis of some 65 of these studies.
Their criteria for inclusion of a study were as follows: 1) it had to
assess the relation between computer programing and general cognitive skills
such as planning skills, thinking skills, reasoning skills, and metacognitive
skills (all cognitive skills were measured by standardized tests); 2) it had to
take place in an actual classroom setting; and 3) it had to have a control
group which did not require students to learn a computer language. Of the 65 studies in their meta-analysis, only
nine were conducted with postsecondary samples.
Those nine studies yielded 20 effect sizes. We took the raw statistical data provided by
Liao and Bright for the postsecondary studies and estimated that college
students required to learn a computer program language had an advantage of .35
of a standard deviation (14 percentile points) in various general cognitive
skills over their counterparts who did not learn a computer program
language. This effect size was statistically
significant at p <.05. Thus, it would
appear that the impact of learning a computer language may extend beyond the
specific computer language to the development of general cognitive
capabilities.
Although they do not focus specifically on learning a computer
program language, reasonably consistent results are reported in correlational
studies by Flowers, Pascarella, and Pierson (1999) and Kuh and Hu (2000). Analyzing the National Study of Student
Learning data, Flowers et al. sought to determine if different types of
computer use influenced critical thinking during the first year of college. To this end, they introduced controls for an
extensive series of confounding influences such as pre-college critical
thinking, academic motivation, full- or part-time enrollment, coursework taken,
and quality of instruction received. In
the presence of these controls, the extent to which coursework required students
to learn to use computers had a modest, but significant positive relationship
with end-of-first-year scores on a standardized critical thinking measure for
students in five community colleges. The
corresponding effect for students at four-year colleges was not
significant. In analysis of data from
over 70 four-year colleges, Kuh and Hu found that using computers for such
learning activities as searching the internet for course material, analyzing
data, and making visual displays each had small, positive effects on student
self-reported gains in intellectual development (e.g., synthesizing, thinking
analytically and logically). These
positive effects persisted even in the presence of statistical controls for such
factors as sex, race, socioeconomic background, grades and educational
aspirations, academic major, work responsibilities, and institutional
characteristics.
We also uncovered one study that focused on the use of electronic mail as a pedagogical tool for influencing aspects of general cognitive development. Marttunen (1997) conducted a study in which undergraduates practiced informal argumentation with each other using e-mail. Argumentation was based around ideas presented in two books, and level of argumentation was determined by the extent to which an argument was grounded in reason or evidence. Over a six-week period, the student groups did not have face-to-face meetings, but there was a significant increase in the level of their argumentation in e-mail messages. The absence of a control group of individuals who engaged in the same type of argumentation, but without using e-mail, makes it difficult to determine the unique effects of e-mail itself. Nevertheless, the results do suggest the possibility that electronic mail is a feasible tool in practicing and improving one’s level of argumentation.
Collaborative/Cooperative Learning
An interesting correlational study by Karabenick and Collins-Eaglin (1996) suggests why one might expect collaborative or cooperative learning approaches to foster general cognitive skills and intellectual growth. Using data from over 1000 students in 57 classes, they found that the greater the class emphasis on collaborative learning and the lower the emphasis on grades, the more likely students were to use higher-order learning strategies of elaboration, comprehension monitoring, and critical thinking. (Elaboration is the attempt to relate ideas in one’s class to ideas in other courses; comprehension monitoring is the attempt to try to figure out a point when one becomes confused; and critical thinking refers to consideration of alternatives to a conclusion or point made in class.) To the extent that use of higher-order learning strategies leads to the development of higher-order thinking skills, one might then expect collaborative and/or cooperative approaches to learning to facilitate the development of general cognitive skills and intellectual development during college. Although not unequivocal, there is a body of evidence to support this expectation.
Qin, Johnson, and Johnson (1995) conducted a meta-analysis of 43 experimental and quasi-experimental studies that considered the effects of cooperative versus individualistic or competitive learning approaches on general problem solving skills. Problem solving was operationally defined as “a process that required participants to form a cognitive representation of a task, plan a procedure for solving it, and execute the procedure and check the results” (Qin et al., 1995) (p. 131). Four types of problems were considered: 1) linguistic problems, which are primarily solved in written or oral languages; 2) nonlinguistic problems, which are primarily represented and solved in pictures, graphs, mazes, symbols, or formulas; 3) well-defined problems, which have well defined operational procedures and solutions (e.g., a chess problem); and 4) ill-defined problems, which have uncertainty with regard to operational procedures and ultimate solutions (e.g., real-world problems such as deciding which car to buy). We took the raw statistical data provided by Qin, Johnson, and Johnson for 20 of the 43 studies that were carried out with postsecondary samples and conducted our own (secondary) meta-analysis. We estimate that, compared to their counterparts not learning in a cooperative format, college students learning in cooperative groups had a statistically significant advantage in overall problem solving of .47 of a standard deviation (18 percentile points). The magnitude of this advantage was largely unchanged by differences in the methodological quality of studies. (Estimates of the methodological rigor of each study were provided in the Qin, Johnson, & Johnson meta-analysis). Similarly, it appeared that the advantage in problem solving accruing to students engaged in cooperative learning was essentially the same for both well-defined problems (.46 of a standard deviation) and ill-defined problems (.49 of a standard deviation).
Additional secondary analysis of the studies conducted with postsecondary samples provided in the Qin, Johnson, and Johnson (1995) meta-analysis suggested a substantial difference in the magnitude of the cooperative learning advantage in solving linguistic problems (.31 of a standard deviation) versus nonlinguistic problems (.96 of a standard deviation). This finding with college students paralleled Qin, Johnson, and Johnson’s results when studies were aggregated across all education levels. They suggest that one reason for this difference in problem-solving advantages accruing to cooperative learning is that there may be more ways to solve nonlinguistic problems than linguistic problems. Thus, cooperative group discussion, “which may result in a great number of strategies being suggested may give cooperators a greater advantage over competitors on nonlinguistic than on linguistic problems” (Qin et al., 1995) (p. 139).
Johnson, Johnson, and Smith (1996) conducted an additional meta-analysis of the effects of academic controversy on the development of cognitive reasoning skills. Recall from our more detailed description in Chapter 3 that academic controversy is a variation of cooperative learning in which a cooperative group of four students is divided into two pairs who are then assigned to opposing positions on a major topic of controversy in a discipline or field of study. As previously indicated in Chapter 3, their meta-analysis was based on 26 published studies, 75% of which were randomized true experiments and about 42% of which were conducted with postsecondary samples. No separation of effect sizes is made for postsecondary and non-postsecondary studies. Nevertheless, they found that, compared with approaches in which the student learns on his or her own, academic controversy produced an advantage of .90 of a standard deviation (22 percentile points) in measures of cognitive reasoning.
Results from the National Study of Student Learning pertaining to the net effects of collaborative/cooperative learning experiences on students’ self-reported gains in analytical skills (Cabrera, Nora, Bernal, Terenzini, & Pascarella, 1998) are quite consistent with the postsecondary-level findings derived from the Qin, Johnson, and Johnson (1995) and the Johnson, Johnson, and Smith (1996) meta-analyses. Evidence with respect to the impact of collaborative/cooperative learning on critical thinking, however, is less consistent, and depends largely on whether critical thinking is assessed with student self-reports or objective, standardized measures. For example, analyzing the 1985-1989 Cooperative Institutional Research Program Data, Tsui (1999) found that participating in group learning projects had a significant, positive influence on seniors’ self-reported growth in critical thinking. This effect persisted even in the presence of controls for coursework taken and other instructional variables. Conversely, a class-level quasi-experiment (Miller & Groccia, 1997), and a multi-institutional correlational study with statistical controls for precollege critical thinking and a battery of other confounding influences (Doyle et al., 1998) found that cooperative learning experiences had only small and chance effects on standardized measures of critical thinking skills.
A substantial body of research in the 1990s has sought to determine if purposefully designed instructional interventions during college can positively influence general cognitive skills and intellectual growth. The majority of this research has focused on critical thinking skills (Dale, Ballotti, Handa, & Zych, 1997; Gadzella, Ginther, & Bryant, 1996) or the disposition to think critically (Bers et al., 1996). Although some studies employ self-report measures of critical thinking strategies or critical thinking gains (Logan & Salisbury-Glennon, 1999; Peterson, 1996; Reiter, 1994), the preponderance of investigations employ objective, standardized tests of critical thinking, such as the Watson Glaser Critical Thinking Appraisal, the Cornell Critical Thinking Test, and the California Critical Thinking Skills Test (Dale et al., 1997; Forbes, 1997; Gadzella et al., 1996; Gadzella, Hartsoe, & Harper, 1989; Inlow & Chovan, 1993; Langer & Chiszar, 1993; West, 1994) or essay measures of critical thinking (MacPherson, 1999; Price, Wilmes, & Turmel, 1994; Weast, 1996). The clear majority of studies describe the experimental intervention as one in which students are explicitly or implicitly taught critical thinking or problem-solving skills, typically imbedded within a semester- or quarter-long course; but there does not appear to be a clear operational consensus as to just exactly what that means pedagogically. Put another way, it is not particularly clear that instruction or practice in critical thinking skills means anywhere near the same thing across studies. As a result, it is not surprising that the body of evidence from this research is far from unequivocal. [Tsui (1998a) reached a similar conclusion, though based on a substantially smaller sample of studies.]
All of the studies we uncovered using pretest-posttest designs without a control group report that students exposed to critical thinking skills interventions make statistically significant gains in measured critical thinking skills (Gadzella et al., 1996; Logan & Salisbury-Glennon, 1999; MacPherson, 1999; Peterson, 1996). The absence of a control group, however, makes it difficult, if not impossible, to determine what part of these gains is uniquely attributable to instruction in critical thinking skills and what part is due to confounding factors, such as maturation, taking the same test twice, or simple exposure to the academic content in which critical thinking instruction might be embedded. The results of experimental and quasi-experimental investigations which include a control group of students not receiving explicit instruction in critical thinking skills are more equivocal than studies without a control group. Some find that critical thinking interventions confer a statistically significant advantage in critical thinking skills on students exposed to them (Dale et al., 1997; P. Facione et al., 1998; Weast, 1996), some report mixed effects (Bers et al., 1996; Langer & Chiszar, 1993; Reiter, 1994; West, 1994), and some report no statistically significant advantage associated with exposure to critical thinking interventions (Forbes, 1997; Gadzella et al., 1989; Inlow & Chovan, 1993; Price et al., 1994). The methodological rigor of the studies does not appear to be associated with whether or not they find positive, mixed, or chance results. Not all of the studies provide the requisite statistical information to compute effect sizes. However, using the statistical information from those studies that do, we estimate that purposeful instruction in critical thinking skills, broadly defined, leads to an advantage in students’ measured critical thinking skills of approximately .23 of standard deviation (9 percentile points). We would caution, however, that this is a rough estimate.
Consistent with the earlier conclusions of Halpern (1993), we would conclude from the evidence we reviewed that development in critical thinking can be enhanced by purposeful instruction and practice in critical thinking and/or problem solving skills. We would amend that conclusion, however, by pointing out that the effects may be widely variable and, therefore, not particularly large. We suspect that, in large measure, this may be due to the general absence of a consensus across studies with respect to the operational definition of what constitutes instruction in critical thinking.
In addition to the research on critical thinking, a smaller body of studies has focused on the impact of purposefully designed course or instructional interventions on the development of postformal reasoning (Kronholm, 1996; Marra, Palmer, & Litzinger, 2000; McAdams & Foster, 1998; S. Thompson, 1995). Postformal reasoning in these studies is measured either in terms of reflective thinking (P. King & Kitchener, 1994) or the Perry (1970) continuum. Overall, this body of research is based on carefully conducted experiments or quasi-experiments with relatively sound internal validity. Moreover, there is a general, if not total, consistency of findings across studies. Students experiencing instructional interventions designed to increase their intellectual development and skills in addressing ill-structured problems tend to score at more advanced levels on measures of reflective thinking or intellectual development than their counterparts not exposed to the intervention.
For example, Kronholm (1996) conducted an experiment to determine if a one semester instructional intervention would positively influence students’ reflective thinking. Students in two general education science courses were assigned to one of three groups: an intervention group and two control groups. Students in all three groups were assessed with the Reflective Judgment Interview (RJI) at the beginning and end of the course. The intervention group was taught according to King and Kitchener’s (1994) reflective judgment-developmental instruction model, while the two control groups received course instruction without the reflective judgment-developmental instruction emphasis. Reflective judgment-developmental instruction begins with the introduction of an ill-structured problem or issue and progresses to an exploration of the problem or issue. This is accomplished by very focused questions and activities aimed at encouraging students to think about epistemological assumptions - or how they come to know (P. King & Kitchener, 1994). With statistical controls for initial RJI position, Kronholm found that students exposed to reflective judgment-developmental instruction made significantly greater average gains in reflective thinking than the average of the students in the two control groups.
Generally, if not totally, consistent results are reported by Thompson (1995) in a quasi-experimental assessment of the impact of a similar intervention emphasizing reflective judgment-developmental instruction on gains in reflective thinking. Students exposed to the intervention scored significantly higher at the end of the study on the Reflective Thinking Appraisal (a written measure of reflective thinking or judgment) than did their counterparts not receiving the intervention. (The two groups demonstrated only chance differences in ACT scores and RTA scores at the beginning of the study.) The difference between the two groups at the end of the study on the Reflective Judgment Interview were in the same direction, but not statistically significant.
Slightly different approaches were taken by Marra, Palmer, and Litzinger (2000) and by McAdams and Foster (1998). Marra et al. investigated the impact of a project-focused, active-learning, team problem-solving course in engineering design, while McAdams and Foster estimated the effects of Deliberate Psychological Education (DPE). DPE stresses active practice in problem solving related to actual role-taking experiences and augmented interactive exchanges with peers. With controls introduced for sex, academic aptitude, prior grades, class standing and honors program participation, Marra et al. found that students who had completed the engineering design course had significantly more advanced positions on the Perry (1970) continuum, as measured by a semi-structured interview, than students who were yet to take the course. McAdams and Foster reported that students randomly assigned to Deliberate Psychological Education instruction made positive, though not statistically significant, advances on the Perry continuum, while their counterparts not receiving DPE instruction actually retrogressed slightly. Similarly, the DPE students made a significant gain on a measure of conceptual complexity, while the control students did not.
The McAdams and Foster (1998) study did not report sufficient statistical information to compute an effect size. However, the other three studies did (Kronholm, 1996; Marra et al., 2000; S. Thompson, 1995). Using the information they provided, we estimated that interventions designed to improve intellectual development, in terms of postformal reasoning skills, provide an average advantage in the development of postformal reasoning of about .65 of a standard deviation (24 percentile points). We would caution, however, that because it is based on only three studies this estimate may not be particularly robust.
Finally, there is some evidence that constructivist-oriented pedagogy can facilitate students’ movement toward more sophisticated and complex epistemological beliefs. For example, recall from Chapter 3 the quasi-experimental study conducted by Hofer (1994; 1998-99) in which sections of a calculus course were randomly assigned to a constructivist-oriented approach called “new wave” and a traditional approach. The new wave sections emphasized active and collaborative learning activities in and out of class, use of graphing calculators, and a curriculum based on textual material that was largely a collection of complex work problems for which answers were not provided. The traditional sections used a standard calculus text that proceeded sequentially, and material was covered primarily through lecture and presentation by the instructor. The new wave and traditional students differed in only chance ways in terms of potentially confounding influences such as gender, ethnicity, high school calculus background and grades, and tested mathematics ability. At the end of the course, however, the new wave students were significantly more likely to hold more sophisticated epistemological beliefs about mathematics knowledge (e.g., math problems may have more than one right answer) than the traditionally taught students. The advantage accruing to the students in the constructivist-oriented, new wave sections was about .16 of a standard deviation (6 percentile points).
Generally consistent findings are reported by Baxter Magolda (1992b) in an intricately detailed qualitative investigation that analyzed the experiences of a sample of students during their four years of college. In the course of periodic interviews, students reported a number of factors that promoted their epistemological development. One of the important factors was teachers defining learning as mutually constructed meaning and, thereby, encouraging students to engage in thought processes much like critical thinking.
We uncovered only a smattering of evidence concerning the impact of teacher behaviors on general cognitive skills and intellectual development. The research that speaks to this issue most directly comes from analyses of the National Study of Student Learning data. Pascarella, Edison, Nora, Hagedorn and Braxton (1996) and Edison, Doyle, and Pascarella (1998) sought to determine if students perceptions of the extent of teacher organization and preparation (e.g., class time is used effectively) and teacher instructional skill and clarity (e.g., instructors give clear explanations) in the overall teaching they receive at their institution significantly influenced critical thinking. In both studies critical thinking was assessed with an objective, standardized measure; and each introduced statistical controls for an extensive set of confounding influences such as precollege critical thinking, academic motivation, race, sex, age, full- or part-time enrollment, patterns of coursework taken, and work responsibilities. In the presence of these controls, extent of teacher organization and preparation had modest, but statistically significant, positive effects on critical thinking at the end of the first- and third-years of college. The net effects on critical thinking of teacher instructional skill and clarity were nonsignificant. Thus, it would appear that a teacher behavior (i.e., organization and preparation) that, as was seen in Chapter 3, has been shown to positively influence course-level content mastery, may also have implications for students’ development of more general cognitive skills during college.
Somewhat more indirect evidence of the impact of teacher behaviors or orientations to teaching is suggested in a single-institution study conducted by Kember and Gow (1994). They sought to determine how two teaching orientations termed “knowledge transmission” and “learning facilitation” influenced the ways in which students approached learning. Teachers with a knowledge transmission orientation tended to think that disciplinary or subject-matter knowledge is most important, to be imparted by clear presentation of information to students. Conversely, teachers who were oriented toward a learning facilitation approach emphasized the development of problem-solving skills, critical thinking, and independent learning; they believed that good teaching should motivate students and be an interactive process. The findings suggested that departments with faculty who exhibited a greater use of learning facilitation were significantly more likely to have students who took a “deep approach” to learning. A deep approach characterizes learners who have an intrinsic interest in the subject and who search for personal meaning in learning activities. In contrast, where a knowledge transmission approach predominated, students tended not only to engage in surface learning, but also to change more in the direction of surface learning. Students who utilize a “surface approach” to learning are extrinsically motivated and therefore concentrate on memorizing content that might appear in examinations.[5] Generally consistent results are reported by Tsui (2000) in a four-institution study that uses self-reported gains in critical thinking as the dependent measure.
Academic Effort/Involvement
In the preceding chapter we saw that the development of verbal, quantitative, and subject matter competence was determined, not only by a student’s formal coursework and instructional experiences, but also by his or her individual level of academic effort or involvement. How much students learn is determined to a great extent by how much personal effort and time they are willing to invest in the process of learning itself. It should come as no great surprise that level of academic effort and involvement also weighs in as a substantial influence on the growth of general cognitive skills and intellectual development during college.
For example, a series of analyses of the National Study of Student Learning have attempted to determine the factors influencing critical thinking during college (Doyle et al., 1998; Terenzini, Springer, Pascarella, & Nora, 1995). Both of these studies assessed critical thinking with an objective, standardized measure (the critical thinking module of the Collegiate Assessment of Academic Proficiency), and both introduced statistical controls for an extensive battery of potential confounding influences, such as precollege critical thinking, academic motivation, student demographic variables, full- or part-time enrollment, work responsibilities, patterns of coursework taken, and measures of student social involvement. In the presence of such controls, factors such as hours studied per week, number of non-assigned books read, and an academic effort/involvement scale (e.g., “took detailed notes in class,” “did additional readings,” “participated in class discussions”) had statistically significant, positive effects on critical thinking skills at the end of the first year of college.
Similar results are reported for measures of postformal reasoning in studies by Kitchener, Wood, & Jensen (1999) and May (1990). Kitchener et al. found that first year growth in reflective thinking or judgment, as measured by the Reasoning About Current Issues Test (an objective test of reflective thinking), was significantly linked with the extent to which students were actively involved in learning experiences. This included such dimensions as involvement in writing experiences and engagement in course learning. The more students actively processed information and ideas acquired in class the greater their gains in reflective thinking. May reported that freshman to senior advances in intellectual development on the Perry (1970) continuum were significantly and positively correlated with a scholarly/intellectual involvement scale that assessed students’ effort in such areas as use of the library, writing experiences, and course learning.
In addition to those studies that measure general intellectual or cognitive development with standardized assessment instruments, there is also a substantial body of evidence to suggest that level of academic effort/involvement has a positive net influence on student self-reported gains in critical thinking and intellectual development during college. While the research designs of these studies are variable, nearly all attempt to statistically control for important confounding influences. Though not all factors are controlled in each study, these controls in different studies include such factors as precollege academic ability, race, sex, educational aspirations, family socioeconomic status, institutional characteristics, and extracurricular involvement. In the presence of such controls, measures of student self-reported gains in critical thinking/intellectual development appear to be positively influenced by such specific factors as amount studied (Astin, 1993b; Cabrera et al., 1998, Volkwein, Valle, Parmley, Blose, & Zhou, 2000); participation in honors programs (Kim, 1996); library use and involvement (Whitmire, 1998; Whitmire & Lawrence, 1996; Williams, 1996); engagement in course learning activities (Whitmire, 1998; Williams, 1996); and more global measures of academic effort and engagement that combine specific dimensions such as studying, reading for pleasure, involvement in library experiences, involvement in course learning, and involvement in writing experiences (Arnold et al., 1993; Franklin, 1995; Grayson, 1995; Kaufman & Creamer, 1991; Kuh et al., 1991; Li et al., 1999; Watson & Kuh, 1996).
As with the preceding chapter on the development of verbal, quantitative, and subject matter competence, there is a substantial amount of evidence pertaining to the ways in which student social and extracurricular involvement influence general cognitive skills and intellectual growth. To maintain a parallel organization with Chapter 3 we review that evidence with respect to: interactions with peers; interactions with faculty; Greek affiliation; intercollegiate athletic involvement; service involvement; diversity experiences; work responsibilities; and on- or off-campus residence.
Interactions With Peers
A modest, but relatively consistent, body of research indicates that students’ peers play a substantial role in their general cognitive growth and intellectual development in college. Indeed, some studies suggest that one’s peers may be an influence that is equal to, and in some cases perhaps even greater, than one’s formal classroom experience (Astin, 1993b; Terenzini et al., 1995; Terenzini, Springer et al., 1994). The vast majority of studies are correlational and typically attempt to determine the net impact of peer interactions on various measures of general cognitive development while statistically controlling for important confounding influences.
For example, analyzing data from the National Study of Student Learning, Whitt, Edison, Pascarella, Nora, & Terenzini (1999a), sought to determine the unique, or net, impact of out-of-class interactions with peers on an objective, standardized measure of critical thinking. To do this, they introduced statistical controls for an extensive set of potential confounding factors such as precollege critical thinking, academic motivation, student demographic characteristics, full- or part-time enrollment, hours spent studying, work responsibilities, patterns of coursework taken, and the average academic ability of the first-year students at the institution attended. In the presence of these controls, a scale measuring the nature and level of involvement in out-of-class-interactions with peers (e.g., discussions about art, music, and theater; discussions with students whose personal values, political beliefs, religious beliefs, or national origin were different from one’s own) had a modest, but statistically significant positive impact on critical thinking skills at the end of the first year of college. Analyzing essentially the same sample, and employing much the same set of statistical controls, Prendergast (1998) found that a somewhat expanded version of the out-of-class interactions scale used by Whitt et al. had a significant, positive net effect on critical thinking skills at the end of the third year of college. Generally consistent findings with independent samples have been reported for scores on the Analytical section of the Graduate Record Examination (Astin, 1993b); for critical thinking skills, as measured by the critical thinking module of the Collegiate Assessment of Academic Proficiency (Twale & Sanders, 1999); for women’s freshman to senior growth on the Perry (1970) continuum, as assessed by the Measure of Intellectual Development (May, 1990); and for first-year gains on the recognition dimension of reflective thinking or judgment, as measured by the Reasoning About Current Issues Test (Kitchener et al., 1999; Kitchener, Wood, & Jensen, 2000). Examples of items measuring students’ interactions with peers from this research included: socializing with peers; hours spent outside of class discussing current issues with peers; having serious discussions with peers whose interests, values, and philosophy of life were different from one’s own; and having conversations with peers that referred to knowledge acquired in classes or readings.
Although less extensive than evidence pertaining to the impact of student interactions with peers, there is related evidence to suggest that student extracurricular or cocurricular involvement may also have positive implications for cognitive development. Analyzing data from a subsample of six National Study of Student Learning institutions, in which approximately half the students commuted to college, Inman and Pascarella (1998) found that a measure of involvement in college clubs and organizations positively influenced end-of-first-year scores on a standardized measure of critical thinking skills. [The same net effect is also reported by Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) with the complete NSSL sample of 23 institutions.] This positive effect could not be explained away by differences in such confounding influences as precollege critical thinking level, academic motivation, student demographic characteristics, full- or part-time enrollment, work responsibilities, measures of academic involvement, and institutional selectivity. Similar findings are also reported by Baxter Magolda (1992a; 1992b) in her detailed qualitative investigation of growth in students’ epistemological sophistication and reasoning skills over four years of college. As suggested by one of the participants in her study: “I think some of my best experiences were outside the classroom, where I could take what I learned in the classroom and apply it” (Baxter Magolda, 1992b, p. 296).
In addition to those studies that employ objective, standardized measures, there is also a body of research that estimates the net impact of interactions with peers on student self-reports of their intellectual or cognitive growth. These studies are also largely correlational in design and tend to introduce statistical controls to adjust for the influence of potential confounding factors. The weight of evidence from this research is also consistent in suggesting that various measures of students’ interactions with peers outside of class have modest, but statistically significant, positive effects on self-reported gains during college in general cognitive, analytical, or intellectual competencies (Arnold et al., 1993; Franklin, 1995; Kaufman & Creamer, 1991; Kim, 2002a; Kuh et al., 1991; Li et al., 1999; Volkwein & Carbone, 1994; Watson & Kuh, 1996).
Interactions With Faculty
A modest body of research permits one to estimate the net impact of student interactions with faculty on general cognitive skills and intellectual development. Although there are exceptions (Doyle et al., 1998; Inman & Pascarella, 1998), the weight of evidence from this body of research suggests that student-faculty interactions that 1) tend to reinforce or extend the intellectual ethos of the classroom, or 2) focus on issues of student development, can have positive implications for general cognitive development during college (Astin, 1993b; Dey, 1991; Franklin, 1993; Frost, 1991; Kim, 1996; 2002a; Kitchener et al., 1999; Kuh, 1995; Terenzini, Springer et al., 1994; L. Tsui, 1999).
For example, Kitchener, Wood, and Jensen (1999) measured gains in reflective thinking during the first year of college with the Reasoning About Current Issues Test. They found that gain-scores on two dimensions of reflective thinking—recognition and discrimination—were both significantly and positively associated with a scale that measured students’ out-of-class-interactions with faculty focusing on such things as course issues, work on a faculty research project, career choice, and personal development. Similar findings are reported in multi-institution-sample, correlational studies that use standardized measures of cognitive development such as the Analytic score of the Graduate Record Examination (Dey, 1991) or that employ student self-reports of growth in critical thinking, intellectual development, or problem-solving skills as the dependent variable (Astin, 1993b; Franklin, 1995; Kim, 1995, 1996). In all of these studies, the effect of student-faculty interaction persists even in the presence of controls for confounding influences such as precollege academic ability, student demographic characteristics, and institutional selectivity.
There is also evidence to suggest that general cognitive skills may be enhanced when student-faculty interactions focus on, or emphasize, student development. In analyses of the National Study of Student Learning data, Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) found that a scale measuring the extent to which faculty were perceived as being concerned with student development and teaching had a significant, positive effect on end-of-first-year scores on a standardized measure of critical thinking skills. This positive effect persisted even in the presence of statistical controls for an extensive list of potential confounding influences that included precollege critical thinking scores, academic motivation, student demographic characteristics, institutional characteristics, coursework taken, full-or part-time enrollment, and measures of academic effort/involvement. The findings of the Terenzini et al. investigation are generally consistent with those of Frost (1991). Frost uncovered evidence in one of two institutional samples that the developmental emphasis of female students’ advising contacts with faculty had a significant, positive effect on both the recognition of assumptions scale and the deductions scale of the Watson-Glaser Critical Thinking Appraisal. These effects persisted even with statistical controls for prior level of critical thinking, SAT scores, high school grades, and family income. Developmental advising (Winston & Sandor, 1984) regards the advising process as developmental education, not just providing information. Consequently, it emphasizes activities designed to help students assume responsibilities for problem-solving and decision making.
We uncovered only two studies that investigate the impact of fraternity or sorority membership on the development of general cognitive skills during college (Pascarella, Edison, Whitt et al., 1996; Pascarella, Flowers, & Whitt, 1999). Both are based on analyses of the National Study of Student Learning data, and both employ a standardized test of critical thinking skills (the critical thinking module from the Collegiate Assessment of Academic Proficiency) as the dependent variable. Pascarella et al. (1996) considered the impact of Greek affiliation during the first year of college at 18 four-year institutions. Statistical controls were introduced for an extensive set of potential confounding influences, including precollege level of critical thinking, academic motivation, student demographic characteristics, full- or part-time enrollment, work responsibilities, patterns of coursework taken, and the academic selectivity of the institution attended. In the presence of these controls, Greek-affiliated men, as compared to their male counterparts who remained independent, had a statistically significant disadvantage of .27 of a standard deviation (11 percentile points) in end-of-first-year critical thinking. The corresponding critical thinking disadvantage for Greek-affiliated women was only about 40% as large as that found for Greek-affiliated men and was not statistically significant.
Pascarella, Flowers, and Whitt (1999) followed the same NSSL sample through the end of the third year of college to determine if the critical thinking disadvantage accruing to Greek-affiliated men persisted beyond the first year. Thus, critical thinking scores at the end of the third year of college were the dependent variable. With statistical controls introduced for the same set of confounding influences as the Pascarella et al. (1996) study, the negative effect of fraternity membership on critical thinking found after one year of college was reduced by about 50% and become statistically nonsignificant. Similarly, the small, nonsignificant negative effect of sorority membership on critical thinking after one year of college remained small and statistically nonsignificant through the end of the third year.
Taken together, the NSSL-based studies suggest that any substantive negative effects of fraternity membership on male critical thinking occur during the initial year of college. Thereafter, the impact of fraternity membership on critical thinking becomes small and statistically nonsignificant. There is little evidence that being a member of a sorority has anything but small and nonsignificant effects on critical thinking skills. These conclusions are based on analyses of a single sample, however, and await replication.[6]
Intercollegiate Athletic Involvement
A small body of research has addressed the impact of intercollegiate athletic participation on objective, standardized tests of critical thinking. Although limited by the small number of studies, this research suggests: 1) that intercollegiate athletes may not be as predisposed as their nonathlete counterparts to think critically, and 2) that male football and basketball players (i.e., those involved in revenue-producing sports) may not be making the same gains in critical thinking skills as either nonathletes or athletes in nonrevenue sports.
McBride and Reed (1998) administered the New Jersey Test of Reasoning Skills (a standardized test purporting to measure critical thinking) and the California Critical Thinking Dispositions Inventory (which assesses the disposition to actually employ critical thinking skills) to small samples of intercollegiate athletes and nonathletes attending a university with a national-level athletic program. Irrespective of gender, athletes scored significantly lower on both tests than did nonathletes. Thus, intercollegiate athletes not only had significantly lower critical thinking skills, they also demonstrated a significantly lower disposition to actually utilize critical thinking skills than nonathletes.
The design of the McBride and Reed (1998) study makes it difficult to determine if the differences in critical thinking between athletes and nonathletes are attributable to participation in intercollegiate athletics during college or simply reflect differences on these traits that the two groups bring to college. To address just this type of methodological issue a series of analyses were conducted with the National Study of Student Learning (NSSL) data (Pascarella, Bohr, Nora, & Terenzini, 1995b; Pascarella, Truckenmiller et al., 1999). These analyses were based on student samples from 18 four-year institutions and employed the standardized critical thinking test from the Collegiate Assessment of Academic Proficiency as the dependent variable. Separate effects of intercollegiate athletic participation were estimated for men and women, with male athletes being further divided into those involved in revenue sports (football and basketball) and those involved in nonrevenue sports (all others). In the first study (Pascarella et al., 1995b), statistical controls were made for precollege level of critical thinking, academic motivation, student demographic characteristics, full- or part-time enrollment, on- or off-campus residence, the academic selectivity of the institution attended, and whether the institution attended fielded an NCAA Division I or a non-Division I athletic program. In the presence of these controls, there were only trivial and nonsignificant differences in end-of-first-year critical thinking scores between women athletes and women nonathletes and among male athletes in revenue sports, male athletes in nonrevenue sports, and male nonathletes.
A follow-up study of the same NSSL sample was conducted to determine if the same parity among intercollegiate athletes and nonathletes in critical thinking extended beyond the first year of college (Pascarella, Truckenmiller et al., 1999). The dependent variable was critical thinking scores at the end of three years of college, and, in addition to controlling for the same variables as in the earlier study (Pascarella et al., 1995b), statistical controls were also introduced for work responsibilities, time spent studying, and patterns of coursework taken. In the presence of these controls there were still only trivial and statistically non-significant differences between women intercollegiate athletes and their non-athlete counterparts on critical thinking after three years of college. There were also only small and non-significant critical thinking differences between male athletes participating in nonrevenue sports and male nonathletes. However, male revenue athletes (i.e., those participating in intercollegiate football and basketball) had significantly lower end-of-third-year critical thinking scores than either male nonrevenue athletes or male nonathletes. Compared to male athletes participating in nonrevenue sports, male football and basketball players had a net critical thinking disadvantage after three years of college of about .35 of a standard deviation (14 percentile points). The corresponding third-year critical thinking skills disadvantage for football and basketball players relative to nonathletes was about .39 of a standard deviation (15 percentile points).
As with previously reviewed findings from the Pascarella et al. (1995b; 1999) studies, summarized in Chapter 3, the negative effects on end-of-third-year critical thinking of participation in intercollegiate football and basketball were similar in magnitude, irrespective of one’s levels of precollege critical thinking, academic motivation, socioeconomic status, or ethnicity. In short, no particular type of male student received greater critical thinking penalties from participating in revenue-producing sports than any other. Furthermore, no evidence was found to indicate that the disadvantage in third-year critical thinking skills accruing to male football and basketball players differed in magnitude at NCAA Division I or non-Division I institutions or at institutions differing in the academic selectivity of their undergraduate student bodies. It is worth noting, however, that all the findings pertaining to the impact of intercollegiate athletic participation on critical thinking are based on analyses of a single sample and await replication.
In Chapter 3 we reviewed evidence suggesting that service experiences, particularly when integrated with course activities through a reflective component (i.e., service learning), may foster increased acquisition of subject matter knowledge. There is also evidence to suggest that such service learning experiences can contribute in positive ways to the development of students’ general cognitive skills and intellectual growth. For example, Batchelder and Root (1994) conducted a quasi-experiment in which experimental sections of a course were taught with an integrated service-learning component and the control sections were taught without the integrated service-learning component. Both the experimental and control courses were similar in content and were taught by the same instructors, although it was not clear if students self-selected themselves into the service-learning sections. At the beginning and at the end of the course, students wrote a 30 minute essay on how they would respond to two situations (e.g., child abuse, alcoholism, initiating a community recycling program) that were designed to assess their dimensions of thinking about a social problem. With statistical controls for pretest scores, students in the service-learning sections demonstrated a significantly greater increase in use of complex and multidimensional perspectives in their essays than did students in the course sections without a service-learning component. We converted the partial correlations reported by Batchelder and Root to an effect size and estimate that the advantage in more complex and multidimensional perspectives accruing to the service-learning students was approximately .25 of a standard deviation (10 percentile points).
An important series of multi-institutional studies by Eyler and her colleagues corroborate and extend the Batchelder and Root (1994) findings (Eyler & Giles, 1999; Eyler, Giles, Lynch, & Gray, 1997; Eyler, Giles, Root, & Price, 1997). However, these basically quasi-experimental investigations take a somewhat more fine-grained approach to understanding the impact of service learning on different dimensions of cognitive development. Instead of simply comparing service-learning with a control group, Eyler and her colleagues hypothesized that student cognitive development is influenced by the degree to which service learning classes are well integrated and highly reflective. Integrated and reflective service-learning classes tend to: 1) integrate service experiences with course content; 2) provide for reflection about the service experience; and 3) permit one to apply subject matter learning to the service experience, and vice versa. The results of their program of research tend to support this hypothesis. The greater the extent to which service-learning courses met the criteria for being highly integrative and reflective, the stronger the positive effects of service-learning on such outcomes as reflective judgment (Eyler & Giles, 1999; Eyler, Giles, Lynch et al., 1997) and the use of complex and multiple perspectives in identifying the causes of and formulating the solutions to specific social problems (Eyler, Giles, Root et al., 1997). The simple addition of service experiences to a course typically had little impact on cognitive outcomes. To have an impact these experiences had to be highly integrated into course content and provisions made for continuous reflection. The effects reported by Eyler and colleagues tended to persist even in the presence of statistical controls for precourse measures of the specific cognitive outcomes. Since the studies do not report all the requisite statistical information, however, we could not estimate an effect size.
There is also quasi-experimental evidence from undergraduate samples at 42 institutions that students involved in four different types of service experiences (education, human services, environment, and public safety) indicated greater self-reported gains in critical thinking skills than their counterparts not involved in service experiences (Astin, 1996a; Astin & Sax, 1996). This effect persisted even when statistical controls were introduced for such factors as volunteering in high school, precollege leadership ability, commitment to participating in community action programs, tutoring other students in high school, gender, and monetary orientation. Consistent results are also reported in a more recent study by Vogelgesang and Astin (2000).
Diversity Experiences
In the preceding chapter we presented evidence that involvement in diversity experiences (e.g., taking ethnic or gender studies courses, discussing racial issues, attending a racial/cultural awareness workshop, socializing with someone from a different racial/ethnic group) can have a positive influence on students’ academic skill development and knowledge acquisition during college. There is also replicated evidence, based on both standardized measures and students’ self-reports, suggesting that involvement in diversity experiences can modestly enhance more general cognitive skills and intellectual development. For example, analyzing the multi-institutional 1985-89 Cooperative Institutional Research Program data, Dey (1991) sought to identify experiences that influenced three measures of cognitive development: the Analytical subtest score on the Graduate Record Examination, self-reported growth in critical thinking ability, and self-reported growth in analytical and problem solving skills. With statistical controls introduced for such factors as SAT verbal scores, secondary school experiences, intellectual self-esteem, self-rated mathematics ability, and institutional characteristics, discussing racial/ethnic issues during the past year had a statistically significant, positive effect on all three measures of cognitive development. Markedly consistent results are reported by Terenzini, Springer, Yaeger, Pascarella, and Nora (1994) in their analyses of data from the 23 two- and four-year institutions participating in the National Study of Student Learning. Attending a racial/cultural awareness workshop had a small, but significant positive impact on end-of-first-year scores on a standardized test of critical thinking skills. This effect could not be explained away by differences in such factors as precollege critical thinking level, student demographic characteristics and aspirations, characteristics of the institution attended, full- or part-time enrollment, work responsibilities, patterns of coursework taken, or extracurricular involvement. Similarly, Kitchener, Wood, and Jensen (2000) found that growth on the Endorsement scale of the Reasoning About Current Issues Test, a measure of reflective thinking, was significantly and positively related to making friends and having discussions with students whose race was different from one’s own. The effect persisted even when initial scores on the RCI Endorsement scale were taken into account.
Analyses of a different iteration of the Cooperative Institutional Research Program data from that analyzed by Dey (1991) have also yielded evidence supporting the positive impact of involvement in diversity experiences on general cognitive development. Both Hurtado (1999) and Kim (1995; 1996; 2002a) analyzed data from the multi-institutional 1987-1991 CIRP sample, with Kim restricting the focus of her investigation to women. Both investigators introduced statistical controls for such factors as institutional selectivity, secondary school achievement, academic self-concept, study involvement, and leadership experiences during college. In the presence of such controls, measures of involvement in diversity experiences (e.g., studied with someone from a different racial/ethnic background, enrolled in an ethnic studies course, attended a racial/cultural awareness workshop) tended to have significant positive impacts on self-reported growth in critical thinking (Hurtado, 1999) and on a measure combining self-reported growth in both critical thinking and problem-solving skills (Kim, 1995, 1996, 2002a).
A more focused approach is reported by Gurin (1999) in two related single-institution studies. The first was essentially a quasi-experiment that compared students who took and did not take an introductory course in a program termed “Intergroup Relations, Community, and Conflict” (IRCC). The IRCC course covered the history of group experiences in the United States, a contemporary analysis of group inequalities in economic, political, and educational arenas, and an analysis of political issues and policies (e.g., immigration, bilingual education, affirmative action, sexual harassment, Middle East peace initiatives, etc.) that are contested by various groups in American society. In addition to lectures and discussion sections, students who took the IRCC course also took part in an ongoing dialogue group designed to: 1) examine between-group and within-group differences on a contested issue; 2) identify and negotiate group conflicts; and 3) find a basis for joint action on an issue. Students who took the IRCC course and the control group who did not were matched in such areas as first-year residence hall, race/ethnicity, gender, and in-state and out- of-state residency prior to college. At entrance to college and at the time of graduation, both groups completed a self-report index designed to measure complexity of thinking (e.g., prefer complex rather than simple explanations, enjoy discussions of causes of behaviors). With statistical controls for precollege level of complexity of thinking, students who completed the IRCC course had significantly higher complexity of thinking levels at the end of college than the control group students who did not. (An effect size could not be estimated from the results reported.) Similar results are reported by Adams and Zhou-McGovern (1994) with respect to a social and cultural diversity course that focused on such topics as racism, sexism, and social justice. Students exposed to the course made advances of about 1 standard deviation (34 percentile points) on the Perry (1970) continuum, but there was no control group.[7]
The second study conducted by Gurin (1999) attempted to determine what diversity experiences other than the IRCC course influenced the complexity of thinking index. A more comprehensive sample was employed, and it was assessed when it entered the institution (1990) and once again prior to graduation. Statistical controls were introduced for such factors as entering level of thinking complexity, SAT/ACT score, high school grades, gender, and racial composition of one’s high school and home neighborhood. With these factors taken into account, a number of diversity experiences had significant positive effects on complexity of thinking at the end of college for White students. These experiences included: 1) classroom diversity (extent to which students reported they had been exposed to diversity issues in classes); 2) personal interactions (extent to which interactions with other racial/ethnic groups involved “honest discussions about race” and “sharing of personal feelings and problems”); 3) number of multicultural events attended (e.g., Black History Month, Hispanic Heritage Celebration, etc.); and 4) participation in dialogue groups (an ongoing program of intergroup dialogue and conflict resolution at the institution). Diversity experiences had no significant effects on thinking complexity at the end of college for African-American students. This may not mean that African-American students receive smaller cognitive benefits from diversity experiences than White students, however. As far as we could tell, Gurin did not report statistically significant differences between regression coefficients for White and African-American students. Thus, the failure to find significant diversity experience effects on thinking complexity for African-American students may be attributable in large measure to the weak statistical power associated with the relatively small number of African-American students in the sample.
The body of evidence concerning the impact of work during college on the development of general cognitive skills and intellectual growth is inconsistent. Some analyses of the National Study of Student Learning data have found small negative effects of hours worked per week on a standardized measure of end-of-first-year critical thinking (Inman & Pascarella, 1998; Terenzini, Springer et al., 1994).