Validating College-level Reading Placement Test Standards
Anthony R. Napoli, Ph.D.
Suffolk Community College & the State University of New York at Stony Brook
Paul M. Wortman, Ph.D.
State University of New York at Stony Brook
Abstract
College placement tests are being increasingly used as indices of academic preparedness for placing students in remedial courses prior to enrollment in traditional programs of study. Such tests must conform to accepted and established psychometric practices to be considered valid. In particular, test users must empirically establish the criterion-related predictive validity of the decision or cutoff points used in making placements. The present report describes results of an ongoing research effort to empirically establish the cutoff points for the CPT Reading Comprehension Test (CPT-Read) at Suffolk Community College (SCC).
Introduction
The establishment of valid placement test standards to serve as indices of academic preparedness for exemption from remedial course work and entry into traditional academic programs of study must conform to accepted and established psychometric practices. In addition to selecting an assessment instrument with empirically substantiated construct validity (Nunnally, 1978) and reliability (Cronbach, 1970), test users must empirically establish the criterion-related predictive validity (Anastasi, 1982) of the decision or cutoff points on a placement test's score distribution. The present report describes results of an ongoing research effort to empirically establish the cutoff points for the CPT Reading Comprehension Test (CPT-Read; 1990) at Suffolk Community College (SCC).
The College Board/Educational Testing Service (1990) has recommended that colleges establish cutoff scores which meet the specific and individual requirements of the institution. Two criteria were considered meaningful and appropriate to validate CPT Reading test score cutoffs, performance in Introductory Psychology and overall Grade Point Average (GPA). The psychology course was selected since it relies heavily on reading (typically 650 pages per semester) to cover its broad content areas. Success in the course, therefore, serves as an index of reading skills proficiency commensurate to the demands of college curricula. From a practical perspective, the high enrollment associated with the course would substantially guarantee (far beyond any other survey course offered at SCC) sufficient data for quantitative analysis. The inclusion of GPA as a validating criterion is based on its overall importance with regard to program completion
and its ability to convey, in a global manner, success with college-level academic demands.
It was expected that both performance in the psychology course and overall GPA would be significantly related to CPT-Read scores. If sufficient correlation between the measures exists, then regression analysis could identify those points on the CPT-Read distribution that predict academic outcomes with a high level of statistical accuracy. Once identified, those points would serve as the cutoff by which students would be placed into or exempted from a remedial reading course. This paper reports on these analyses and the resulting cutoff points identified.
Method
Sample/Procedure
The academic records for approximately sixteen thousand (16,000) students, attending classes at SCC during the years 1988 through 1990, were electronically searched to identify those individuals who had CPT-Read scores, and who had not been placed into a remedial reading course. These dual criteria identified 1,450 who students served as the sample for the current study. It's important to note that although SCC had been employing a CPT-Read cutoff score of 90 and above to exempt students from taking remedial reading courses, fifty percent (50%) of the sample (or 722 students) had CPT-Read scores below 90.^{1} The mean CPT-Read score for the sample equalled 86.1, the standard deviation equalled 16.3. The range of CPT scores equalled 27 to 120, which closely approaches the actual limits of the test which range from 20 to 120. An analysis
of the form of the CPT-Read distribution, employing Pearson's coefficient of skew, failed to detect any significant deviation from normality (sk = .34).
Results
Results for Introductory Psychology
Pearson's Product-Moment correlation coefficient was employed to assess the relationship between the predictor variable (CPT-Read) and targeted criteria variables. As seen in Table 1, CPT-Read scores were found to be reliably related to psychology grades (PSYGPA; r = .519, p < .0001). To assess the possibility that CPT-Read scores and PSYGPA may be related in a non-linear or higher-order manner (Pedhazur, 1982) hierarchical regression was employed in which PSYGPA was regressed on the quadratic, cubic, quartic, and quintic forms of CPT-Read. Results from the regression model failed to detect any significant higher-order relationships and confirms the assumption of a linear relationship between CPT-Read and PSYGPA.
Results for the simple regression of PSYGPA on CPT-Read are presented in Table 2. Employing the general linear regression equation (Y' = A + bX), the next phase of analysis applied the regression constants (see Table 2) to the raw CPT-Read scores to produce predicted PSYGPA scores. This was done for each case in the sample. To determine the degree of concordance between predicted psychology grades and actual
grades, both score distributions were dichotomized to form groups of students who scored grades of "C" and above and grades below the "C" level.
The crosstabulation of predicted and actual grade dichotomies is presented in Table 3. Results indicate, that approximately seventy-six percent (76%) of the 281 students who received grades below "C," were correctly classified based on their CPT-Read scores. Slightly more than seventy-seven percent (77%) of the 1,169 students who received "C" or better grades were correctly identified. Overall, a seventy-seven percent (77%) concordance rate was observed between actual and predicted psychology grade outcomes (p < .000001, Fisher's Exact Test).
To locate the cutoff point on the CPT-Read distribution which is predictive of the "C" or better, or below "C" outcome in the psychology course, we employed the general linear regression equation. Listed in Table 4 are select CPT-Read scores and corresponding predicted psychology grades. The regression equation identifies a CPT-Read score of 75 and above as the point on the CPT-Read distribution that is predictive of the "C" and better psychology grades. CPT-Read scores below 75 predict below "C" performances in the psychology course.
Results for Overall Grade Point Average
Appearing in Table 1 are the correlation coefficients for overall GPA with CPT-Read, and total credits completed (Credits). As expected the CPT-Read scores are positively and significantly related to GPA (r = .41, p < .0001). Results from a hierarchial regression analysis failed to detect any significant higher-order relationships, and again confirm a linear association between CPT-Read and GPA. In addition, GPA is also significantly dependent (r = .270, p < .0001) on Credits. Since GPA has been shown to increase as the number of credits increases (Goldman & Gillis, 1989; Jesse & Gregory, 1987), hierarchical regression analysis (Tabachnick & Fidell, 1983) was employed to control for confounding influence of Credits on overall GPA. In the hierarchical regression model, the variable Credits is entered on the first step of the analysis. This procedure produces residualized GPA scores which are adjusted for total credits completed. On a second step of the regression model, the residualized GPA scores were regressed on CPT-Read to produce the adjusted regression constants (regression coefficient and intercept) appearing in Table 5.
The degree of concordance between predicted GPA and actual GPA, was determined following the procedure described above. Both score distributions were dichotomized to form two groups of students, those who scored grades of "C" and above and those scoring below the "C" level. The crosstabulation of predicted and actual grade dichotomies is presented in Table 6. As seen in the table, of the 157 students who actually received grades below "C," approximately fifty-six percent (56%) were correctly classified based on their CPT-Read scores. Slightly more than seventy percent (70%) of the 1,293 students who received "C" or better grades were correctly identified. Overall, the sixty-nine percent (69%) concordance rate observed between actual and predicted grade outcomes is a highly significant finding (p < .000001, Fisher's Exact Test).
Appearing in Table 4 are select CPT-Read scores and corresponding predicted GPA scores obtained from the regression model. The model indicates that a CPT-Read score of 70 and above is predictive of "C" and above grade point averages. CPT-Read scores below 70 predict below "C" GPA performances.
Summary
The objective of the current research effort was to identify CPT-Reading Comprehension test standards to serve as decision points for either placing students into or exempting them from remedial reading courses. The study employed criterion-referenced performance outcomes, specifically, performance in introductory psychology classes and overall grade point averages, to serve as indices of reading comprehension proficiency. Upon confirming the presence of significant correlation between the predictor and criterion outcome variables, regression analysis and statistical modeling was employed to predict performance outcomes from CPT-Reading Comprehension Test scores. Actual and predicted grade distributions were then dichotomized to produce performance outcome categories of At/Above "C" Grade Levels and Below "C" Grade Levels to assess the agreement rates between predicted and actual grades. A comparison of predicted and actual performance categories demonstrates significantly high concordance rates of 77% and 69% for psychology and overall GPA, respectively. Importantly, results for both sets of analyses suggest that, and converge on, a CPT-Read score in the mid seventies (75) would serve as a reliable "cutoff point" for the SCC population.
In the present study, the application of statistical modeling and verification techniques produced significant and meaningful enhancements to the placement decision process. By selecting criterion-based performance outcomes it became possible, with a high level of statistical accuracy, to identify CPT-Reading Comprehension placement test standards which convey performance capabilities. Whether the standards identified for the SCC population would generalize to other institutions is not clear and should be the subject of further inquiry.
Based on the above results, the SCC Developmental Studies Committee decided to
adopt a CPT-Read cutoff score of 80, and has used this score for the past two- years.
Although a score of 80 is slightly higher than the identified value of 75, it was chosen to accomodate for placement errors which typically occur around the cutoff point. It also reflects a greater tolerance among committee members to err on the side of a developmental reading placement. Specifically, for the results presented on overall GPA we observed a discordance rate of approximately 30 percent between actual and predicted GPA categorization. When disagreements occurred, however, they were 5.5 times more likely to be of a false positive type, and would result in a developmental placement.
It is noteworthy that the impact of lowering the cutoff score from 90 to 80 has been to reduce enrollment in developmental reading classes by 24 percent, or approximately 700 students. This reduction is based on the distributional properties of CPT-Read scores at SCC, where a score of 90 is at the seventy-fourth percentile and the lower score of 80 is at the fiftieth percentile. The cutoff adjustment has not only lead to a better and more justifiable placement model, but also represents an economic savings to the college and to those students who would have been unnecessarily required to enroll in a developmental reading course.
A recent evaluation indicates that the remedial reading program has had the desired beneficial effect, and suggests that exposure to the reading program produces meaningful enhancement in reading comprehension levels (Napoli, Wortman, & Norman, 1994). Future assessment efforts will require continued monitoring of students' academic performance and progress to assure that the adopted cutoff points continue to serve as valid indices of academic preparedness.
This paper illustrates a simple procedure for validating standards for determining assignment of students to remedial programs. Given the nationwide emphasis on improved education, the determination of such standards is increasingly important in matching student needs to limited resources.
References
Anastasi, A. (1982). Psychological Testing (5th ed.). New York: Maxmillan.
College Entrance Examination Board and Educational Testing Service (1990).
Coordinator's Guide for Computerized Placement Tests, Version 3.0. Princeton, NJ: Author.
Cronbach, L. (1970). Essentials of Psychological Testing (3rd ed.). New York: Harper and Row.
Goldman, B.A., and Gillis, J.H. (1989) Graduation and attrition rates: A closer look at influences. Journal of the Freshman Year Experience, 1(1), 65-77.
Jesse, D.M., & Gregory, W.L. (1987) A comparison of three attribution approaches to maintaining first-year college GPA. Educational Research Quarterly, 11(1), 13-25.
Napoli, A. R., Wortman, P. M., & Norman C. (1994). A Critical Multiplist Evaluation of Developmental Reading Instruction at Suffolk Community College. Colleague., 16-25.
Nunnally, J. (1978). Psychometric Theory (2nd ed.). New York: McGraw-Hill.
Pedhazur, E.J, (1982). Multiple Regression in Behavioral Research. New York: Holt, Rinehart, & Winston.
Tabachnick, B. G., and Fidell, L. S. (1983). Using Multivariate Statistics. New York:
Harper and Row.
Table 1: Correlation Coefficients.
PSYGPA | GPA | M | SD | |
CPT - READ | .52* | .41* | 86.1 | 16.3 |
Credits | - | .27* | 32.9 | 18.5 |
PSYGPA | - | - | 2.5 | 1.1 |
GPA | - | - | 2.76 | .71 |
* p.<.0001, df = 1448
Table 2: Regression of PSYGPA on CPT-Read.
Variable | Rē | b | A | t |
CPT - Read | .2690 | .0346 | -.4879 | 23.08** |
**p.<.0001, df = 1448
Table 3: Crosstabulation of Actual by Predicted Psychology Grades.
Predicted Grades
Actual Grades | Below "C"
n (row %) |
"C" & Above
n (row %) |
Row Totals
n (row %) |
< "C" | 214 (76.2%) | 67 (23.8%) | 281 (19.4%) |
"C" | 264 (22.6%) | 905 (77.4%) | 1169 (80.6%) |
Column Totals | 478 (33.0%) | 972 (67.0%) | 1450 |
Total Cases Correctly Classified = 77.2%.
Fisher's Exact Test, p. < .000001.
Table 4: Predicted Psychology Grades and Overall Grade Point Averages for Selected CPT-Reading Comprehension Scores.
CPT - Read | Predicted Psychology Grade (letter grade) | Predicted GPA Grade (letter grade) |
40 | .90 (F) | 1.52 (D+) |
45 | 1.07 (D) | 1.61 (D+) |
50 | 1.24 (D) | 1.70 (D+) |
55 | 1.42 (D) | 1.80 (D+) |
60 | 1.59 (D+) | 1.89 (D+) |
65 | 1.67 (D+) | 1.98 (D+) |
70 | 1.93 (D+) | 2.08 (C) |
75 | 2.11 (C) | 2.17 (C) |
80 | 2.28 (C) | 2.26 (C) |
85 | 2.45 (C) | 2.35 (C) |
90 | 2.63 (C+) | 2.45 (C) |
95 | 2.80 (C+) | 2.54 (C+) |
100 | 2.97 (C+) | 2.63 (C+) |
105 | 3.15 (B) | 2.73 (C+) |
110 | 3.21 (B) | 2.82 (C+) |
115 | 3.50 (B+) | 2.91 (C+) |
120 | 3.66 (B+) | 3.00 (B) |
Table 5: Regression of GPA on CPT-Read controlling for credits
Variable | Rē | b | A | t |
CPT - Read | .251 | .0185 | .777 | 18.46** |
**p.<.0001, df = 1447
Table 6: Crosstabulation of Actual GPA by Predicted GPA.
Predicted Grades
Actual Grades | Below "C"
n (row %) |
"C" & Above
n (row %) |
Row Totals
n (row %) |
< "C" | 88 (56.1%) | 69 (43.9%) | 157
(10.8%) |
"C" | 377 (29.2%) | 916 (70.8%) | 1293
(89.2%) |
Column Totals | 465 (32.1%) | 985 (67.9%) | 1450 |
Total Correctly Classified = 69.2%.
Fisher's Exact Test, p. < .000001.
^{ 1} These students had successfully petitioned the Dean of Instruction to be excused from developmental reading courses based on their high school grades and standardized test scores.