June 9, 2003

News Bytes
Institutional Research Spotlight: Grades in Prior Basic Skills Courses Strongly Influence Success in Subsequent Ones

According to a new Institutional Research (IR) study approach, students who complete English Writing (EWRT) 1A or English as a Second Language (ESL) 5 successfully complete introductory psychology, economics and sociology courses at much higher success rates than those who do not take or succeed in EWRT 1A or ESL 5.

Researcher Andrew LaManque recently outlined an approach to validate the statistical association between successfully completing a prescribed course before attempting another course. The study suggests using this approach as one of multiple measures to support the effectiveness of a course advisory. A course advisory is official advice to students, noted in the catalog, that prior or concurrent enrollment (and success) in a specified course or courses is highly recommended though not required.

The methodology answers the following question: Do students who have passed a pre-course before completing the target course succeed in the target course at a higher rate than students without this preparation?

Definitions

  • Target Course: A course for which mastery of a certain set of knowledge or skills would be useful prior to enrollment.
     
  • Pre-Course: A course that provides the student with learning useful for successfully completing other courses in a future term.
     
  • Success: A student is considered successful if they receive and A, B, C or P grade in the course.
     
  • Non-Success: A student is considered to have not succeeded in a course if they receive a grade of D, F, NP, I or W in the course, or if they do not attempt the course.
     

Assumptions

  • A student who attempts a pre-course and is non-successful is considered to have the same level of course knowledge of a student who has not attempted the pre-course.
     
  • Since the focus of the review is on the impact of the pre-course on the target course, only the first attempt of the target course will be considered, while only the last attempt of the pre-course will be included.
     
  • The test of validation assumes that the content taught in the target and pre-courses has remained and will remain consistent across time and instructors.
     
  • Advisories listed as a choice (“or”), for example, “EWRT 1A or ESL 5” should be considered fulfilled if the student has succeeded in either course.
     
  • Advisories listed as multiple criteria (“and”), for example, “EWRT 1A and MATH 105” should be considered independent of each other in the analysis of significance.
     
  • The review assumes that the sample data consists of students who have randomly chosen (or not) the pre-course and that the selection of the pre-course is not biased towards students, on average, with better overall academic preparation.

How the Methodology Works

The first step is setting up a 2 x 2 matrix containing the raw scores that track the number of students who have passed and not passed the target course, and whether they have passed or not passed the pre-course.

Figure 1: 2 X 2 Matrix Example

Figure 1Click here/image
for larger view.

 

Time Period: When possible, examine enrollment in the target course for at least six academic year quarters. Consider enrollment in the pre-course for the six quarters under review for the target course, as well as six academic years quarters prior to that time. A student’s enrollment for the most recent pre-course should have taken place before the enrollment term of the target course.

Test of Statistical Significance: Use the Chi Square test to determine whether the 2 x 2 matrix is considered statistically significant. The test will examine whether there is enough of a difference to reject the null hypothesis that there is no difference between the success rate in the target course for those students with and without the pre-course. You may use a 0.10 level of significance as a guide to whether there is a difference.

Example Analyses: Eligibility for EWRT 1A or ESL 5 is suggested prior to enrollment in General Psychology (PSYC 1), Principles of Macroeconomics (ECON 1) and Introduction to Sociology (SOC1). To demonstrate this approach, the presentation below examines students who have completed EWRT 1A or ESL 5 (rather than EWRT 100B/ESL 163 or equivalent). This offers one way to examine whether a prescribed pre-course has an association with success in a target course.

Time Period and Methodology: The following terms (end of term data) are considered for the target course:

  • Greater than or equal to Winter 1999
     
  • Less than or equal to Winter 2002
     

The following terms are considered for the pre-course:

  • Greater than or equal to Winter 1996
     
  • Less than or equal to Winter 2002
     

“We pull all enrollment records for students in the target course for the terms under study from the end of term database,” said LaManque. “We then run an additional query to select the first enrollment for the target course for each student from the initial file. The result of the two queries is a file containing one record per student with information on the success in the target course.”

 

About Institutional Research

This regular column highlights an institutional research (IR) study conducted by the staff of the Institutional Research and Planning office (IR&P) in each News Bytes issue. The goal is to inform more people about IR research so that more effective decisions can be made throughout the District.

The IR function supports a “culture of evidence” through comprehensive information, data and research. College administrators, faculty and staff use these results for decision-making, planning, evaluation and reporting -- to maintain and continuously improve programs, processes and structures to better serve and educate students.

You can get more information about IR, including complete reports, program reviews and research data, on the IR&P site (at the above link) or by contacting Executive Director of IR&P Bob Barr at barrbob@fhda.edu

All enrollment records for either of the pre-courses are pulled for the students listed as having enrolled in the target course. An additional query is then run to select the last enrollment for the pre-course for each student, from this initial file. The result of the two queries is a file containing one record per student with information on the success in the pre-course.

In the final step, the two files are merged by student identification number to produce a one-record-per-student file of all students who had enrolled in the target course and whether or not they had enrolled and succeeded in the pre-course. This information is then summarized into a 2 x 2 matrix.

Results

The results for three course examples are listed in Figures 2, 3 and 4 below:

Figure 2: PSYC 1 Matrix Example

Click here/image
for larger view.





Chi-square = 472.59
p is less than or equal to 0.001.
The distribution is significant.

Interpretation

In this case, we find that only a small percentage of students who successfully completed the pre-course were not successful in the target course. 84 percent of the students completing the pre-course successfully completed the target course, compared with 64 percent success for students without the pre-course. For PSYC 1, successful completion of EWRT 1A can be seen a reliable indicator of success, although a high portion of students without EWRT 1A do successfully complete PSYC 1.

Figure 3: ECON 1 Matrix Example

Click here/image
for larger view.





Chi-square = 100.52
p is less than or equal to 0.001.
The distribution is significant.

Interpretation

In this case, we find that 25 percent of the students who successfully completed the pre-course were not successful in the target course. 75 percent of the students completing the pre-course successfully completed the target course, compared with 61 percent success for students without the pre-course. For ECON 1, successful completion of EWRT 1A can be seen a reliable indicator of success, although a high portion of students without EWRT 1A do successfully complete ECON 1.

Figure 4: SOC 1 Matrix Example

Click here/image
for larger view.





Chi-square = 99.56
p is less than or equal to 0.001.
The distribution is significant.

Interpretation

In this case, we find that less than 18 percent of the students who successfully completed the pre-course were not successful in the target course. 82 percent of the students completing the pre-course successfully completed the target course, compared with 67 percent success for students without the pre-course. For SOC 1, successful completion of EWRT 1A can be seen a reliable indicator of success, although a high portion of students without EWRT 1A do successfully complete ECON 1.

Summary

This study outlines an approach to examine the success of students in one course as a function of success in a previous related course. Using three courses and the same pre-course as examples, the study demonstrates the strong association between the success of students completing EWRT 1A or ESL 5 and success in PSYC1, ECON1 and SOC1A. Slightly lower success rates were seen with ECON1, which may be an indication that math skills are relatively more important for success in ECON1 than in PSYC1 or SOC1A.

In all three cases, additional research will be needed to isolate the academic factors associated with success in the three social science courses used as examples. While EWRT 1A or ESL 5 are not listed as an advisory to these three courses, they were selected as an example in this paper to demonstrate a general approach that can be used when examining the impact of completion of one course on success in a future course.

A future analysis might examine the impact of successfully completing EWRT 100B or ESL 163 (or placement equivalent) to determine whether the statistical evidence is strong enough to warrant advisory status.

Visit the ETS Web site to view the full study, which is in a Word document format, or contact Andrew LaManque at lamanqueandrew@fhda.edu if you would like to apply this methodology to any other courses in the Social Science/Humanities divisions or if you would like more information.

rule
ETS - Educational Technology Services

[Home]