Quantitative instruments include:
1. Pre and Post Student Surveys
2. Beginning of the Semester Pre-Quiz
3. Daily Short Quizzes
4. Content Module Tests
5. Comprehensive Final Exam
The pre and post student survey instruments can be viewed using the links below:
-
The ‘Student Self-Efficacy for Cognitive Ability’ survey is based on Bloom’s taxonomy, which is a multi-tiered model that measure six levels of cognitive ability including knowledge, comprehension, application, analysis, synthesis, and evaluation. This survey includes 30 questions measuring students’ degree of self-efficacy to remember and understand course content as well as to solve, analyze, evaluate, and create a problem related to the main topics presented in the course. Students were asked to rate themselves on a five-point Likert scale where “1” stands for “cannot do at all” and “5” stands for “certainly can do”.
-
A second ‘Student Self-Efficacy for Application of Knowledge’ survey includes 21 questions that were developed by the research team to measure student self-efficacy to accomplish specific tasks associated with the content in the course. Students were asked to rate themselves using the same Likert scale.
-
The last survey evaluated ‘Self-Regulated Learning Strategies’ and included 13 questions that were developed according to Zimmerman’s social cognitive theoretical framework of self-regulation to measure student use of self-regulated learning strategies in college. Students were asked to report the frequency that they used the 13 strategies described in the survey where “1” stands for “not at all” and “5” stands for “all the time”.
The following figure summarizes the structure of the quantitative assessment plan including the 80-point pre-quiz conducted at the beginning of the semester, the daily quizzes through the semester for each of the four content modules, the four content module tests, and the comprehensive final exam for the course.
An 80 point, multiple-choice pre-quiz covering material from all four content modules was conducted at the beginning of each semester to assess prior knowledge of the material and provide a baseline for each student participant. It was expected that the students would have little (if any) prior knowledge of the subject matter unless they had previously taken the class and/or had geotechnical work experience. Throughout the semester, students were given short quizzes at the end of most lecture periods. These quizzes were comprised of short answer and true-false questions. The type and difficulty of the question(s) presented on each short-quiz paralleled the type and difficulty of the questions presented on the pre-quiz to ensure parallel form reliability. As a result, the five short-answer questions on the short quiz were compared to parallel questions on the pre-quiz to assess comprehension (per lecture).
A test was conducted at the end of each content module. Exactly 20% of each test included short answer questions that were parallel in type and difficulty to the questions presented on the short quizzes and the pre-quiz previously described. The remaining 80% portion of the test required a higher level of problem solving ability. The comprehensive final exam conducted at the end of the semester was longer but had the same format. The final exam was organized to reflect equal points from each content module, and it consisted of 20% short answer questions with 80% work out problems that paralleled the type and difficulty of the content module tests.
Retention and comprehension was assessed by comparing the answers from the pre-quiz to the results of the short answer problems on the four content module tests and the final comprehensive exam. The remaining 80% of each content module test will be compared to parallel sections on the final exam to assess retention during the semester. Finally, the results from the ‘Control Group’ participants during the first two semesters were compared to the results from the ‘Treatment Group’ participants during the last two semesters to determine the numerical impact of the newly implemented GCT. It is important to note that points were assigned to each question ahead of time to eliminate subjectivity during the grading process.
While GCT were not implemented until Semester 3 and Semester 4, the same quantitative instruments were used during all four project semesters. The instructor teaches this course every year during the fall and spring semesters, but quantitative data (e.g., quiz and test grades) acquired during semesters prior to this project cannot be used since the course structure and instrumentation for this project was specifically organized so that a clear evaluation could be performed between the ‘Control’ and ‘Treatment’ years. Additionally, prior to this project, the instructor continuously modified her curriculum, assessment tools, and teaching methods with each semester as she developed her skills as an educator so consistency from semester to semester may not exist. For these reasons, it was critical that an equivalent ‘Control Group’ was assessed during the project for a valid comparison with the ‘Treatment Group’.
Quantitative data from criteria-based assessments was analyzed using statistical procedures. While all quantitative instrumentation questions on the quizzes and tests are identical for ‘Control’ and ‘Treatment’ Groups, it is important to note that this methodology assumed the overall intellects of the students across all four semesters were approximately equivalent and, therefore, comparable when analyzed as a ‘Control Group’ versus a ‘Treatment Group’. This was a fair assumption since previous enrollment numbers and student demographic information did not appear to change significantly from year to year over a time period consisting of only four semesters.
Repeated-measures t-tests were employed to measure statistically significant gains of ‘student comprehension per lecture’ measured by the short-quizzes against their prior knowledge of the fundamental concepts measured by the pre-quizzes. Recall that each pre-quiz evaluated their knowledge of the material to be covered over the following comparative content module, and the mean student scores on each quiz were compared to the mean scores for each corresponding section in the pre-quiz.
Repeated-measures t-tests were also employed to measure statistically significant gains of ‘student retention during the course’ as measured by the four content module tests conducted each semester against their performance on the final exam. The final exam contained most of the content covered in each of the previous tests, and the mean student scores on each quarterly test were compared to the mean scores for each corresponding section in the final exam.
Independent sample tests were employed to compare students’ comprehension and retention of the material in the same way as described for repeated-measures but not within the same group of the students. Instead, ‘treatment group’ student data from the second two semesters will be compared to ‘control group’ student data collected during the first two semesters, cross-sectionally.
Student data on criteria-based assessments were analyzed using two methods. First, participating students were considered as individual samples during each semester. Second, participating students were considered as a complete sample during each academic year, which includes a fall and spring semester. This identified possible differences between the students who enroll in the fall and the spring semesters (if any) and also aided in the formative evaluation. If no differences were identified between the fall and spring semesters for each year, then all students in a single academic year were merged into one large group to increase the sample size and the statistical power of the analyses. Mixed design Analysis of Variance (ANOVA), also known as Split-Plot ANOVA, as well as t-tests will be used for statistical analyses, and a family-wise alpha-level of 0.05 will be used for statistically significant difference.
