By: Anthony Marini
One of my roles as Senior Teaching Associate at the EDC is to consult with instructors around issues related to using multiple-choice test items in their assessment activities. As a faculty member with a focus on educational measurement and assessment, I have worked with many instructors in a wide range of disciplines, helping them to establish the appropriate use of this testing format.
The decision to use multiple-choice testing is often deliberate, but in many circumstances the choice has more to do with practical issues, such as large class sizes. Regardless of the motivation, helping instructors recognize the value of the information this testing tool provides is a very satisfying part of my work here at Carleton.
When an instructor completes a Scantron request form at the EDC, there is an option to request an item analysis consultation, which involves a meeting at the EDC or over the phone. An item analysis provides a statistical summary of how the group performed on the test the overall, including the mean for the exam, the minimum and maximum score and the reliability of the test. It also provides information for each question on the test, including the percentage of students choosing the correct option, the distribution of choices for each option and how well the item discriminated between the students achieving the highest overall scores on the test and those obtaining the lowest total scores on the test.
Best practice in the field of testing and assessment provides guidelines to determine the appropriate statistical characteristic of test items. These guidelines serve as a mechanism for identifying test items that I would like to discuss with the instructor.
On to the intriguing part
During the consultation, the instructor and I review each item on the test in terms of overall class performance. Generally, the percent correct (i.e. percentage of all students getting the item correct) that is recommended for a multiple-choice item is between 20% and 90%. When less than 20% of the students get the correct answer, the item should be reviewed for possible deletion from scoring or refinement before using it on subsequent exams. The process of reviewing items to ensure that they have performed appropriately is an important step in ensuring that the final scores students obtain are fair and valid.
Instructors are often fascinated by the choices students make if they do not pick the correct response. The pattern of incorrect responses can lead an instructor to identify the confusion students are encountering, which can be helpful the next time they are teaching that particular concept.
Instructors also find information about the choices that the strongest students make versus the weakest students (these groups are defined in terms of their total score on the exam) quite interesting. Again by noting the incorrect choices the weaker students select, the instructor can hypothesize on the difficulties these students are encountering and address these issues the next time they are teaching the material.
The consultation also gives me an opportunity to review principles of effective item construction and help the instructor produce multiple-choice test items that make a meaningful and valid contribution to the assessment process.
Item analysis and consultations with instructors can really enhance the contribution multiple-choice tests can make to the teaching and learning process. If you would like to book a consultation, please contact the EDC at edc@carleton.ca, extension 4433.