Evaluating the consistency of verbal reports and the use of cognitive models in educational measurement

Abstract

In the field of psychology, verbal reports are commonly used as a data source to explain human information processing. To date, few studies have investigated the accuracy of verbal reports for providing information on students’ reasoning and problem solving on educational tasks. The purpose of this study is to evaluate the consistency of verbal data, as well as the effects of student achievement, interviewer knowledge level, and item difficulty on the consistency of verbal reports. Seventy-one Grade 12 students from two high schools provided verbal responses to 15 multiple choice test items from the Alberta Pure Mathematics Diploma Examination. Results indicate higher-achieving students demonstrate greater consistencies in verbal reports than moderate achieving students. The implications of the results are discussed and the limitations of the present study are also presented.