The peer rating system used here advances the quantitative literacy goals outlined in social sciences. We instituted a mid-semester intervention to teach rating skills and used an objective index to track longitudinal changes of skill mastery over the course of the semester. Seventy-four students in five advanced research classes followed the procedure of the existing peer rating system by completing reading assignments, writing reflections online, engaging in class discussions, rating their peers’ reflections, and receiving feedback of their group effort. Unique to our modified system, peer ratings were compared with each other and also with the instructor ratings to derive individualized indices of reliability and validity. These technical indicators enabled two rounds of assessment before and after a class-wide intervention. An omnibus test across the five classes showed a significant improvement in rating quality due to the intervention. Our courses not only met a quantitative learning outcome but also promised vocational competence.
Cheung-Blunden, V., Khan, S.R. A modified peer rating system to recognise rating skill as a learning outcome (2017) Assessment and Evaluation in Higher Education, pp. 1-10. Article in Press. DOI: 10.1080/02602938.2017.1280721