Five go marking an exam question: the use of Adaptive Comparative Judgement to manage subjective bias

Barber, Jill (2018) Five go marking an exam question: the use of Adaptive Comparative Judgement to manage subjective bias. Practitioner Research in Higher Education, 11 (1). pp. 94-100.

Preview

PDF - Published Version Available under License CC BY-NC Download (554kB) | Preview

Abstract

Adaptive Comparative Judgement (ACJ) is an alternative to conventional marking in which the assessor (judge) merely compares two answers and chooses a winner. (Scripts are typically uploaded to the CompareAssess interface as pdf files and are presented side-by-side.) Repeated comparisons and application of the sorting algorithm leads to scripts sorted in order of merit. Boundaries are determined by separate review of scripts. A small pilot of ACJ in the fourth year of the Manchester Pharmacy programme is described. Twelve judges used ACJ to mark 64 scripts previously marked conventionally. 50 students peer-marked their own mock examination question using ACJ. Peer-marking was successful with students learning from the process, and delivering both marks and feedback within two weeks. There was very good consistency among the students acting as judges, and accuracy (as defined by Pollitt, 2012) of 0.94. Staff were similarly consistent, but the agreement with marks obtained by conventional marking was disappointing. While some discrepancies could be attributed to conventional marking failing under the stress of marking during teaching term, the worst discrepancies appeared to originate from inadequate judging criteria. We conclude that ACJ is a very promising method, especially for peer assessment, but that judging criteria require very careful consideration.

Item Type:	Article
Journal / Publication Title:	Practitioner Research in Higher Education
Publisher:	University of Cumbria
ISSN:	1755-1382
Departments:	Academic Departments > Institute of Education (IOE) > Initial Teacher Education (ITE)
Additional Information:	Jill Barber, University of Manchester, UK.
Depositing User:	Anna Lupton
Date Deposited:	24 May 2018 11:46
Last Modified:	12 Jan 2024 19:46
URI:	https://insight.cumbria.ac.uk/id/eprint/3846

Downloads