EAG@IOE: Do we need marking any more? Psychometrics, large language models, and educational assessment

Dr Alexander Scharaschkin (AQA) Educational assessment can take many forms, but always involves a process of making inferences from learners’ performances on (explicit or implicit) tasks. Traditional quantitative methods of test theory use quantitative scores, derived from learners’ responses to test items, to underpin claims about their proficiency in a domain. Yet in many cases (for example GCSE or A level examinations in England) the “raw data” that forms the input for these methods is inherently qualitative: written responses to questions, which are appraised against fuzzy scoring criteria expressed in natural language. The recent development of large language models, such as BERT and GPT4, has opened new possibilities for processing such linguistically expressed information mathematically. In this talk Alex will explore the scope for using large language models to extend psychometric approaches to educational assessment.