Assessing large language models as essay raters: Scoring behavior and fairness in source-based writing
SYSTEM, cilt.143, 2026 (SSCI, Scopus)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 143
- Basım Tarihi: 2026
- Doi Numarası: 10.1016/j.system.2026.104184
- Dergi Adı: SYSTEM
- Derginin Tarandığı İndeksler: Social Sciences Citation Index (SSCI), Scopus, IBZ Online, Periodicals Index Online, Communication Abstracts, EBSCO Education Source, Educational research abstracts (ERA), MLA - Modern Language Association Database, MLA International Bibliography, EBSCO Communication Source, Academic Search Ultimate (EBSCO), Communication Source (EBSCO), Education Source Ultimate (EBSCO)
- Manisa Celal Bayar Üniversitesi Adresli: Evet
Özet
Large language models (LLMs) are increasingly used as essay raters but evaluated mainly by agreement with human scores. Using 2100 source-based argumentative essays from ASAP 2.0, this study calibrated two LLM raters, Gemini 2.5 Flash and GPT-4.1 mini, and an adjudicated human benchmark on a common many-facet Rasch scale. The raters were compared on severity, model-data fit, consistency, scale use, and differential severity by gender, English language learner (ELL) status, disability status, economic disadvantage, and race/ethnicity. Robustness checks used two rescoring occasions, a few-shot prompt, and rater-specific thresholds. Both models were more severe than the benchmark, showed slight overfit and higher discrimination, concentrated scores at 2 and 3, and agreed with it to a similar, moderate degree. No statistically significant gender-related differential severity was detected. In the pooled analysis, Gemini scored ELL writers more severely and GPT more leniently than expected; within prompts, these contrasts kept their direction but did not reach the corrected significance threshold. Gemini's pooled effect persisted across occasions and prompting conditions, whereas GPT's did not persist under the fewshot prompt. Gemini's greater severity held in every calibration, whereas GPT's severity relative to the benchmark changed with the prompt, and each model agreed less closely with its own scores across four months than across two weeks. Models with similar agreement can thus differ as raters.