EdTech & Innovation
LLM Scores Medical Student OSCE Notes Lower Than Faculty Graders
Nash et alAcademic Medicine

Objective Structured Clinical Examinations (OSCEs) require substantial faculty time to assess post-encounter medical documentation. To evaluate the feasibility of automated grading without task-specific fine-tuning, researchers analyzed 433 medical student notes across 7 standardized OSCE cases at the University of Kentucky. Using an existing institutional rubric and prompt-based workflows via API, OpenAI's GPT-4o mini evaluated each note under two conditions: numerical scoring without rationale (GPT-NE, 30 iterations per note) and numerical scoring with rationale (GPT-E, 4 iterations per note), compared against historical faculty grades.
The model demonstrated high internal consistency under the numerical-only approach (intraclass correlation coefficient [ICC] = 0.94) and moderate-to-high consistency when generating rationales (ICC = 0.76). However, absolute scores from the LLM were significantly lower than faculty grades on a 1–4 scale, with mean differences of -1.12 (95% CI, -1.17 to -1.08) for GPT-NE and -0.65 (95% CI, -0.69 to -0.61) for GPT-E (both P < .001). Spearman rank-order correlations ranged from r = 0.43 to 0.53, indicating moderate preservation of relative student ranking.
The authors concluded that LLMs demonstrate significant potential to automate OSCE documentation grading and reduce administrative burden, but notable discrepancies in absolute scoring underscore the need for improved rubric alignment and refined grading methodologies prior to routine implementation. For CE assessment design, the findings suggest that LLM-assisted scoring of documentation needs rubric calibration against human raters before it is relied on for grading.
Key Takeaways
- The large language model demonstrated high internal consistency for numerical-only scoring (intraclass correlation [ICC] = 0.94) and moderate-to-high consistency for scoring with rationale (ICC = 0.76).
- Across a 1–4 scale, model scores were significantly lower than faculty scores (mean difference: -1.12 for numerical-only, 95% CI -1.17 to -1.08; -0.65 with rationale, 95% CI -0.69 to -0.61; both P < .001), while showing moderate rank-order preservation (Spearman r = 0.43–0.53).
Read the original sourceAcademic Medicine
