EdTech & Innovation
Interview Scores Are Higher With Human Standardized Patients Than AI-SPs
Yanagita et alJMIR medical education

In a randomized single-center crossover trial conducted at Chiba University Hospital, researchers evaluated medical interview performance and learner-reported experiences comparing a text-based artificial intelligence standardized patient (AI-SP) powered by GPT-4o against traditional human standardized patients (SPs). The study evaluated 48 fifth-year medical students completing clinical clerkships who were randomized to interact with an AI-SP followed by a human SP, or the reverse sequence, separated by a washout period of at least 3 days. Each student completed two clinical scenarios—depression and appendicitis—under both conditions. For AI-SP interactions, chat logs were stored and evaluated with physician feedback provided after completion of both cases, whereas human SP encounters were video recorded with immediate feedback delivered by the physician acting as the SP.
Interview performance was independently scored by two general medicine specialists using selected items from the Mini-Clinical Evaluation Exercise (Mini-CEX), with an interrater reliability intraclass correlation coefficient (2,1) of 0.76 (95% CI 0.66–0.83). Overall Mini-CEX mean total scores were significantly higher for encounters with human SPs compared with AI-SPs (23.9, SD 1.7 vs 22.7, SD 1.7; mean difference 1.24; t47 = 3.7; P = .001), with higher ratings specifically recorded for history taking and overall clinical competence.
On post-encounter 7-point Likert questionnaires, participants reported significantly higher active engagement in history taking during the AI-SP condition compared to the human SP condition (mean 6.2, SD 0.7 vs 5.8, SD 1.1; t47 = 2.7; P = .009). No significant differences between modalities were observed for learning satisfaction (AI-SP: mean 6.0, SD 0.8; human SP: mean 5.9, SD 1.1) or perceived rapport formation (AI-SP: mean 5.3, SD 1.3; human SP: mean 5.4, SD 1.3). The study authors noted that the text-based interface and lack of nonverbal cues may have contributed to the lower clinical performance scores in the AI group. Additionally, the trial evaluated performance during simulated sessions rather than measuring subsequent downstream learning retention or clinical performance with actual patients.
Key Takeaways
- Mini-CEX mean total scores were significantly higher during encounters with human standardized patients compared to the AI-based standardized patients (23.9, SD 1.7 vs 22.7, SD 1.7; mean difference 1.24; P = .001), with higher subscores observed for history taking and overall clinical competence.
- Learners reported significantly higher active engagement in history taking with the AI standardized patient compared to the human standardized patient (mean 6.2, SD 0.7 vs 5.8, SD 1.1; P = .009), while ratings for learning satisfaction and perceived rapport formation showed no significant differences between groups.
Read the original sourceJMIR medical education
