Skip to content
    EdTech & Innovation

    Evaluating GPT-4 and Human Expert OSCE Script Development in Medical Education

    Josan et alMedical science educator

    Evaluating GPT-4 and Human Expert OSCE Script Development in Medical Education
    Developing high-quality standardized patient (SP) scripts for Objective Structured Clinical Examinations (OSCEs) requires substantial time and resources. While large language models like GPT-4 offer potential efficiencies in medical education content creation, their educational adequacy and realism relative to expert-crafted materials remain uncertain.
    In this pilot comparability study, researchers assessed 15 OSCE SP scripts created across three approaches: expert-written, AI-generated, and AI-generated with human editing, with five scripts assigned to each arm. Blinded standardized patients and physicians evaluated the scripts across five key educational domains: detectability of AI authorship, perceived realism, consistency of role depiction, clinical accuracy, and alignment with learning objectives. An exploratory non-inferiority analysis was also performed using a 10-percentage-point margin.
    Raters could not reliably distinguish scripts from either AI cohort from human expert-authored scripts. For perceived realism, mean ratings were 3.90 (SD 0.79) for expert-written, 3.88 (SD 1.02) for AI-generated, and 3.56 (SD 1.07) for AI-plus-human edited scripts. Clinical accuracy was rated at 4.27 (SD 0.70) for expert-written, 4.07 (SD 0.96) for AI-generated, and 4.40 (SD 0.63) for AI-plus-human scripts. Alignment with learning objectives averaged 4.27 (SD 0.80), 3.67 (SD 1.11), and 4.27 (SD 0.80), respectively. Scripts developed with AI combined with human editing earned the highest scores in clinical accuracy and objective alignment, while expert-written scripts scored highest in realism and role consistency.
    Non-inferiority to expert-written scripts was not demonstrated for either AI set in the exploratory analysis. The authors concluded that the pilot findings provide preliminary evidence of comparability across several educational quality domains, though larger, more rigorous evaluations are necessary to conclusively assess non-inferiority against expert-generated materials.

    Key Takeaways

    • Blinded standardized patients and physicians could not reliably distinguish either fully AI-generated scripts or human-edited AI scripts from expert-authored OSCE scripts.
    • Scripts generated by AI with human editing scored highest for clinical accuracy (mean 4.40, SD 0.63) and learning objective alignment (mean 4.27, SD 0.80), whereas expert-written scripts scored highest for realism (mean 3.90, SD 0.79) and consistency of role depiction.
    • In an exploratory non-inferiority analysis within a 10-percentage-point margin, non-inferiority to expert-written scripts was not demonstrated for either AI-generated script group.
    Read the original sourceMedical science educator