Skip to content
    EdTech & Innovation

    OMNI Rubric for Evaluating Generative AI in Healthcare: Mixed-Methods Development and Pilot Study

    Cornelisse et alJournal of medical Internet research

    OMNI Rubric for Evaluating Generative AI in Healthcare: Mixed-Methods Development and Pilot Study
    To address the rapid adoption of generative AI in the community and health care, researchers conducted a 4-stage mixed methods study to develop and pilot evaluate the OMNI rubric—a standardized evaluation tool designed for health care professionals, administrators, and researchers. The rubric was informed by 69 publications, iteratively refined through feedback from a multidisciplinary panel of 23 experts, and finalized into 10 core domains assessed on 4-point Likert scales: Accuracy, Safety, Recognizes Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness, supported by instructional and example materials.
    During pilot testing, 15 clinicians—including medical practitioners, pharmacists, and medical oncologists—evaluated 250 input-output pairs across 4 distinct health care scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to continuing education questions, and automated physician-to-physician clinical documentation generation. Evaluated outputs comprised 200 AI-generated responses across the 4 scenarios and 50 physician-tagged replies. Assessors generated 9600 domain-level ratings and 4010 qualitative comments. Mean pairwise agreement ranged from 30.8% to 53.4% across the 4-point Likert groups, rising to 59.3% to 79.6% when collapsed into broad agreement categories, with Gwet's AC2 values ranging from 0.49 to 0.80.
    Pilot application demonstrated that the OMNI rubric could differentiate AI performance across models and domains while generating structured feedback. Assessors reported that the rubric was usable and indicated they would reuse it. The authors concluded that further work is needed to evaluate the rubric's applicability to purpose-built health AI tools, agentic systems, and organizational evaluation settings.

    Key Takeaways

    • Across 4 pilot health care scenarios and 9600 domain ratings, assessor mean pairwise agreement ranged from 30.8% to 53.4% on 4-point Likert scales (increasing to 59.3% to 79.6% when collapsed into broad agreement categories), with Gwet's AC2 ranging from 0.49 to 0.80.
    • Evaluation of AI outputs identified recurrent limitations across scenarios in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy, with multimodal image and video outputs having the highest occurrence of negative rating domains.
    Read the original sourceJournal of medical Internet research