Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study

Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study. Cornelisse E(1), Kousa D(1), Menz BD(1), Murphy-Reuter B(2), Shahnam A(3), Chin-Yee B(4), Calabro R(5), Li LX(1), Ramsey I(6), Abuhelwa AY(7), Logan JM(8), Ng HS(1), Aravinthan M(9), Goonan J(9), Carthew H(9), Molyneux P(10), Tang J(11), Hitchen N(12)(13), Hong M(14), Gandhi J(8)(15), Fontanarosa S(15), Scarfo NL(15), Daniel K(15), Marotti SB(1)(8)(15), Mariadas J(1), van Diermen B(10), Susai Manickam DJ(9), Ruanglertboon W(16)(17), Postma D(1), McCradden MD(18)(19)(20), Reddy S(21), Bacchi S(1)(22), Mangoni AA(1)(23), McKinnon RA(1), Kichenadasse G(1)(24), Lyman GH(25)(26), Kuderer NM(27), Sorich MJ(1), Rowland A(1), Hopkins AM(1). BACKGROUND: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation. OBJECTIVE: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts. METHODS: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and refined through iterative feedback from a multidisciplinary expert panel (n=23). The pilot rubric was applied to 4 health care scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to health professional continuing education questions, and automated physician-to-physician clinical documentation generation. Via these scenarios, 15 clinicians (medical practitioners, pharmacists, and medical oncologists) evaluated 250 input-output pairs using domain-level ratings and qualitative comments. The evaluated outputs included 200 AI-generated responses across the 4 health care scenarios and 50 physician-tagged replies i…