Currently submitted to: JMIR Formative Research
Date Submitted: Mar 10, 2025
Open Peer Review Period: Mar 24, 2025 - May 19, 2025
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Assessing the Quality of Artificial Intelligence Explanations on Atrial Fibrillation: A Comparative Analysis of ChatGPT and Google Gemini
ABSTRACT
Background:
Atrial fibrillation (AF), a common arrhythmia, is a major stroke risk factor, making patient education critical. Artificial intelligence (AI) platforms like Google Gemini and ChatGPT are emerging tools for medical education.
Objective:
This study aimed to (1) assess the quality of ChatGPT and Google Gemini’s explanations of AF and its treatment, (2) compare responses from both platforms and (3) analyze differences in interpretation between cardiologists and non-medical professionals.
Methods:
On September 6, 2024, the prompt “Explain atrial fibrillation and how to treat it to a patient” was entered into ChatGPT and Google Gemini. A survey based on PEMAT-P and DISCERN criteria was completed by 11 cardiologists and 17 non-medical professionals. Averages and standard deviations were calculated and compared using the Wilcoxon signed-rank test.
Results:
No significant quality difference was observed between ChatGPT and Google Gemini. Cardiologists rated bias lower (3.82 vs. 4.33, p=0.04) and explanations of consequences of no treatment higher (2.85 vs. 1.86, p=0.005) compared to non-medical professionals. Visual cues, informative headers, concise sections, actionable advice, and direct addressing of the reader received significantly higher ratings from cardiologists.
Conclusions:
The comparable quality of ChatGPT and Google Gemini suggests that both are viable for AF education. Cardiologists’ higher ratings for critical aspects of explanation highlight a gap in patient understanding, underscoring the need for clearer AI-driven educational tools. Clinical Trial: n/a
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.