Does AI Pass the Smell Test? Evaluating Artificial Intelligence Applications in Rhinology Patient Education

Abstract

Introduction: Rhinologic procedures and the management of rhinological conditions require a deep understanding of intricate anatomy and complex decision-making. To address the need for a patient learning tool and a resource to answer common rhinology-related questions, we explored the utility of ChatGPT, a state-of-the-art language model to assist in patient education on common rhinologic questions and procedures. We conducted a comprehensive evaluation of ChatGPT's responses, comparing them with a panel of expert reviewers and explored its potential applications in the field of rhinology.

Methods: ChatGPT was utilized to generate responses to common questions related to rhinology and rhinologic procedures. Topics for evaluation included reasons for a stuffy nose, explaining chronic rhinosinusitis, applications and indications for sinus surgery, and post-surgical expectations and associated risks of sinus surgery. These responses were meticulously recorded and subsequently incorporated into a survey for evaluation. The survey was distributed to a panel of experienced rhinologists, who were tasked with evaluating the quality of responses. Evaluation criteria included an overall quality score of the responses, identifying errors or missing information if applicable, and recognizing any aberrant medical advice. Rhinologists were asked to assess the correspondence of ChatGPT's responses with their current medical standard.

Results: We performed a systematic literature review to identify current use cases of artificial intelligence in rhinology. This review highlighted the evolving role of AI in diagnosing, treatment planning, radiographic interpretation, and post-operative monitoring in rhinological cases. AI-based tools have demonstrated promise in automating repetitive tasks, enhancing diagnostic accuracy, and aiding in surgical planning. Very few centered on the patients' experience or education so we sought to address this gap by evaluating publicly available tools to answer commonly asked questions from patients.

To do this, a panel of experienced rhinologists evaluated ChatGPT's responses to provided prompts. The evaluation revealed that ChatGPT's responses varied in quality, with a mix of accurate information, minor errors, and occasionally suboptimal advice. While ChatGPT's responses demonstrated potential as an educational tool, they often lacked the depth of understanding and context that experienced rhinologists possess. Despite some pitfalls in the responses, rhinologists scored the responses on average 8.17 out of a maximum of 10. The responses were found to be valuable in addressing many common questions but less reliable in complex decision-making scenarios.

Conclusion: ChatGPT offers a valuable resource for educating patients and providing quick answers to common rhinological inquiries. However, its limitations become evident when addressing intricate, multifaceted issues or complex medical decision-making. Further developments in AI, specifically tailored to rhinology, hold potential for improving patient education, diagnostics, and surgical planning in this specialized medical field. The integration of AI tools in rhinology has the potential to enhance patient care, but careful consideration of their limitations and the need for expert oversight is essential in their implementation.