Exploring ChatGPT’s potential to answer genetic oncology queries
Researchers apply OpenAI’s ChatGPT to expert-curated questions in gynecologic oncology to reveal impressive performance for genetic counselling. Their research provides a peek into how such technology could be used to improve counselling services in the future.
A recent study, conducted by a team at NYU Langone Health (NY,USA) and its Perlmutter Cancer Center and presented at the Society of Gynecologic Oncology’s Annual Meeting on Women’s Cancer (March 16–18 2024; CA, USA), has underscored the potential of generative AI in enhancing patient care by accurately answering common genetic counseling questions.
Specifically, the NYU research team, led by Jharna M Patel, focused on evaluating the efficacy of ChatGPT in responding to 40 questions curated by gynecologic oncologists. These included 20 questions pertaining to genetic counselling, 17 pertaining to genetic testing for BRCA genes, and three pertaining to Lynch syndrome. The responses were then assessed by attending gynecologic oncologists based on accuracy and comprehensiveness.
ChatGPT delivered correct and comprehensive answers to 33/40 (82.5%) of the questions, followed by correct but not comprehensive responses to 6/40 (15%) of the questions, and just 1/40 (2.5%) partially incorrect. No responses were deemed incorrect. The chatbot answered all 20 questions in the genetic counselling category correctly, however performed poorest in questions related to Lynch syndrome with a score of 66% (2/3). For BRCA questions, it was correct for 88.2% (15/17) of questions.
Why is this important?
For patients receiving diagnoses for genetic conditions, or learning of their risk, the burden can be catastrophic. Genetic counselling is a critical service that helps individuals understand and adapt to the medical, psychological, and familial implications of genetic contributions to disease. However, the availability of counselling and the need for more genetic counsellors is a global healthcare challenge; one that is only set to become more apparent in the advent of personalized medicine and wider use of genetic testing.
Looking to the future
AI has the potential to revolutionize counselling care in a number of key ways. For example, it can be tailored precisely to a patient, capable of understanding the nuances of conditions some counsellors may be unfamiliar with. In essence, AI would not be limited by expertise for specific conditions. Additionally, 24/7 availability means patients could access counselling services on demand, being there for them when and wherever they need it.
The present research highlights how OpenAI’s freely available GPT-3.5 model can be used to answer basic genetic counselling questions. This is important because even without counselling-specific training, the model is capable of handling related queries.
With GPT-4 and other newer large language models (Claude 3 and Gemini), performance on these tasks is likely to be significant higher. It will certainly be interesting to see how they could perform on more complex questions patients are likely to ask.
