A modern NYU Langone Health study shows that generative AI compares favorably with physicians’ responses when answering patient questions in electronic health record emails — and could lend a hand reduce physicians’ documentation burden.
A study by researchers at Modern York University’s Grossman School of Medicine found that the genAI messaging tool not only produced exact responses to patient queries in the EHR, but also demonstrated greater perceived empathy.
“Our results suggest that chatbots can reduce the workload of care providers by enabling them to respond effectively and empathetically to patient concerns,” Dr. William Miniature, an assistant clinical professor in the NYU Grossman Department of Medicine and the study’s lead author, said in an announcement Tuesday.
WHY IS THIS IMPORTANT
During the pandemic, physicians have been receiving an average of more than 150 In Basket messages per day, up more than 30% year over year, according to Dr. Paul Testa, chief medical information officer at NYU Langone.
NYU Langone looked at ways to deal with note-taking and other documentation burdens, which are leading causes of physician burnout. For the modern study, medical researchers asked 16 primary care physicians to rate 344 randomly assigned pairs of AI-human responses to patient messages for accuracy, relevance, completeness, and tone.
According to researchers in the report Huge Language Model–Based Responses to Patients’ In-Basket Messages, published by JAMA, the AI’s generative responses outperformed human responses in terms of intelligibility and tone by 9.5%.
For the blinded testParticipating physicians indicated whether they would utilize the AI response as a first draft or have to start over. They did not know which responses were composed by humans or the AI tool.
As a result, the researchers say, AI responses were more than 125% more likely to be perceived as empathetic, and 62% more likely to utilize language that conveyed positivity and belonging.
However, they were 38% longer and 31% more likely to contain convoluted language.
“While humans answered patients’ questions at the level of sixth-grade students, the AI wrote at the level of eighth-grade students, according to a standard measure of readability called the Flesch-Kincaid scale,” the researchers said.
They also noted that further research is needed to confirm whether private data specifically improves the performance of AI tools.
BIGGER TREND
In 2023, NYU Langone licensed GPT4 to allow physicians to experiment using real patient data in a protected environment.
Also last year, the health research organization invited teams of doctors, educators, and researchers to work together to test large-scale language models as part of its Generative AI Prompt-A-Thon in Healthcare, for which no experience is required.
Attended by 70 people and more than 500 others via live webinar, the health system aimed to utilize the insights gained to inform the inherent potential of generative AI.
The researchers later used AI to analyze the transcripts of 820 people who received psychotherapy during the first wave of COVID-19 in the United States and compared them with the results of health care workers.
This AI-powered tool can detect stress in overworked hospital workers – those who discussed lack of sleep or mood issues during therapy sessions were more likely to be diagnosed with anxiety and depression.
IN THE DOCUMENT
“We found that EHR-integrated AI chatbots that leverage patient-specific data can produce messages that are similar in quality to those produced by human providers,” Miniature said in a statement.
“With this physician consent, the quality of GenAI messages will be equal to human-generated responses in quality, communication style, and usability in the near future,” said corresponding author Devin Mann, Ph.D., senior director of information innovation at NYU Langone Medical Center Information Technology.