Physicians at Mayo Clinic are beginning to explore how they could exploit enormous healthcare-specific language models – accessible through a generative artificial intelligence chat application – to improve patient care and streamline clinical decisions.
While ChatGPT and Google Gemini can only generate meaningful, evidence-based healthcare answers in a fraction of the cases, California-based Atropos Health says its federated healthcare data network can provide healthcare users with detailed and precise consultations for even the most unclear medical questions because it is based only on verified, real-world data.
For example, when clinicians are considering how to treat a patient with an unusual genetic condition that predisposes them to a particular heart disease, using data from millions of patients to identify similar patients and understand their outcomes could aid guide treatment decisions, Dr. Peter Noseworthy, chief of cardiac electrophysiology at Mayo Clinic.
Last year, Atropos launched an AI-enhanced generative platform that allows users to explore clinical data resources. It claimed that this was what happened the largest healthcare data network in the US in June. The company recently released its chat-based interface, ChatRWD.
“It’s a way to interact with real-world data in real time and then share those insights at the point of care,” Noseworthy, who is leading case-control trials and research using national datasets, began testing ChatRWD on Friday.
Assessing the reliability of healthcare-specific data
The Atropos platform provides Real World Data Score – a snapshot of data quality at a given point in time as measured by size, completeness, patient scheduling and more – and Real World Fitness Score – based on a proprietary algorithm that takes into account how well question criteria are represented within the data set – for each data set.
These ratings can aid users select the dataset that is most appropriate to answer each of their questions on the platform, the company said in June.
Saurabh Gombar, assistant professor at Stanford Healthcare and chief medical officer at Atropos, said he led a study that analyzed the accuracy and effectiveness of five healthcare-specific LLMs, including OpenEvidence and ChatRWD, to test the accuracy and effectiveness of the model’s results.
In terms of reliability, a one-size-fits-all LLM largely fails to answer doctors’ questions, he said.
“Whereas OpenEvidence and ChatRWD were able to provide useful, reliable evidence 42% or 60% of the time – much higher than the general-purpose LLM,” Gombar said in July.
Since 2022, Atropos has been collaborating with Mayo Clinic pilot and develop data-driven methods that can improve health care – both techniques and improved care delivery for historically underrepresented patients – by making real-world evidence available through automated reports called forecasters.
The collaboration enabled physicians and researchers using the Atropos digital consulting platform to access Mayo Clinic’s deidentified data repository and its analytics tools.
“It’s a way to interact with real-world data in real time and then share those insights at the point of care.”
Dr. Peter Noseworthy, Mayo Clinic
For patients in intensive care units, the ability for their care teams to find answers to research questions through the platform can save time. While classic methods can take weeks to determine treatment, Atropos says an AI-based prognosticator can be completed in a matter of days.
Noseworthy noted that observational clinical researchers have access to substantial real-world data, “but the time to generate insights is months.” They not only have to retrieve data, but also pristine and analyze it.
“You have to have statistics you can work with,” he said.
“With a tool like this, which can essentially set up research in real time and pull that data, you can get research-level or publication-level information through a chat interface.”
Atropos said it forecasts a more than 200% escalate in the availability of additional datasets over the next year.
The power of patient data comes with artificial intelligence
Where experienced clinicians can recognize patterns in patient outcomes and treatment responses based on experience, capturing the entire experience with a drug or treatment – the “representatives of patient outcomes” – has been circumscribed by classic medical research methods, Noseworthy explained.
“We could achieve this through clinical trials, but it is a slow process and patients are highly selected.”
However, LLMs can provide doctors with faster answers to medical questions, which could aid improve treatments for patients who have historically been beyond the reach of clinical trials.
“Rare or unusual manifestations of a disease or rare conditions, or rare combinations of conditions, are not well characterized in clinical trials but do occur in a large sample of data,” Noseworthy said.
Mayo Clinic is working to expand the boundaries of clinical research beyond the walls of major academic medical centers, including: decentralized clinical research program last year.
Access to clinical trials has exacerbated health disparities, according to Dr. Tufia Haddad, a medical oncologist, faculty development officer in Mayo Clinic’s Department of Oncology, and co-head of the Office of Digital Platforms and Innovation at the Comprehensive Cancer Center.
“We underrepresent racial-ethnic minority and patient populations in our research, as well as underrepresentation of underserved rural populations,” she said.
The overarching goal of improving access to clinical trials is “to provide more medicines to more people,” she said after the program’s launch.
While some practices at Mayo Clinic are piloting ChatRWD, Noseworthy said he was interested in the solution because colleagues in his cardiology group have experience using real-world data from other data engines.
“What was attractive to me was that we could essentially generate data in real time and at the point of care,” he said.
Using real clinical data – “this is completely different to using ChatGBT” or other LLMs.
While ChatRWD hasn’t yet been implemented on a enormous scale at Mayo Clinic, “it was able to give us some interesting information,” Noseworthy said.