These days there are a bunch of online apps that attempt to help patients figure out what their health symptoms might mean. These symptom-checkers ask patients to enter in information about their symptoms and background and return a set of likely diagnoses. They use algorithms to come up with the most likely explanation for what is ailing the patient, but do they work?
The history of computer-aided diagnosis is much longer than you might expect. In the 1970’s researchers at the University of Pittsburgh created software to diagnose complicated problems in internal medicine. It eventually evolved into a commercial program called Quick Medical Reference. Massachusetts General Hospital has been working on and refining DXplain since the 1980’s. It provides a ranked list of clinical diagnoses from information about patient symptoms and laboratory data.
A new study, published on October 10 in JAMA Internal Medicine suggests that they really don’t. It turns out (shocker) that doctors are significantly better than symptom-checker software at getting the diagnosis right. The results do imply, however, that doctors are not perfect and they might benefit from using computer-based tools to complement their skills and experience.
Symptom Checkers Fail 66% of the Time
The study was designed to compare doctors against symptom-checkers head-to-head. When given the same information about the patient’s current symptoms and medical history, doctors got the right result 72% of the time. The symptom-checkers did not fare so well, coming back with the right diagnosis in only 34% of the cases.
A total of 23 symptom checkers were included. Some are online websites and others are mobile apps. The group included software offer by WebMD and the Mayo Clinic in the United States and the Isabel Symptom Checker in the UK.
The senior author of the study, Dr. Ateev Mehrotra of Harvard Medical School was not shocked. “The current symptom checkers, I was not surprised do not outperform doctors,” he said. In an interview with Reuters Health he explained that human doctors should be teamed up with software tools to augment their thinking, rather than seeing it as an either-or situation. “Physicians do get it wrong 10 to 15 percent of the time, so maybe if computers were augmenting them the outcome would be better,” Dr. Mehrotra said.
The researchers created 45 clinical scenarios. Fifteen of them described acute conditions, 15 were moderately serious, and 15 required little or no intervention. Most of them described common ailments, but 19 depicted less common conditions. The researchers sent them to 234 physicians. The doctors could not do a physical examination (which would be hard on a hypothetical patient) or run any tests. They could not ask questions. They had to make a call based only on the information provided. The physicians replied with free text answers with possible diagnoses ranked in the order they found most likely.
The same information was then entered into the symptom checkers. The doctors place the correct diagnosis in the top spot more often in every case. The software algorithms did better at more common and less serious conditions than they did for the more severe and rare. Doctors significantly outperformed the apps in these outlier cases.
This study backs up a similar one published in BMJ in 2015. “Although there was a range of performance across symptom checkers, overall they had deficits in both diagnosis and triage accuracy. If symptom checkers are seen as a replacement for seeing a physician, they are likely an inferior alternative,” that study read.
“In medical school, we are taught to consider broad differential diagnoses that include rare conditions, and to consider life-threatening diagnoses,” said Dr. Andrew M. Fine of Boston Children’s Hospital, who was not part of either study. “National board exams also assess our abilities to recognize rare and ‘can’t miss’ diagnoses, so perhaps the clinicians have been conditioned to look for these diagnoses,” he said.
Symptom-checkers are attractive to patients because they can quickly get an idea of what their symptoms might mean. Of course trusting this information could lead to unnecessary worry, or worse, lack of attention to a serious problem. But it isn’t always easy or convenient to get a doctor’s appointment, especially if you are unsure that the problem actually requires medical attention.
Telemedicine offers one alternative for people who need their symptoms diagnosed quickly and conveniently. Physicians practices and urgent care centers are adding the option of video visits for their patients. Adding the video element increases the physician’s chances of making a correct diagnosis and improves patient confidence in the treatment plan.
Just like with the symptom-checker, patients can get a diagnosis right from the comfort of their home or office. They’re just a lot more likely to get the right one.
Read the Fine Print
Before taking the results of a symptom-checker too seriously, it’s a smart idea to read the inevitable disclaimer. Take this one from WebMD to heart,
“This tool does not provide medical advice It is intended for informational purposes only. It is not a substitute for professional medical advice, diagnosis or treatment. Never ignore professional medical advice in seeking treatment because of something you have read on the WebMD Site. If you think you may have a medical emergency, immediately call your doctor or dial 911.”