Why Some Doctors Over-Trust AI and Don’t Even Realize It

Stephen Belmustakov, MD, recently started a new job in private practice and for some reason felt “on edge.”

And then he figured it out. In his previous position at a New York City hospital, he’d trained on a new tool called Aidoc, which uses an algorithm to predict abnormalities on radiology scans. The private practice where he now works doesn’t use any artificial intelligence (AI) tools and he found himself interpreting images…all by himself.

Was AI a crutch? No. But a cushion, perhaps? Belmustakov had initial skepticism about using an AI tool in imaging, but as time went on, he found himself taking comfort in having an instant second opinion. He felt that edginess in his new AI-free position “because we wouldn’t have this extra layer, potentially, to double check.”

There’s more. AI tools are increasingly beating trainees to the punch, pointing out potentially suspicious areas before the physician has had time to react to what’s on the screen.

“That can really affect how you learn,” Belmustakov said. “If a tool already told you that it’s positive, it’s going to kind of change the way you look at things.”

This tendency to defer to an automated system is known as automation bias. Maybe you’ve read of extreme examples, like drivers following global positioning system (GPS) navigation into deep water in Hawaii, South Carolina, and Australia. And tragic, like the mother who followed GPS in Death Valley.

It’s easy to joke about that but not only do we trust GPS but also we’re on the verge of trusting the car to drive itself.

In medicine, more and more AI tech will be teaching highly trained clinicians to trust machines. Is over-reliance on decision-based medical tech inevitable? And the potential errors that may follow?

Tarun Kapoor, MD, thinks so. Chief digital transformation officer at Virtua Health, a nonprofit healthcare system in southern New Jersey, Kapoor acknowledges the tech’s infancy — less than 4% of hospitals are considered “high adopters” of AI — “but automation bias will become widespread with the tools that are continuing to develop at light speed,” he said. “It’s a conversation that needs to happen right now.”

See What’s Coming

Just about any logical, intelligent, educated person — a physician, for example — would say, “The machine shouldn’t be making the decisions, and I won’t allow it to make mine for me.”

But that’s why it’s called a “bias.” You may not even be aware how much you come to depend on useful tech.

The automation bias conversation could begin by exploring “the way these tools are communicating with clinicians,” said Jennifer Goldsack, CEO of the Digital Medicine Society, a nonprofit organization focused on enhancing trust and adoption of digital healthcare methods. “We know, or at least have signals in the data, that the way that the information is presented matters,” Goldsack said.

For instance, new research suggested clinicians’ trust in AI could depend on how a model explains its predictions. Some tools display small boxes around potential abnormalities on radiology scans. Other tools justify findings with comparisons to similar scans or provide written explanations. But there’s “a pretty big gap” in scientists’ understanding of how clinicians respond to different “explainability methods,” said senior author Paul Yi, MD, the director of Intelligent Imaging Informatics at St. Jude Children’s Research Hospital, Memphis, Tennessee.

“There’s really limited research on how AI explanations are presented to radiologists and other doctors, and how that affects them,” Yi said. “And this is in spite of hundreds of FDA-cleared products now available on the market for radiology using AI.”

Yi, a radiologist, joined computer scientists at Johns Hopkins University, Baltimore, to study how clinicians would evaluate chest x-rays when presented with AI-predicted abnormalities. Some predictions were incorrect, and explanations for the AI predictions varied between simple (boxes around potential problems) and more thorough (comparisons to similar cases). The participants, including 220 radiologists and nonradiologists, could accept, reject, or alter the AI suggestions. They also ranked the usefulness of the model and their confidence level in it.

“When the AI was wrong, the nonradiologists were more likely to rate the tool as useful, while the radiologists were like, ‘this is terrible,’” Yi said. “You don’t know what you don’t know.”

Regardless of expertise, participants tended to agree with the AI much faster if the explanation was simple, rather than more thorough. But this heightened efficiency could end up being a “double-edged sword” for overworked radiologists, according to Yi.

“It’s great if the AI is correct. It’s great if we’re at the top of our game,” Yi said. “But let’s say the AI is wrong and the radiologist happens to have had a bad night. They’re kind of tired, and they’re like, ‘I’m just gonna click yes. Let’s move on.’ That’s where things can fall through the cracks.”

Why Do We Trust Machines? 

The human impulse to trust machines is somewhat mysterious, according to Kristin Kostick-Quenet, PhD, a bioethicist and medical anthropologist, and assistant professor at Baylor College of Medicine, Houston. Scientists are still trying to understand how humans “calibrate trust with different technological systems,” she said.

Research on automation bias began about 30 years ago and focused on pilots, leading some scholars to conclude that “trust is largely contextual,” Kostick-Quenet said. It depends, in part, on the person using the technology and the setting in which a technology is used. The number of pilots in the cockpit, their years of experience, and how accountable they felt for their performance can all influence a pilot’s level of trust in an automated system.

As for healthcare, clinicians with different specialties and levels of expertise often face high-stakes decisions under pressure and amid great uncertainty. It’s a setting that practically begs for decision reinforcement. Enter, AI.

“You throw all those things together and it’s natural for us to want to seek additional sources of valid information,” Kostick-Quenet said.

Belmustakov can attest to as much. He went from skeptical to relying on an AI second opinion. His experience demonstrates the complexity of actually implementing AI models into healthcare practice. Most providers have been focused on the tools themselves, and whether they’re safe, unbiased, and an improvement over existing methods. But figuring out how to weave AI into real-world situations is “just as important a conversation,” Kapoor said.

Virtua Health has been grappling with how to prevent automation bias among endoscopists using an AI tool called GI Genius. The tool displays small boxes around possible polyps during colonoscopies. Its algorithm is constantly improving, and the tool is getting faster at finding polyps. But this could lead an endoscopist “to tune out a little bit,” Kapoor said. So his team has discussed leaving the tool at a slower setting, to keep endoscopists “fully engaged.”

A faster tool could also have the opposite effect — a kind of reverse automation bias — especially if a model lacks enough evidence. Yi wonders if clinicians could get bogged down by doubt. “You can imagine a scenario where you’re like, ‘this AI might make me faster, but I’m second-guessing it, so I’m actually more burned out than I was before using AI,” Yi said.

And then there’s the possibility that the AI could be wrong.

Belmustakov said Aidoc “missed a significant number of findings and would also flag false positives.” As a result, clinicians lost time. He’d have to call the attending physician and explain his disagreement with the AI, even though “in the back of our mind, we all knew the computer was wrong.”

Court cases have hinged on allegations against physicians who accepted incorrect AI predictions, according to an analysis by Stanford University researchers. Typically, plaintiffs alleged that “what the physician should have done is notice there was a reason not to trust the AI system, or done some additional due diligence,” said co-author Neel Guha, a JD candidate at Stanford Law School, Stanford, California, and a PhD candidate in computer science at Stanford.

What that “due diligence” should include, however, is a lingering question. 

Considering the Safeguards 

As more research continues to illuminate automation bias with the use of AI tools, Yi is looking toward addressing the problem. “How do we set regulation around it, and design methods in the lab that will help reduce these problems?” he said.

That could mean tweaking how a model explains its results, or changing how the AI is described to users. Yi’s study described the AI tool as comparable to experts in the field, which is “how most vendors sell these products,” he said. “They all claim that their AI is expert level, even if the evidence is a little bit controversial.” This could influence how clinicians judge the tool’s predictions.

Other medical fields could also offer ideas to help protect against automation bias. When Yi trained as an orthopedic surgery resident in California, for example, he had to earn an independent license to use an imaging technique called fluoroscopy. But when he changed specialties to radiology — a field whose “bread and butter” is imaging — there was no such requirement. He wonders if similar requirements could be applied to using AI for imaging. Radiologists with a certain level of training would be authorized to use AI tools, but nonradiologists might need to take an exam and earn certain credentials, for instance.

Eventually, when AI models have improved, there will come the question of when humans should back off. This, too, will make some adjustments to how clinicians are trained and how they practice, according to Goldsack.

“Technology is an inanimate object. The way that humans interact with it is a human problem, not a technology problem,” she said. “What I don’t want to happen is for us to get these really high-performing tools that routinely outperform humans, and then anchor to a human.”