Contributor: The crucial medical question that AI can’t ever answer

One of us got a call last spring from a longtime friend. The story was familiar: two doctors, an MRI, an online AI tool, a stack of articles — and one anxious question. “Everything tells me something different. The AI says I might need surgery. What should I do?”

We believe there’s one key response to anyone in this all-too-common conundrum: “What matters most to you?”

There was a long pause.

That pause is one of the most important moments in modern healthcare — and it is exactly the question artificial intelligence is unable to address.

In our careers as physicians and researchers, we have found, clearly and repeatedly, that for many common conditions the medical evidence does not point to a single “right” answer. The biology is often close. What determines the success of an outcome is whether the choice fits the person making it.

Some patients with back pain want the fastest possible return to physically demanding work, even if it means surgery. Others want to avoid an operation at almost any cost, even if recovery takes longer. The scan may look the same. The lives behind the scan are not.

That insight is becoming critically important as artificial intelligence moves deeper into everyday health decisions.

In our research on AI and clinical decision-making, we’ve studied what happens when systems are trained to optimize medical outcomes but are blind to human values. In plain English, today’s AI is very good at telling you what usually works for people like you with similar demographics and medical histories. It is far less capable of understanding what you are trying to protect, avoid or prioritize.

This matters because some of the most common and most expensive medical decisions are not purely biological. Should someone with low-risk prostate cancer choose surgery, radiation or careful monitoring? Should a person with atrial fibrillation undergo a procedure or manage the condition with medication? Should a patient with chronic knee or back pain operate now or try months of physical therapy to see whether surgery can be avoided?

In these situations, the medical differences between options are often small or uncertain. What makes the biggest difference is whether the treatment aligns with the patient’s goals: tolerance for risk, willingness to undergo recovery, ability to adhere to long-term therapy or simply what kind of life they want to live.
AI systems can calculate probabilities. They cannot determine what those probabilities mean to a particular person.

In some respects, artificial intelligence may know more medicine than any individual physician. It can synthesize millions of scientific papers, clinical studies and patient records in seconds. Yet it knows remarkably little about the person sitting across from it. AI does not know a patient’s goals, fears, obligations, tolerance for risk or personal definition of a good outcome. And because it knows little about either the patient or the physician, it knows even less about the conversation between them — the place where facts, values and trust come together to produce the right decision for a particular person.

A second patient story brought this home. A retired teacher was referred after an AI-based symptom checker flagged a heart rhythm abnormality and “favored” an invasive procedure. The patient arrived frightened, convinced there was one correct path. When we talked, it became clear that what mattered most was avoiding a long recovery and staying healthy enough to travel to see grandchildren.

Medication and monitoring — less dramatic, but well-supported by evidence — fit those goals better. The AI wasn’t wrong. It just didn’t know what mattered.
This blind spot is not trivial. Roughly a quarter of U.S. healthcare spending flows through decisions in which patient preferences meaningfully affect outcomes. When those preferences are ignored — by people or by algorithms — care becomes misaligned. That can mean unnecessary procedures, poor adherence, regret and rising costs without better health.

So what should consumers do when an app, portal or “smart” tool recommends a course of action?

Start with three questions.

First: “Best for whom?” If a tool says one option is best, ask whether it means best on average — or best for someone with your priorities.

Second: “What does this system not know about me?”
AI can see lab values and imaging results. It cannot see your job, your family responsibilities, your fears or what you are trying to get back to.

Third: “What happens if I wait or choose differently?”
Many important medical decisions are not emergencies. When options are close, taking time to reflect is often part of good care.

Artificial intelligence is becoming a powerful partner in medicine. It can help explain options, surface evidence and reduce confusion. But it should inform human decisions, not replace them.

AI may know more medicine than any physician.

It knows far less about any patient.

And it knows least about the conversation between them.

The most important variable in your healthcare is not in any algorithm. It is you.

James N. Weinstein is a surgeon and former chief executive of Dartmouth Health. He is a clinical professor at Northwestern University’s Kellogg School of Management and global head of Health Futures at Microsoft, which develops AI systems. Ogan Gurel is a physician and assistant professor at the University of Texas at Arlington, where he researches AI, causal inference and patient decision-making.