Renewal, Not New Prescribing

Utah started testing AI for prescription refills without physician signoff in January, as PYMNTS reported at the time. The state partnered with startup Doctronic to cover common chronic medications like statins and blood pressure drugs, spanning nearly 200 medications across primary care, according to Fierce Healthcare.

Legion’s scope is narrower, aimed squarely at mental health access. Most Utah counties are designated mental health provider shortage areas, leaving up to 500,000 residents without adequate behavioral care, according to the Utah Office of AI Policy.

The AI’s guardrails are tight. It cannot issue new prescriptions, adjust doses or handle controlled substances, benzodiazepines or antipsychotics. Patients must be stable and on an existing treatment plan with a licensed psychiatrist and must not have had a psychiatric hospitalization in the past year. Any signs of suicidality, mania, severe side effects or pregnancy trigger an immediate handoff to a human clinician, as detailed by the Utah Office of AI Policy.

The oversight structure is phased. The first 250 renewals by the AI require physician review before reaching the pharmacy, with a minimum agreement rate of more 98% required to proceed.

The next 1,000 renewals are reviewed after the fact, requiring a greater-than-99% threshold before shifting to randomized monthly tests, the Utah Office of AI Policy stated. Legion is required to file monthly reports on accuracy, physician alignment and any adverse outcomes under the policy.

The structure reflects Doctronic’s earlier mishaps. Within weeks of its launch, security researchers were able to push the system to triple a patient’s opioid dosage and generate misinformation about vaccines, as reported by The Verge.

The Access Case and Its Limits

State officials said the program would allow patients to get care “much more quickly and affordably,” freeing providers to focus on more complex cases, according to The Verge. Legion Co-founder and CEO Yash Patel described the pilot as “the beginning of something much bigger than refills.”

The demand for AI in healthcare is already there. More than 40 million people worldwide use ChatGPT daily for health-related queries, with about 70% happening outside clinic hours, as covered by PYMNTS.

Stanford GSB research found that a customized AI system cut prescription near-misses by about 33% in a pharmacy setting, but only with tight domain constraints and human review at dispensing. Without those conditions, broader AI models produced error rates between 50% and 400% higher than existing systems.

Critics aren’t convinced the access argument holds. Brent Kious, a psychiatrist and professor at the University of Utah School of Medicine, told The Verge the benefits of an AI refill system “may be overstated” and won’t reach the patients who need care most, since users must already be in treatment. He also warned of an “epidemic of over-treatment,” with patients staying on medications longer than necessary.

Utah’s 12-month pilot is designed to collect safety data to determine whether the model can expand to other states or tighten the limits regulators allow. Findings are due before the end of the year.