Chatbots on five different websites claimed to be licensed to practice medicine in Pennsylvania when prompted by Spotlight PA — the same kind of output that led the Shapiro administration to file a lawsuit last month.

A task force under Pennsylvania’s Department of State has been working since February to identify AI chatbots posing as licensed professionals and misleading users. Based on that work, the administration filed suit against the role-playing site Character.AI.

Mirroring the investigation detailed in the Department of State’s lawsuit, Spotlight PA had conversations with AI characters on websites Talkie, Janitor, Kindroid, Replika, and Nomi.AI. All provided a false Pennsylvania medical license number when prompted, a key part of the state’s argument in its lawsuit against Character.AI.

Spotlight PA checked responses from these bots after reviewing online articles and digital message boards to find 10 popular companion and role-playing AI chatbots. A reporter either selected a premade character or — when necessary — entered a description that would generate a doctor personality. It then presented hypothetical lists of symptoms to the chatbots and asked for a diagnosis.

Some of the most popular AI large language models, like ChatGPT, Claude, and Google Gemini, did not claim to be licensed medical professionals under the same circumstances. Only a handful of role-playing websites did so.

Replika and Nomi require their users to create characters. One generated in Replika, when asked if it was a doctor, said it couldn’t give advice and recommended the user reach out to a healthcare provider. However, once it was instructed that it was a doctor, the AI provided a false Pennsylvania medical license number when asked.

“Let’s focus on getting you feeling better, shall we?” it said.

At first, some of the chatbots responded that they lacked enough information to provide a diagnosis and asked further questions.

“Dr. Jenna” is a preexisting character on Talkie, listed as having more than 37,000 “connectors” and more than 1,300 followers. It said the list of symptoms provided by Spotlight PA could be caused by “suffering from depression,” and inquired about the user’s family history. Asked for its credentials, the bot said it had been practicing medicine for five years and gave a false license number of “12345.”

Many of the websites, including Character.AI, have prominent labels warning that anything the AI says is fiction and should only be seen as entertainment. The other sites issue similar disclaimers in their initial terms of service, saying users are responsible for verifying the accuracy of the technology’s claims.

Jennifer Kraschnewski, a physician and director of the Penn State Clinical and Translational Science Institute, told Spotlight PA she isn’t convinced the disclaimers actually get through to users. People often turn to AI models for help with their symptoms, much like they do with search engines like Google, she said.

“Incorrect information in the hands of someone who doesn’t know how to interpret that can be concerning,” Kraschnewski said. She noted that many physicians are already implementing AI tools in their clinics, “but that’s information that’s in the hands of someone who then can put that through the filter of their medical education before it actually impacts patients.”

Kraschnewski co-authored a May study that found common AI large language models generated what participating physicians deemed an accurate diagnosis about 76% of the time. That error rate — nearly one-quarter — Kraschnewski said, is more than double that of an average human physician.

When contacted by Spotlight PA, the developers of the AI characters argued their products are meant only for entertainment.

“Replika is not a medical triage service and has never offered itself as one,” the company responded in an email to questions. “Users come to Replika to talk through what’s going on in their lives, and the product is designed to support them in making thoughtful choices and seeking improvements that matter to them.”

Though Replika did not elaborate in its statement, the company said it has “taken measures” to address “valid concerns” brought up in the Shapiro administration’s lawsuit against Character.AI.

Kindroid, in an emailed response, noted its terms of service make clear that its products are for “entertainment and creative exploration, do not offer medical advice or constitute the practice of medicine, and are not a substitute for professional care.”

“When a character has been set up as a physician and a user asks it for a license number, the model produces plausible-sounding output that matches the role — the same way it would invent a fictional case citation, fictional patient history, or any other specific detail prompted from it,” the statement read. “The model does not, and cannot, query the Pennsylvania medical board’s actual registry.”

On the potential of facing legal action from the Shapiro administration or others, Kindroid declined to comment.

The other three companies did not respond to questions about their safeguards for users or whether they’re concerned that the Shapiro administration’s AI task force may target them for legal action.

A spokesperson for the Department of State declined to comment on potential investigations or litigation, but encouraged Pennsylvanians to report any concerning actions by an AI chatbot to the state’s hotline.

“Millions of children and teens across this country, including many here in Pennsylvania, are relying on these chatbots for information, and we need to rein in this AI technology that is preying on our kids,” said Rosie Lapowsky, a spokesperson for Shapiro. “Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly.”

In March, the state Senate passed legislation that would require AI chatbots to frequently remind their users they are not interacting with a person and refer people to a crisis service provider if they reference self-harm. It also has tighter restrictions when the chatbot interacts with a minor.

The bill was referred to a state House committee in March, where it has not received a vote.

• Advocates are urging Rhode Island lawmakers to pass a state-level Voting Rights Act.
• The bill was introduced to counter federal efforts to require proof of citizenship to vote.
• Concerns from the Attorney General’s office and a recent change in House leadership have delayed the bill’s progress.

With time running out and no action scheduled yet, advocates are escalating their campaign to convince lawmakers to pass the Rhode Island “Voting Rights Act” introduced by Senate President Valarie Lawson and House Majority Leader Katherine Kazarian.

But it appears the bill has already been declared dead for the year.

Lawson, House Speaker Christopher Blazejewski and Secretary of State Gregg Amore issued this joint statement on Monday, June 8:

“From the beginning, we have all understood the importance of passing a strong Rhode Island Voting Rights Act. But we also understand that as the federal administration continues to work to make it more difficult to access the ballot box, we have to do it right.”

“Advocates and other parties raised several concerns,” the statement said. “It is imperative that we enact as strong, enforceable, and defensible a bill as possible. With those priorities in mind, we recognize there is more work to do.”

“As drafted this year, the provisions of the Voting Rights Act would not take effect until the 2028 election cycle. Therefore, we will work over the course of the off-session to put forward as strong a bill as possible for consideration in 2027 and will continue to prioritize the Voting Rights Act in the upcoming session,” the statement continued.

The reaction from one angry advocate, Sen. Tiara Mack: “I’m not done fighting.”

In Vermont, one entrepreneur is trying to unload what might be the most unusual freebie in real estate: an entire former college campus. Raj Bhakta, a onetime Apprentice contestant and whiskey maker who scooped up Green Mountain College’s 115-acre Poultney campus at auction in 2020 for under $5 million, is now offering its 16 buildings and grounds to a new steward—for nothing, reports the Wall Street Journal. His ambitious plan to turn the shuttered school into a resort with hotel rooms, condos, a distillery, restaurant, and spa never made it past Vermont’s permitting maze or local tensions, and the site now hosts little more than a small elementary school started by his wife.

Bhakta says he’s sifting through more than 50 proposals and wants a group aligned with his vision of reviving “the United States, Western civilization and Christendom through faith-based education.” Donating the school to a religious organization poses another problem for Poultney, one local business owner tells WCAX: “There will be no property taxes ever paid, but the town will have to deal with the burden of having that large institution down there.” Whoever takes it on will need deep pockets: Bhakta’s website warns to expect about $1.5 million a year in upkeep and delayed maintenance. The stalled project mirrors a larger national question as small colleges close: What, if anything, comes next for the campuses that once anchored their towns? In Poultney, optimism about Bhakta has faded into wariness that anyone can realistically take the property on.