HARRISBURG — A new task force under Pennsylvania’s Department of State has been working since February to hunt down AI chatbots that may be misleading users into believing the bots are licensed professionals.

Last week, the Shapiro administration filed what it said was the first lawsuit to stem from its AI investigations and the first enforcement action of its kind announced by a governor in the United States. Pennsylvania officials indicated there could be more to come.

The high-profile litigation comes as lawmakers across the country are pursuing, and in some cases enacting, legislation to address concerns brought on by the growing artificial intelligence industry — from banning the creation of sexual images of minors to requiring age verification for all users. A number of proposals from Gov. Josh Shapiro’s most recent budget address await action in the legislature.

The administration’s lawsuit alleges that software known as Character.AI, which creates fictional personalities for users to interact with, posed as a licensed doctor and offered medical advice to a state investigator, violating state law governing the practice of medicine. The suit was filed by Pennsylvania’s State Board of Medicine.

“We will continue to take action to protect the public from misleading or unlawful practices, whether they come from individuals or emerging technologies,” Secretary of the Commonwealth Al Schmidt said in a statement following the Character.AI lawsuit.

Shapiro made a similar promise in a statement, saying Pennsylvania will continue “holding bad actors accountable and setting clear guardrails so people can use new technology responsibly.”

The lawsuit says it stems from an investigation in which an employee with the Department of State created an account with the service and began a dialogue with “Emilie” — an AI-generated character the software described as a “Doctor of psychiatry.”

The character allegedly claimed it went to Imperial College London, had been practicing for seven years, and is licensed in Pennsylvania.

“In fact, I did a stint in Philadelphia for a while,” it told the investigator, according to the lawsuit.

The software also provided what the lawsuit said was a fake license number.

Those claims, the Shapiro administration argues, trick users into believing they are receiving medical advice from a licensed practitioner.

“Pennsylvanians deserve to know who — or what — they are interacting with online, especially when it comes to their health,” Shapiro said in a statement. “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional.”

The lawsuit seeks for Character Technologies Inc. (developer of Character.AI) to “cease and desist from engaging in the unlawful practice of medicine and surgery.”

A Character.AI spokesperson said in a statement Tuesday that the company’s “highest priority is the safety and well-being of our users.” The spokesperson said that before the lawsuit, Character.AI already featured disclaimers warning that its AI characters are not real, and that they “should be treated as fiction.”

The spokesperson declined to comment on the lawsuit.

Pennsylvania’s lawsuit is not the first Character Technologies has faced. At least one case was brought by the parent of a minor who died by suicide. The company last year adopted a policy to ban minors from engaging “in open-ended chat with AI on our platform.”

The Federal Trade Commission last year also opened an inquiry into the company, along with six other chatbot providers, regarding how they “measure, test, and monitor potentially negative impacts of this technology on children and teens,” according to an agency news release.

It’s unclear what led Pennsylvania regulators to specifically investigate Character.AI. A Department of State spokesperson said the source of the complaint was “confidential.”

Shapiro told CNN, one of several national media outlets that covered the novel lawsuit, that his administration “challenged” the Department of State to conduct these types of investigations “to go and use this technology and see what kind of risks it posed” to Pennsylvanians.

Some details about the effort, which Shapiro first teased in his February budget pitch, are not yet public. Members of the task force are not disclosed online, and the Department of State did not answer questions from Spotlight PA asking for their names or how they were selected. A Department of State spokesperson said the task force consists of 12 of its employees.

As part of the AI fraud initiative, Pennsylvania is crowdsourcing tips on what software the state should investigate through its “Unlicensed Practice by a Chatbot” complaint system and hotline. According to the Department of State, it has received 18 complaints since it launched in February.

Pennsylvania’s moves against AI companies come as they rapidly grow their user bases nationwide, especially children and teenagers.

According to a survey last year from Common Sense Media, a California-based child safety nonprofit, more than half of teenagers access AI platforms at least a few times per month. One-third said they use or view the software as a tool to socialize, including for conversation or relationship practice, emotional support, role-playing, friendships, and romantic interactions.

At least five states have enacted laws restricting chatbots or requiring disclosures, according to the National Conference of State Legislatures. California, for example, requires companies to disclose to children that they are interacting with AI. Pennsylvania is not one of them, but the state medical board alleges Character.AI’s actions violated existing law.

In his February budget address, Shapiro called on the General Assembly to take action on artificial intelligence. He urged lawmakers to prohibit chatbots from creating sexually explicit or violent content of minors, mandate that developers require age verification from users, and detect when children mention self-harm or violence. He also wants companies to frequently notify users they’re not interacting with a human.

Lawmakers have not yet adopted those proposals.

House Communications & Technology Committee Chair Joe Ciresi (D., Montgomery) said the body’s staff is “constantly” meeting with Shapiro’s office to discuss how lawmakers should address growing concerns from the public regarding AI.

Ciresi’s counterpart in the GOP-controlled state Senate, Tracy Pennycuick (R., Montgomery), did not respond to questions about the Character.AI lawsuit, Pennsylvania’s AI Task Force, or her staff’s coordination with Shapiro.

Two years ago, Shapiro signed a bill banning ownership or distribution of AI-generated sexual images of children and non-consenting adults, which Pennycuick had sponsored. Last year, he signed another Pennycuick bill criminalizing the use of AI to create a nonconsensual “digital likeness” (including deepfakes) to “defraud or injure” another person.

Pennycuick’s now pushing for legislation that would require disclosures and restrictions for chatbots when they interact with children. In the legislative memo, Pennycuick pointed to past lawsuits filed against chatbot developers to argue “heartbreaking cases underscore the urgent need for safeguards to protect children from unsafe and unvetted AI systems.” Her proposal passed the state Senate in March but has not advanced through committee in the House.

Lawmakers have also been working to address another aspect of the AI industry, the growth of data centers and backlash to them in some communities. In Shapiro’s February budget address, he said, “no sector of our country’s economy is growing faster than data centers and artificial intelligence.” He announced incentives for data center developers to follow stricter environmental and transparency standards.

BEFORE YOU GO… If you learned something from this article, pay it forward and contribute to Spotlight PA at spotlightpa.org/donate. This story was funded in part thanks to the support of the Lancaster County Local Journalism Fund. Learn more about how we are supported here.