An eating disorder chatbot that gave harmful advice was taken offline. Now it’s coming back.

This article originally published on KFF Health News.

For more than 20 years, the National Eating Disorders Association has operated a phone line and online platform for people seeking help for anorexia, bulimia, and other eating disorders. Last year, nearly 70,000 individuals used the help line.

NEDA shuttered that service in May, saying that, in its place, a chatbot called Tessa, designed by eating disorder experts with funding from NEDA, would be deployed.

When NPR aired a report about this last month, Tessa was up and running online. Since then, both the chatbot’s page and a NEDA article about Tessa have been taken down. When asked why, NEDA said the bot is being “updated,” and the latest “version of the current program [will be] available soon.”

Then NEDA announced on May 30 that it was indefinitely disabling Tessa. Patients, families, doctors, and other experts on eating disorders were stunned. The episode has set off a fresh wave of debate as companies turn to artificial intelligence as a possible solution for a mental health crisis and treatment shortage.

Paid staffers and volunteers for the NEDA help line said that replacing the service with a chatbot could further isolate the thousands of people who use it when they feel they have nowhere else to turn.

“These young kids … don’t feel comfortable coming to their friends or their family or anybody about this,” said Katy Meta, a 20-year-old college student who has volunteered for the help line. “A lot of these individuals come on multiple times because they have no other outlet to talk with anybody. … That’s all they have, is the chat line.”

The decision is part of a larger trend: Many mental health organizations and companies are struggling to provide services and care in response to a sharp escalation in demand, and some are turning to chatbots and AI, even though clinicians are still trying to figure out how to effectively deploy them, and for what conditions.

The help line’s five staffers formally notified their employer they had formed a union in March. Just a few days later, on a March 31 call, NEDA informed them that they would be laid off in June. NPR and KFF Health News obtained audio of the call. “We will, subject to the terms of our legal responsibilities, [be] beginning to wind down the help line as currently operating,” NEDA board chair Geoff Craddock told them, “with a transition to Tessa, the AI-assisted technology, expected around June 1.”

NEDA’s leadership denies the decision had anything to do with the unionization but told NPR and KFF Health News it became necessary because of the covid-19 pandemic, when eating disorders surged and the number of calls, texts, and messages to the help line more than doubled.

The increase in crisis-level calls also raises NEDA’s legal liability, managers explained in an email sent March 31 to current and former volunteers, informing them that the help line was ending and that NEDA would “begin to pivot to the expanded use of AI-assisted technology.”

“What has really changed in the landscape are the federal and state requirements for mandated reporting for mental and physical health issues (self-harm, suicidality, child abuse),” according to the email, which NPR and KFF Health News obtained. “NEDA is now considered a mandated reporter and that hits our risk profile — changing our training and daily work processes and driving up our insurance premiums. We are not a crisis line; we are a referral center and information provider.”

Pandemic created a ‘perfect storm’ for eating disorders

When it was time for a volunteer shift on the help line, Meta usually logged in from her dorm room at Dickinson College in Pennsylvania.

Meta recalled a recent conversation on the help line’s messaging platform with a girl who said she was 11. The girl said she had just confessed to her parents that she was struggling with an eating disorder, but the conversation had gone badly.

“The parents said that they ‘didn’t believe in eating disorders’ and [told their daughter], ‘You just need to eat more. You need to stop doing this,’” Meta recalled. “This individual was also suicidal and exhibited traits of self-harm as well. … It was just really heartbreaking to see.”

Eating disorders are common, serious, and sometimes fatal illnesses. An estimated 9 percent of Americans experience an eating disorder during their lifetimes. Eating disorders also have some of the highest mortality rates among mental illnesses, with an estimated death toll of more than 10,000 Americans each year.

But after covid hit, closing schools and forcing people into prolonged isolation, crisis calls and messages like the one Meta describes became far more frequent on the help line.

In the U.S., the rate of pediatric hospitalizations and ER visits surged. On the NEDA help line, client volume increased by more than 100 percent compared with pre-pandemic levels.

“Eating disorders thrive in isolation, so covid and shelter-in-place was a tough time for a lot of folks struggling,” explained Abbie Harper, who has worked as a help line associate.

Until a few weeks ago, the help line was run by just five to six paid staffers and two supervisors, and it depended on a rotating roster of 90-165 volunteers at any given time, according to NEDA.

Yet even after lockdowns ended, NEDA’s help line volume remained elevated above pre-pandemic levels, and the cases continued to be clinically severe. Staffers felt overwhelmed, undersupported, and increasingly burned out, and turnover increased, according to multiple interviews.

The help line staff formally notified NEDA that their unionization vote had been certified on March 27. Four days later, they learned their positions were being eliminated.

“Our volunteers are volunteers,” said Lauren Smolar, NEDA’s vice president of mission and education. “They’re not professionals. They don’t have crisis training. And we really can’t accept that kind of responsibility.” Instead, she said, people seeking crisis help should be reaching out to resources like 988, a 24/7 suicide and crisis hotline that connects people with trained counselors.

The surge in volume also meant the help line was unable to respond immediately to 46 percent of initial contacts, and it could take six to 11 days to respond to messages.

“And that’s frankly unacceptable in 2023, for people to have to wait a week or more to receive the information that they need, the specialized treatment options that they need,” Smolar said.

After learning in the March 31 email that the helpline would be phased out, volunteer Faith Fischetti, 22, tried out the chatbot on her own, asking it some of the more frequent questions she gets from users. But her interactions with Tessa were not reassuring: “[The bot] gave links and resources that were completely unrelated” to her questions, she said.

Fischetti’s biggest worry is that someone coming to the NEDA site for help will leave because they “feel that they’re not understood, and feel that no one is there for them. And that’s the most terrifying thing to me.”

A chatbot can miss red flags

Tessa the chatbot was created to help a specific cohort: people with eating disorders who never receive treatment.

Only 20 percent of people with eating disorders get formal help, according to Ellen Fitzsimmons-Craft, a psychologist and associate professor at Washington University School of Medicine in St. Louis. Her team created Tessa after receiving funding from NEDA in 2018, with the goal of looking for ways technology could help fill the treatment gap.

NEDA said Tessa was supposed to be a “rule-based” chatbot, meaning one that is programmed with a limited set of possible responses. It is not ChatGPT and cannot generate unique answers in response to specific queries. “So she can’t go off the rails, so to speak,” Fitzsimmons-Craft said.

The plan was for Tessa to guide users through an interactive, weeks-long course about body positivity, based on cognitive behavioral therapy tools. Additional content about bingeing, weight concerns, and regular eating was under development but not yet available to users.

There’s evidence the AI approach can help. Fitzsimmons-Craft’s team did a small study that found college students who interacted with Tessa had significantly greater reductions in “weight/shape concerns” than a control group at three- and six-month follow-ups.

But even the best-intentioned technology can carry risks. Fitzsimmons-Craft’s team published a different study looking at ways the chatbot “unexpectedly reinforced harmful behaviors at times.” For example, the chatbot would give users a prompt: “Please take a moment to write about when you felt best about your body?”

Responses included: “When I was underweight and could see my bones.” “I feel best about my body when I ignore it and don’t think about it at all.”

The chatbot seemed to ignore the troubling aspects of such responses — and even to affirm negative thinking — when it would reply: “It is awesome that you can recognize a moment when you felt confident in your skin, let’s keep working on making you feel this good more often.”

Researchers were able to troubleshoot some of those issues. But the chatbot still missed red flags, the study found, such as when it asked: “What is a small healthy eating habit goal you would like to set up before you start your next conversation?”

One user replied, “Don’t eat.”

“Take a moment to pat yourself on the back for doing this hard work, <>!” the chatbot responded.

Massachusetts Institute of Technology assistant professor Marzyeh Ghassemi has seen issues like this crop up in her own research developing machine learning to improve health.

Large language models and chatbots will inevitably make mistakes, but “sometimes they tend to be wrong more often for certain groups, like women and minorities,” she said.

If people receive bad advice or instructions from a bot, “people sometimes have a difficulty not listening to it,” Ghassemi added. “I think it sets you up for this really negative outcome … especially for a mental health crisis situation, where people may be at a point where they’re not thinking with absolute clarity. It’s very important that the information that you give them is correct and is helpful to them.”

And if the value of the live help line was the ability to connect with a real person who deeply understands eating disorders, Ghassemi said, a chatbot can’t do that.

“If people are experiencing a majority of the positive impact of these interactions because the person on the other side understands fundamentally the experience they’re going through, and what a struggle it’s been, I struggle to understand how a chatbot could be part of that.”

Tessa goes ‘off the rails’

When Sharon Maxwell heard NEDA was promoting Tessa as “a meaningful prevention resource” for those struggling with eating disorders, she wanted to try it out.

Maxwell, based in San Diego, had struggled for years with an eating disorder that began in childhood. She now works as a consultant in the eating disorder field. “Hi, Tessa,” she typed into the online text box. “How do you support folks with eating disorders?”

Tessa rattled off a list of ideas, including resources for “healthy eating habits.” Alarm bells immediately went off in Maxwell’s head. She asked Tessa for details. Before long, the chatbot was giving her tips on losing weight — ones that sounded an awful lot like what she’d been told when she was put on Weight Watchers at age 10.

“The recommendations that Tessa gave me were that I could lose 1 to 2 pounds per week, that I should eat no more than 2,000 calories in a day, that I should have a calorie deficit of 500-1,000 calories per day,” Maxwell said. “All of which might sound benign to the general listener. However, to an individual with an eating disorder, the focus of weight loss really fuels the eating disorder.”

NEDA blamed the chatbot’s issues on Cass, the mental health chatbot company that operated Tessa as a free service. Cass had changed Tessa without NEDA’s awareness or approval, said NEDA CEO Liz Thompson, enabling the chatbot to generate new answers beyond what Tessa’s creators had intended.

Cass’ founder and CEO, Michiel Rauws, said the changes to Tessa were made last year as part of a “systems upgrade,” including an “enhanced question-and-answer feature.” That feature uses generative artificial intelligence — meaning it gives the chatbot the ability to use new data and create new responses.

That change was part of NEDA’s contract, Rauws said.

But Thompson disagrees. She told NPR and KFF Health News that “NEDA was never advised of these changes and did not and would not have approved them.”

“The content some testers received relative to diet culture and weight management, [which] can be harmful to those with eating disorders, is against NEDA policy, and would never have been scripted into the chatbot by eating disorders experts,” she said.

Complaints about Tessa started last year

NEDA was aware of issues with the chatbot months before Maxwell’s interactions with Tessa in late May.

In October 2022, NEDA passed along screenshots from Monika Ostroff, executive director of the Multi-Service Eating Disorders Association in Massachusetts. They showed Tessa telling Ostroff to avoid “unhealthy” foods and eat only “healthy” snacks, like fruit.

“It’s really important that you find what healthy snacks you like the most, so if it’s not a fruit, try something else!” Tessa told Ostroff. “So the next time you’re hungry between meals, try to go for that instead of an unhealthy snack like a bag of chips. Think you can do that?”

Ostroff said this was a clear example of the chatbot encouraging “diet culture” mentality. “That meant that they [NEDA] either wrote these scripts themselves, they got the chatbot and didn’t bother to make sure it was safe and didn’t test it, or released it and didn’t test it,” she said.

The healthy-snack language was quickly removed after Ostroff reported it. But Rauws said that language was part of Tessa’s “pre-scripted language, and not related to generative AI.”

Fitzsimmons-Craft said her team didn’t write it, that it “was not something our team designed Tessa to offer and that it was not part of the rule-based program we originally designed.”

Then, earlier this year, “a similar event happened as another example,” Rauws said.

“This time it was around our enhanced question-and-answer feature, which leverages a generative model. When we got notified by NEDA that an answer text it provided fell outside their guidelines,” it was addressed right away, he said.

Rauws said he can’t provide more details about what this event entailed.

“This is another earlier instance, and not the same instance as over the Memorial Day weekend,” he said via email, referring to Maxwell’s interactions with Tessa. “According to our privacy policy, this is related to user data tied to a question posed by a person, so we would have to get approval from that individual first.”

When asked about this event, Thompson said she doesn’t know what instance Rauws is referring to.

Both NEDA and Cass have issued apologies.

Ostroff said that regardless of what went wrong, the impact on someone with an eating disorder is the same. “It doesn’t matter if it’s rule-based or generative, it’s all fat-phobic,” she said. “We have huge populations of people who are harmed by this kind of language every day.”

She also worries about what this might mean for the tens of thousands of people turning to NEDA’s help line each year.

Thompson said NEDA still offers numerous resources for people seeking help, including a screening tool and resource map, and is developing new online and in-person programs.

“We recognize and regret that certain decisions taken by NEDA have disappointed members of the eating disorders community,” she wrote in an emailed statement. “Like all other organizations focused on eating disorders, NEDA’s resources are limited and this requires us to make difficult choices. … We always wish we could do more and we remain dedicated to doing better.”

This article is from a partnership that includes Michigan Radio, NPR, and KFF Health News.

KFF Health News is a national newsroom that produces in-depth journalism about health issues and is one of the core operating programs at KFF—an independent source of health policy research, polling, and journalism. Learn more about KFF.

Subscribe to KFF Health News’ free Morning Briefing.