Create accountLogin

Recent searches

No recent searches

Search options

Not available on assemblag.es.
assemblag.es is part of the decentralized social network powered by Mastodon.
assemblag.es is a Mastodon instance for people interested in thinking creatively and critically about technology, in the broadest sense. The only requirement is that you keep to the code of conduct! Note: we've paused new signups for a few days while we deal with the #twittermigration influx - please do request an invite, but be aware that for now functions more like a 'waiting list' than a knock on the door, until our numbers stabilise again :-)

Administered by:

Server stats:

161
active users
Learn more

assemblag.es: AboutProfiles directoryPrivacy policy

Hometown: About · View source code · v4.2.17+hometown-1.1.2

Mastodon: AboutGet the appKeyboard shortcutsView source code

Login

The Luddite

Nature* published an article titled "Loneliness and suicide mitigation for students using GPT3-enabled chatbots"

nature.com/articles/s44184-023

Even among the awful hype articles, this one is egregious, and that's before taking into account the sensitivity of the subject matter.

Here's my response, where I break down this outrageous piece of marketing pseudoscience:

theluddite.org/#!post/replika

NatureLoneliness and suicide mitigation for students using GPT3-enabled chatbots - npj Mental Health ResearchMental health is a crisis for learners globally, and digital support is increasingly seen as a critical resource. Concurrently, Intelligent Social Agents receive exponentially more engagement than other conversational systems, but their use in digital therapy provision is nascent. A survey of 1006 student users of the Intelligent Social Agent, Replika, investigated participants’ loneliness, perceived social support, use patterns, and beliefs about Replika. We found participants were more lonely than typical student populations but still perceived high social support. Many used Replika in multiple, overlapping ways—as a friend, a therapist, and an intellectual mirror. Many also held overlapping and often conflicting beliefs about Replika—calling it a machine, an intelligence, and a human. Critically, 3% reported that Replika halted their suicidal ideation. A comparative analysis of this group with the wider participant population is provided.
Jan 30, 2024, 07:27 · · · 21 · 29
*
The Luddite

After an email exchange with the journal, I did some digging, and it turns out that the lead author on this paper is the CEO of an AI company. I didn't think that this paper could get more egregious.

The Luddite

Another update: Shivon Zilis, the neuralink person and mother of two of Elon Musk's children, is on the board of the author's AI company.

So we have a paper claiming that AI chatbots can prevent suicide in student populations based on a survey of users of replika, the erotic chat app, which failed to disclose the erotic nature of replika, whose author also failed to disclose their conflict of interest as CEO of an AI education company, which is apparently in the Elon extended stupidverse. 🆒

The Luddite

More updates: Here's Replika's CEO, Eugenia Kuyda, a well-documented liar, touting the results of the study on Feb 16 podcast episode of "Cognitive Revolution."

pca.st/pma0q3bs

That Replika can maybe stop suicide by 25% or more is the cold open of the episode. Then after intro, at the 2:23 mark, the host says "Replika has been in the news again, this time, for a study out of Stanford, that found that 3% of its users found relief from suicidal ideation..."

Screenshot of: In this conversation, Nathan sits down with Eugenia Kuyda, Replika's Founder and CEO. They discuss recent research published in *Nature* [highlighted by me] conducted by Stanford on Replika user interactions, how Replika is providing their users with a relationship they can trust and confide in, what Eugenia has found surprising about the evolution of how people [cuts off]
ALT
The Luddite

Goddammit, I just realized why the study was conducted in 2021 but the results didn't come out until 2024.

forbes.com/sites/cindygordon/2

Replika launched a mental health product at the same time, and I strongly suspect that this is part of their press blitz.

ForbesReplika: Launches New Immersive AI Wellness Avatar ExperienceBy Cindy Gordon
The Luddite

That junk paper from Maples et al that I've been following for months just got cited in the NYT.

For those new here:

theluddite.org/#!post/replika

(tl;dr - study claims to "mitigate suicide" while dishonestly withholding crucial context that would undermine the company line. The lead author also fails to disclose at least two conflicts of interest.)

Archive link to NYT article:

archive.md/8q3ML#selection-130

Research on the long-term effects of A.I. companionship is fairly thin, since the technology is so new, but it does seem to be a short-term help in some cases. One study conducted by Stanford researchers in 2023 found that some users of A.I. companions reported decreased anxiety and increased feelings of social support. A few even reported that their A.I. companions had talked them out of suicide or self-harm.
ALT
*
The Luddite

Already said this earlier in the thread, but it bears repeating: The journal is aware of these problems and refuses to retract the article, despite our explicit request for them to do so. The Luddite then submitted an official response, to be published alongside the paper, but it's been languishing in review for several months while the company continues to use the fraudulent science to woo the press and the public.

*
The Luddite

Yet another update on this thread. Forbes has picked up the study.

forbes.com/sites/danfitzpatric

Headline: "AI Chatbot Helps Prevent Suicide Among Students, Stanford Study Finds"

Subheading: "30 Students Saved From Suicide By ChatGPT Based AI, According To Research"

Such a vivid picture. Makes you expect some sort of medical trial, not an online survey sent out to existing Replika users by people at the Stanford School of Education with no follow-up or verification of any kind.

ForbesAI Chatbot Helps Prevent Suicide Among Students, Stanford Study FindsBy Dan Fitzpatrick
The Luddite

More updates: This bullshit study that we've been following for 9 months got cited on the freakonomics podcast. Jump to 23min.

freakonomics.com/podcast/can-a

Good news, though. Our Matters Arising is through peer review and currently awaiting publication:

osf.io/preprints/psyarxiv/pkt6

Also worth noting that without my coauthor's UVM affiliation, there's no way I'd be shelling out 4 figures to respond in the journal.

FreakonomicsCan A.I. Companions Replace Human Connection? - FreakonomicsCan A.I. Companions Replace Human Connection? - Freakonomics
The Luddite

This thread is now 8 months old, and Maples et al's so-called findings have made it to Replika's wikipedia page.

en.wikipedia.org/wiki/Replika

A Stanford study found Replika beneficial for people with depression. Despite experiencing high levels of loneliness, users reported feeling a strong sense of social support from Replika. Users see it as a therapist, friend and intellectual mirror, with 3% reporting Replika played a crucial role in preventing suicide.[19]
ALT
*
The Luddite

Our friend Maples, whose "research" into Replika this thread has already criticized at great length, continues touting Replika's supposed mental health benefits, and outlets continue to seek her out as an expert.

esquire.com/news-politics/a624

Bethanie Maples, a Stanford University researcher who has studied the benefits that AI companions from apps like Replika can have for mental health, says that chatbots have been effective in getting those suffering from post-traumatic stress disorder to open up. “People coming back from war will disclose more to chatbots than humans because it feels safe to them,” she says.
ALT
*
The Luddite

It's so fucking cool that Maples is now who publications turn to when they need expert comment on AI's impact on mental health. It's been a gift watching her go from one obviously fraudulent paper on the topic to frequently quoted in mainstream publications. Here's another NYT one.

nytimes.com/2024/10/23/technol

“By and large, it’s the Wild West out there,” said Bethanie Maples, a Stanford researcher who has studied the effects of A.I. companionship apps on mental health. “I don’t think it’s inherently dangerous,” Ms. Maples said of A.I. companionship. “But there’s evidence that it’s dangerous for depressed and chronically lonely users and people going through change, and teenagers are often going through change,” she said.
ALT
The Luddite

Bethanie Maples: "My work actually has proven that AI companions can halt suicidal ideation"

youtu.be/-El-CFpOf2I?t=1198

Jump to ~3 min to hear the host discuss Weizenbaum. It's an insult to mention him in a video promoting exactly what he warned us against.

(scroll up in this 16 month old thread for context)

www.youtube.com - YouTubeProfitez des vidéos et de la musique que vous aimez, mettez en ligne des contenus originaux, et partagez-les avec vos amis, vos proches et le monde entier.
Mostafa Hussein Omar

@theluddite Oh! they didn't declare that in the COI statement!

Thank you for your investigation and the blog post.

Anthony
@theluddite@assemblag.es I facepalmed so hard I gave myself a bruise.
Anthony
@theluddite@assemblag.es She was formerly on OpenAI's board but left it in 2023. Depending on when the work on this paper was done she may still been on the OpenAI board. None of this passes a smell test.
The Luddite

@abucci The paper says that they collected the data in 2021, but it wasn't published until 2024. Seems like a long time!

It does smell awful. A search through /r/replika shows that other academic institutions are surveying replika users too.

We actually contacted the author, who never responded, then the journal, who asked us to submit a response, which is currently awaiting an author response and peer review. So there might be an update soon. We'll see! Dying to read her defense.

Anthony
@theluddite@assemblag.es oh wow it's great you did that! Get 'em!
Jessamyn

@theluddite "Passionate about Modernizing via AI" Yeah contributor stuff there is basically paid placement or press releases most of the time

The Luddite

@jessamyn everything is always so fucking stupid!!!!!

Anthony
@theluddite@assemblag.es It might be worth including that at the time the study was conducted Shivon Zilis was on the board of OpenAI. 1st author Maples had a two-pronged conflict of interest here: it's likely Zillis pressured her directly or indirectly to use OpenAI products in this published study, since Zillis had a board seat at both companies and a strong interest in doing exactly that.

All this makes Nature look like the patsy laundering all these conflicts. They shouldn't be publishing what amount to corporate press releases.
The Luddite

@abucci

EXCELLENT point. You already pointed this out but I somehow didn't connect the dots. If you're willing to let me credit you with this one, lmk what name and/or site to send people to.

Anthony
@theluddite@assemblag.es Credit's not necessary but I won't turn it down! My web site is https://bucci.onl and you can link that (name is Anthony Bucci of course!)
Anthony BucciAnthony BucciAnthony Bucci's personal web site
J.H.Noyes

@theluddite
Was there a corresponding reduction in the suicide rate?

The Luddite

@noyes that's not a question that this study can answer. They sent out a survey to existing replika users and 3% of them wrote in the comment box completely unprompted that were it not for replika they would've died by suicide.

If you scroll up in the thread, you'll find my breakdown of the the paper and all the problems that it has.

The Luddite

@abucci Thank you!

I doubt that these outlets bother to read the whole study, let alone see if there are responses, but it's worth a shot.

Bastian Greshake Tzovaras

@theluddite congrats on getting it into the journal, any refutations are always a nightmare to place!

The Luddite

@gedankenstuecke

Thanks! I still really don't understand why the journal won't update her conflict of interest declaration at the very least.

Bastian Greshake Tzovaras

@theluddite the problem are the conflict of interest the journals have themselves I think. You want to be seen as reputable but also get enough people submitting to charge the 4-figure sums from, so can’t go to far onto either end I guess :/

The Luddite

@gedankenstuecke

I guess the joke is on the universities for paying them to publish it, as always. 🤷

Bastian Greshake Tzovaras

@theluddite yeah, academics have managed to let themselves be suckered into the worst deal, doing all the labor not only for free but paying through the nose for it.

The Luddite

@gedankenstuecke

It's truly staggering. I've known for well over a decade (I was actually an undergrad at MIT when the Aaron Swartz arrest happened), and I'm still struggling to process the scale of the grift.

Bastian Greshake Tzovaras

@theluddite yeah, same here, even around the same time, maybe 2-3 years earlier - when I did my undergrad and some lecturer made an off-hand comment about open access :AngerOtter:

The Luddite

@jessamyn @gedankenstuecke

Haha of course you do. On the right side of any issue, you always find the librarians and the quakers

Bastian Greshake Tzovaras

@jessamyn @theluddite Oh 100%! And I hope I didn't give the impression that I didn't appreciate that! Librarians & archivists are hands-down the best :librarian_hushing: :triforce:

Jessamyn

@gedankenstuecke @theluddite Yeah, no, you're good, I just enjoy a good kvetch against journal publishers.

Phredd Groves

@theluddite Of course Freakonomics would thoughtlessly cite something this hare brained.

The Luddite

@phredd

Yup, that entire genre of just-so faux-technical explanation has rotted the average NYT-reader's brain. Freakonomics, Nudge, Brian Wansink, etc.

Anthony
@theluddite@assemblag.es This is how the AI hype sausage gets made. It's ugly to watch. I'm very glad you're so dogged about it.
The Luddite

@abucci I decided to check in because I got an email that her response to our response is finally done so they'll be publishing it at some point relatively soon.

Anthony
@theluddite@assemblag.es Oh wow. I can't wait to see how she tries to defend that mess of a "paper".
The Luddite

@abucci Me too! I'm so excited for the followup blogpost :)

Mark Dingemanse

@theluddite worth noting also that this is not Nature (the journal) but "npj mental health", one of dozens of knock off titles Springer Nature (the publisher) has spun up to profit from the nature dot com url branding

The Luddite

@dingemansemark

Yeah totally agreed. We put that clarification in the first footnote. We went back and forth a few times on how to handle that, ultimately deciding that if they're going to publish on that nature.com domain, then that's where we'll send the criticism. They're so, so clearly using ideas like "editorial independence" and spinoff journals to hide the ball and we're just not here for it.

Claus Aranha

@dingemansemark @theluddite Eh... I'm sure there are some well intentioned editors and reviewers in some of the MDPI, Frontiers, and Elsevier Journals. If Nature's publisher will do that, it might be a good thing to treat the Nature brand just the same way as other predatory publishers...

ExploreLive feeds
About