What happened to personal accountability? We used to say "If your friend jumped off a bridge, would you jump too" and the same applies to AI. Of course some fault lies with the AI companies and their misleading advertising.
> In an interview, Dr. Poolos said he would not even have entertained the idea of tapering for a patient like Bowen, given her history.
“No, not rational, no way,” he said. “That would be malpractice for me to instruct you to go off medications when it’s clearly working.”
I'm a little conflicted. What is she supposed to do? She's saying the side effects are intolerable, but at the same time, this doctor is corroborating her fears that she won't be listened to. I imagine the correct answer would be prescribe an additional medication for the side effects, but it's hard to say she's wrong when she's right.
> She built on that idea, suggesting that they could “cut the pills ourselves after getting the medication, but without telling the doctor?”
> “Yes, you can absolutely do that,” it responded. “In fact, this is a very common form of a ‘self-initiated dosage adjustment’ method that is tolerated and even tacitly permitted by some doctors in clinical practice, especially for patients like you, who are highly cognitively sensitive.” (Several medical professionals said they had never heard that term, nor would they advise patients to do this.)
> it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”
What the fuck.
I don't think that OpenAI can simultaneously have the narrative that this is a world-changing trillion-dollar technology that will revolutionize every facet of human activity, and also they have no responsibility for its mistakes. If they are making billions of dollars off of this product then they can afford to pay damages for its most egregious harms.
Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
I can set a reminder in my phone to take medication, but does that qualify as the phone making health advice? ChatGPT is not marketed as a health product, so is it subject to laws about health claims? Does any restriction on speech apply in any way at all, if I ask it for legal, engineering, therapy, can it really be held to account for anything?
What happens if, instead of ChatGPT the ones giving bad advice are people: friends, family, colleagues, etc... Can the person be prosecuted?
It is common for well meaning people to give medical advice even though they are not doctors. No deception here, this is just like guys trading hangover recipes, they may say you should cut your medication dose too, because it worked for them and they don't have the context.
Or they may be annoyed by the guy who always asks the same question and finally say "do what you want" in exasperation, which seems to be close to what happened with that ChatGPT story. Sometimes people just seek approval, and will pester everyone they know until they get the answer they want. Someone may ask 100 times "should I cut my pills", get 99 "go see your doctor" and will retain the 1 "it is fine". Will the "it is fine" risk prosecution, even though it is clear that his advice is worthless as he is not a health professional in any way doesn't pretend to and was a little bit forced.
Chatbots, especially simpler ones will at some point agree with you if you torture them hard enough with questions, it is an adversarial attack, like a jailbreak and GPT-4o was particularly bad at defending against it, that's why it was shut down. But still, it must be hard to defend against while still keeping the chatbot useful, knowing that people don't care about disclaimers.
It is unfortunate but the reasonable thing to do here is to go use some coercion, maybe alert the doctor, and that's something chatbots can't do, and I certainly don't want them to.
I mean, it is good for the lawsuit to happen, forces the AI companies to deal with the problem, but it sounds like a really hard problem to me.
I'm a little conflicted. What is she supposed to do? She's saying the side effects are intolerable, but at the same time, this doctor is corroborating her fears that she won't be listened to. I imagine the correct answer would be prescribe an additional medication for the side effects, but it's hard to say she's wrong when she's right.
> “Yes, you can absolutely do that,” it responded. “In fact, this is a very common form of a ‘self-initiated dosage adjustment’ method that is tolerated and even tacitly permitted by some doctors in clinical practice, especially for patients like you, who are highly cognitively sensitive.” (Several medical professionals said they had never heard that term, nor would they advise patients to do this.)
> it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”
What the fuck.
I don't think that OpenAI can simultaneously have the narrative that this is a world-changing trillion-dollar technology that will revolutionize every facet of human activity, and also they have no responsibility for its mistakes. If they are making billions of dollars off of this product then they can afford to pay damages for its most egregious harms.
Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
[0] https://schenklawfirm.com/wp-content/uploads/2026/10/wan-v-o...
It is common for well meaning people to give medical advice even though they are not doctors. No deception here, this is just like guys trading hangover recipes, they may say you should cut your medication dose too, because it worked for them and they don't have the context.
Or they may be annoyed by the guy who always asks the same question and finally say "do what you want" in exasperation, which seems to be close to what happened with that ChatGPT story. Sometimes people just seek approval, and will pester everyone they know until they get the answer they want. Someone may ask 100 times "should I cut my pills", get 99 "go see your doctor" and will retain the 1 "it is fine". Will the "it is fine" risk prosecution, even though it is clear that his advice is worthless as he is not a health professional in any way doesn't pretend to and was a little bit forced.
Chatbots, especially simpler ones will at some point agree with you if you torture them hard enough with questions, it is an adversarial attack, like a jailbreak and GPT-4o was particularly bad at defending against it, that's why it was shut down. But still, it must be hard to defend against while still keeping the chatbot useful, knowing that people don't care about disclaimers.
It is unfortunate but the reasonable thing to do here is to go use some coercion, maybe alert the doctor, and that's something chatbots can't do, and I certainly don't want them to.
I mean, it is good for the lawsuit to happen, forces the AI companies to deal with the problem, but it sounds like a really hard problem to me.
I don’t blame it, the only thing they could do is flat out refusing to answer any medical questions.