If we ask "can AI do that?" we're asking the wrong question. A better question is: "Is this an appropriate use of automation?"
Here the answer was obviously no, and that was clear ahead of time:
https://vice.com/en/article/qjvk97/eating-disorder-helpline-disables-chatbot-for-harmful-responses-after-firing-human-staff…