With any tech this engaging and real-seeming, there is bound to be a very small percentage of people with dormant mental health issues who are predisposed to have extremely negative reactions. I can totally see how a hallucinating human-sounding chatbot could be the trigger paranoia/obsession in some people.
I don't think there's something inherently wrong with the technology. Mental stability is a bell curve; the majority of people are "normal", but there will always be an unfortunate subset who can react like this to strange new stimuli, through no fault of their own. It's no different to people getting unhealthily hooked on TV/smartphones and driven into conspiracies.
I disagree, because the television was never talking back to you. By the medium directly engaging with the person, the responsibility / enablement of this issue is more on the company than previous non interactive mediums. It would be like if a movie caused psychosis in people, and the sequel doubled down / enabled that. Even though there's no person (besides the consumer) in the loop, there is responsibility to train these systems to reduce the amount of these cases.
There are people that send the vast majority of the money they make to televangelists who are not speaking to them 'directly' and yet those individuals would tell you said speaker is communicating directly with them.
This is how a lot of propaganda over the radio and TV works.
Great point, actually. In many respects these language models doing this could be similar to how YouTube used to platform Alex Jones before removing him. Regardless, I still believe its the responsibility of the creators of these models to work on mitigations
A key question I think is whether these models can be made safer for mentally ill without cutting into utility for the healthy. Is the danger inherent to this technology, or is it incidental?
And here I think we are fortunate that there doesn't seem to be tradeoff.
Agreed here. My personal opinion is that this is fixable, but it would require reinforcement into the models that counteract the sycophantic stickiness that most companies use to drive engagement, so it won't really ever be a priority.
contradict each other. There is always a person in the loop, and the LLM is actually reacting to their messages, however wrong it turns out. They could have chosen a positive interaction instead. The LLM reflects back what the human puts in.
Sorry, should have clarified. By "no person in the loop" I meant from the side of the language model itself. The person is interacting with a thing which produces content for it, and the only person engaged in that process is the person consuming the content.
In terms of scale what's happening with just the algos, forget AI/drugs/synth bio stuff etc is on par with successful plant or animal domestication which took few centuries because of scaling issues that tech has solved.
A "normal" corn plant today doesn't look anything like the one nature produces. And the "normal" dog, cat, horse, chicken or cow can't survive outside very carefully controlled and built environments.
This generation of technologists aren't taught the Law of Requiste Variety and are totally oblivious about what happens when stability of systems is tied to "normality".
Covid reminded us how "normal" everything is. Feels more and more like a waste of time telling or teaching the tech domesticated herd anything at all.
I don't think there's something inherently wrong with the technology. Mental stability is a bell curve; the majority of people are "normal", but there will always be an unfortunate subset who can react like this to strange new stimuli, through no fault of their own. It's no different to people getting unhealthily hooked on TV/smartphones and driven into conspiracies.