Lawyers are getting in trouble because they use AI and submit fabricated citations about fabricated cases as precedent. A bunch of charges were recently thrown out in Wisconsin because of this, and it's not the first time such behavior has made the news.
The real analog here would be an electronics teacher leading his students to create a circuit that caught fire. If you’re confidently giving faulty information to people that don’t know any better, you’re not teaching them.
https://www.wpr.org/news/judge-sanctions-kenosha-county-da-a...
AI is indeed being understood to be an expert that replaces human judgement, and people are being hurt because of it.