Technology

i don't think it's emphasized enough that AI isn't just making up bogus citations with nonexistent books and articles, but increasingly actual articles and other sources are completely AI generated too. so a reference to a source might be "real," but the source itself is complete AI slop bullshit

https://www.tudelft.nl/en/2025/eemcs/scientific-study-exposes-publication-fraud-involving-widespread-use-of-ai

https://thecurrentga.org/2025/02/01/experts-fake-papers-fuel-corrupt-industry-slow-legitimate-medical-research/

the actual danger of it all should be apparent, especially in any field related to health science research

and of course these fake papers are then used to further train AI, causing factually wrong information to spread even more

Some people even think that adding things like “don’t hallucinate” and “write clean code” to their prompt will make sure their AI only gives the highest quality output.

Arthur C. Clarke was not wrong but he didn't go far enough. Even laughably inadequate technology is apparently indistinguishable from magic.

Everyone knows that AI chatbots like ChatGPT, Grok, and Gemini can often hallucinate sources.

No, no, apparently not everyone, or this wouldn't be a problem.

I had to explain to three separate family members what it means for an Ai to hallucinate. The look of terror on their faces after is proof that people have no idea how "smart" a LLM chatbot is. They have been probably using one at work for a year thinking they are accurate.

No AI needed for that. These bloody librarians wouldn't let us have the Necronomicon either. Selfish bastards...

There's an old Monty Python sketch from 1967 that comes to mind when people ask a librarian for a book that doesn't exist.

They predicted the future.

I believe I got into a conversation on Lemmy where I was saying that there should be a big persistent warning banner stuck on every single AI chat app that "the following information has no relation to reality" or some other thing. The other person kept insisting it was not needed. I'm not saying it would stop all of these events, but it couldn't hurt.

I plugged my local AI into offline wikipedia expecting a source of truth to make it way way better.

It’s better, but I also can’t tell when it’s making up citations now, because it uses Wikipedia to support its own world view from pre training instead of reality.

So it’s not really much better.

Hallucinations become a bigger problem the more info they have (that you now have to double check)

I guess Thomas Fullman was right: "When humans find wisdom in cold replicas of themselves, the arrow of evolution will bend into a circle". That's from Automating the Mind. One of his best.

This and many other new problems are solved by applying reputation systems (like those banks use for your credit rating, or employers share with each other) in yet another direction. "This customer is an asshole, allocate less time for their requests and warn them that they have a bad history of demanding nonexistent books". Easy.

Then they'll talk with their friends how libraries are all possessed by a conspiracy, similarly to how similarly intelligent people talk about Jewish plot to take over the world, flat earth and such.

They really should stop hiding them. We all deserve to have access to these secret books that were made up by AI since we all contributed to the training data used to write these secret books.