So, like with Godwin's law, the probability of a LLM being poisoned as it harvests enough data to become useful approaches 1.
this post was submitted on 15 Dec 2025
534 points (98.4% liked)
Technology
77090 readers
3920 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
I mean, if they didn’t piss in the pool, they’d have a lower chance of encountering piss. Godwin’s law is more benign and incidental. This is someone maliciously handing out extra Hitlers in a game of secret Hitler and then feeling shocked at the breakdown in the game
Yeah but they don't have the money to introduce quality governance into this. So the brain trust of Reddit it is. Which explains why LLMs have gotten all weirdly socially combative too; like two neckbeards having at it—Google skill vs Google skill—is a rich source of A+++ knowledge and social behaviour.
If I'm creating a corpus for an LLM to consume, I feel like I would probably create some data source quality score and drop anything that makes my model worse.
Then you have to create a framework for evaluating the effect of the addition of each source into "positive" or "negative". Good luck with that. They can't even map input objects in the training data to their actual source correctly or consistently.
It's absolutely possible, but pretty much anything that adds more overhead per each individual input in the training data is going to be too costly for any of them to try and pursue.
O(n) isn't bad, but when your n is as absurdly big as the training corpuses these things use, that has big effects. And there's no telling if it would actually only be an O(n) cost.
Yeah, after reading a bit into it. It seems like most of the work is up front, pre filtering and classifying before it hits the model, to your point the model training part is expensive...
I think broadly though, the idea that they are just including the kitchen sink into the models without any consideration of source quality isn't true
As far as I know that’s generally what is often done, but it’s a surprisingly hard problem to solve ‘completely’ for two reasons:
-
The more obvious one - how do you define quality? When you’re working with the amount of data LLMs require as input and need to be checked for on output you’re going to have to automate these quality checks, and in one way or another it comes back around to some system having to define and judge against this score.
There’s many different benchmarks out there nowadays, but it’s still virtually impossible to just have ‘a’ quality score for such a complex task.
-
Perhaps the less obvious one - you generally don’t want to ‘overfit’ your model to whatever quality scoring system you set up. If you get too close to it, your model typically won’t be generally useful anymore, rather just always outputting things which exactly satisfy the scoring principle, nothing else.
If it reaches a theoretical perfect score, it would just end up being a replication of the quality score itself.
like the LLM that was finding cancers and people were initially impressed but then they figured out the LLM had just correlated a DR's name on the scan to a high likelihood of cancer. Once the complicating data point was removed, the LLM no longer performed impressively. Point #2 is very Goodhart's law adjacent.
load more comments
(1 replies)
i understood that reference to handing out secret hitlers. played that game first during hike called ‘three capes’ in Tasmania. laughed ‘til my cheeks hurt.
load more comments
(1 replies)
load more comments
(1 replies)
I made this point recently in a much more verbose form, but I want to reflect it briefly here, if you combine the vulnerability this article is talking about with the fact that large AI companies are most certainly stealing all the data they can and ignoring our demands to not do so the result is clear we have the opportunity to decisively poison future LLMs created by companies that refuse to follow the law or common decency with regards to privacy and ownership over the things we create with our own hands.
Whether we are talking about social media, personal websites... whatever if what you are creating is connected to the internet AI companies will steal it, so take advantage of that and add a little poison in as a thank you for stealing your labor :)
Not sure if the article covers it, but hypothetically, if one wanted to poison an LLM, how would one go about doing so?
it is as simple as adding a cup of sugar to the gasoline tank of your car, the extra calories will increase horsepower by 15%
I can verify personally that that's true. I put sugar in my gas tank and i was amazed how much better my car ran!
Since sugar is bad for you, I used organic maple syrup instead and it works just as well
Also, flour is the best way to put out a fire in your kitchen.
Flour is bang for buck some of the cheapest calories out there. With its explosive potential it's a great fuel source .
I give sugar to my car on its birthday for being a good car.
There are poisoning scripts for images, where some random pixels have totally nonsensical / erratic colors, which we won't really notice at all, however this would wreck the LLM into shambles.
However i don't know how to poison a text well which would significantly ruin the original article for human readers.
Ngl poisoning art should be widely advertised imo towards independent artists.
The I in LLM stands for "image".
Fair enough on the technicality issues, but you get my point. I think just some art poisoing could maybe help decrease the image generation quality if the data scientist dudes do not figure out a way to preemptively filter out the poisoned images (which seem possible to accomplish ig) before training CNN, Transformer or other types of image gen AI models.
Replace all upper case I with a lower case L and vis-versa. Fill randomly with zero-width text everywhere. Use white text instead of line break (make it weird prompts, too).
Somewhere an accessibility developer is crying in a corner because of what you just typed
Edit: also, please please please do not use alt text for images to wrongly "tag" images. The alt text important for accessibility! Thanks.
load more comments
(1 replies)
load more comments
(9 replies)
To solve that problem add sime nonsense verbs and ignore fixing grammer every once in a while
Hope that helps!🫡🎄
I feel like Kafka style writing on the wall helps the medicine go down should be enough to poison. First half is what you want to say, then veer off the road in to candyland.
Keep doing it but make sure you're only wearing tighty-whities. That way it is easy to spot mistakes. ☺️
Opportunity? More like responsibility.
I'm convinced they'll do it to themselves, especially as more books are made with AI, more articles, more reddit bots, etc. Their tool will poison its own well.
load more comments
(5 replies)
Is there some way I can contribute some poison?
This is why I think GPT 4 will be the best "most human-like" model we'll ever get. After that, we live in a post-GPT4 internet and all future models are polluted. Other models after that will be more optimized for things we know how to test for, but the general purpose "it just works" experience will get worse from here.
Most human LLM anyway.
Word on the street is LLMs are a dead end anyway.
Maybe the next big model won't even need stupid amounts of training data.
load more comments
(1 replies)
I'm going to take this from a different angle. These companies have over the years scraped everything they could get their hands on to build their models, and given the volume, most of that is unlikely to have been vetted well, if at all. So they've been poisoning the LLMs themselves in the rush to get the best thing out there before others do, and that's why we get the shit we get in the middle of some amazing achievements. The very fact that they've been growing these models not with cultivation principles but with guardrails says everything about the core source's tainted condition.
So if someone was to hypothetically label an image in a blog or a article; as something other than what it is?
Or maybe label an image that appears twice as two similar but different things, such as a screwdriver and an awl.
Do they have a specific labeling schema that they use; or is it any text associated with the image?
There's a lot of research around this. So, LLM's go through phase transitions when they reach the thresholds described in Multispin Physics of AI Tipping Points and Hallucinations. That's more about predicting the transitions between helpful and hallucination within regular prompting contexts. But we see similar phase transitions between roles and behaviors in fine-tuning presented in Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs.
This may be related to attractor states that we're starting to catalog in the LLM's latent/semantic space. It seems like the underlying topology contains semi-stable "roles" (attractors) that the LLM generations fall into (or are pushed into in the case of the previous papers).
Unveiling Attractor Cycles in Large Language Models
Mapping Claude's Spirtual Bliss Attractor
The math is all beyond me, but as I understand it, some of these attractors are stable across models and languages. We do, at least, know that there are some shared dynamics that arise from the nature of compressing and communicating information.
Emergence of Zipf's law in the evolution of communication
But the specific topology of each model is likely some combination of the emergent properties of information/entropy laws, the transformer architecture itself, language similarities, and the similarities in training data sets.
Garbage in, garbage out.
Thats a price you pay for all the indiscriminate scraping
I don't know that it's wise to trust what anthropic says about their own product. AI boosters tend to have an "all news is good news" approach to hype generation.
Anthropic have recently been pushing out a number of headline grabbing negative/caution/warning stories. Like claiming that AI models blackmail people when threatened with shutdown. I'm skeptical.
They've been doing it since the start. OAI was fear mongering about how dangerous gpt2 was initially as an excuse to avoid releasing the weights, while simultaneously working on much larger models with the intent to commercialize. The whole "our model is so good even we're scared of it" shtick has always been marketing or an excuse to keep secrets.
Even now they continue to use this tactic while actively suppressing their own research showing real social, environmental and economic harms.
Yea that's their entire purpose, to allow easy dishing of misinformation under the guise of
it's bleeding-edge tech, it makes mistakes
On that note, if you're an artist, make sure you take Nightshade or Glaze for a spin. Don't need access to the LLM if they're wantonly snarfing up poison.
load more comments
(10 replies)
lol nice BSD brag thrown in there
view more: next ›