this post was submitted on 01 Jun 2025
261 points (96.1% liked)

Technology

70942 readers
3430 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
261
I'm looking for an article showing that LLMs don't know how they work internally (feddit.it)
submitted 6 days ago* (last edited 6 days ago) by lgsp@feddit.it to c/technology@lemmy.world
72 comments fedilink hide all child comments
 

I found the aeticle in a post on the fediverse, and I can't find it anymore.

The reaserchers asked a simple mathematical question to an LLM ( like 7+4) and then could see how internally it worked by finding similar paths, but nothing like performing mathematical reasoning, even if the final answer was correct.

Then they asked the LLM to explain how it found the result, what was it's internal reasoning. The answer was detailed step by step mathematical logic, like a human explaining how to perform an addition.

This showed 2 things:

  • LLM don't "know" how they work

  • the second answer was a rephrasing of original text used for training that explain how math works, so LLM just used that as an explanation

I think it was a very interesting an meaningful analysis

Can anyone help me find this?

EDIT: thanks to @theunknownmuncher @lemmy.world https://www.anthropic.com/research/tracing-thoughts-language-model its this one

EDIT2: I'm aware LLM dont "know" anything and don't reason, and it's exactly why I wanted to find the article. Some more details here: https://feddit.it/post/18191686/13815095

you are viewing a single comment's thread
view the rest of the comments
[–] theparadox@lemmy.world 12 points 5 days ago (2 children)

More than enough people who claim to know how it works think it might be "evolving" into a sentient being inside it's little black box. Example from a conversation I gave up on... https://sh.itjust.works/comment/18759960

[–] theunknownmuncher@lemmy.world 6 points 5 days ago

I don't want to brigade, so I'll put my thoughts here. The linked comment is making the same mistake about self preservation that people make when they ask an LLM to "show it's work" or explain it's reasoning. The text response of an LLM cannot be taken at it's word or used to confirm that kind of theory. It requires tracing the logic under the hood.

Just like how it's not actually an AI assistant, but trained and prompted to output text that is expected to be what an AI assistant would respond with, if it is expected that it would pursue self preservation, then it will output text that matches that. It's output is always "fake"

That doesn't mean there isn't a real potential element of self preservation, though, but you'd need to dig and trace through the network to show it, not use the text output.

[–] AnneBonny@lemmy.dbzer0.com 2 points 5 days ago

Maybe I should rephrase my question:

Outside of comment sections on the internet, who has claimed or is claiming that LLMs have the capacity to reason?