Technology

https://www.anthropic.com/research/tracing-thoughts-language-model its this one

Can’t help but here’s a rant on people asking LLMs to “explain their reasoning” which is impossible because they can never reason (not meant to be attacking OP, just attacking the “LLMs think and reason” people and companies that spout it):

LLMs are just matrix math to complete the most likely next word. They don’t know anything and can’t reason.

Anything you read or hear about LLMs or “AI” getting “asked questions” or “explain its reasoning” or talking about how they’re “thinking” is just AI propaganda to make you think they’re doing something LLMs literally can’t do but people sure wish they could.

In this case it sounds like people who don’t understand how LLMs work eating that propaganda up and approaching LLMs like there’s something to talk to or discern from.

If you waste egregiously high amounts of gigawatts to put everything that’s ever been typed into matrices you can operate on, you get a facsimile of the human knowledge that went into typing all of that stuff.

It’d be impressive if the environmental toll making the matrices and using them wasn’t critically bad.

TLDR; LLMs can never think or reason, anyone talking about them thinking or reasoning is bullshitting, they utilize almost everything that’s ever been typed to give (occasionally) reasonably useful outputs that are the most basic bitch shit because that’s the most likely next word at the cost of environmental disaster

I don't know how I work. I couldn't tell you much about neuroscience beyond "neurons are linked together and somehow that creates thoughts". And even when it comes to complex thoughts, I sometimes can't explain why. At my job, I often lean on intuition I've developed over a decade. I can look at a system and get an immediate sense if it's going to work well, but actually explaining why or why not takes a lot more time and energy. Am I an LLM?

By design, they don't know how they work. It's interesting to see this experimentally proven, but it was already known. In the same way the predictive text function on your phone keyboard doesn't know how it works.

It's the anthropic article you are looking for, where they performed open brain surgery equivalent to find out that they do maths in very strange and eerily humanlike operations, like they will estimate, then if it goes over calculate the last digit like I do. It sucks as a counting technique though

Define "know".

• An LLM can have text describing how it works and be trained on that text and respond with an answer incorporating that.

• LLMs have no intrinsic ability to "sense" what's going on inside them, nor even a sense of time. It's just not an input to their state. You can build neural-net-based systems that do have such an input, but ChatGPT or whatever isn't that.

• LLMs lack a lot of the mechanisms that I would call essential to be able to solve problems in a generalized way. While I think Dijkstra had a valid point:

  The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.

  ...and we shouldn't let our prejudices about how a mind "should" function internally cloud how we treat artificial intelligence...it's also true that we can look at an LLM and say that it just fundamentally doesn't have the ability to do a lot of things that a human-like mind can. An LLM is, at best, something like a small part of our mind. While extracting it and playing with it in isolation can produce some interesting results, there's a lot that it can't do on its own: it won't, say, engage in goal-oriented behavior. Asking a chatbot questions that require introspection and insight on its part won't yield interesting result, because it can't really engage in introspection or insight to any meaningful degree. It has very little mutable state, unlike your mind.

There was a study by Anthropic, the company behind Claude, that developed another AI that they used as a sort of "brain scanner" for the LLM, in the sense that allowed them to see sort of a model of how the LLM "internal process" worked