politics

Absolutely this. LLM basically is trained to be good at fooling us into thinking it is intelligent, and it is very good at it.

It doesn't demonstrate how good it is in what it is doing, it demonstrates how easy it is to fool us.

My company provides copilot for software engineering and I use it in my IDE.

The problem is that it produces code that looks accurate, but it often isn't. I frequently tend to disable it. I think it might help in area where I don't know what I'm doing, so it can get some working code, but it is a double edged sword, because if I don't know what I'm doing I will not be able to catch issues.

I also noticed that what it produces when correct, I can frequently write a simpler and shorter version that fits my use case. It looks very likely like code you see students put on GitHub when they post their homework assignment, and I guess that's what it was trained on.

It could hallucinate a citation that never even existed as a fictional case

That's what happened in this case reviewed by Legal Eagle.

The lawyer provided a brief that cited cases that the judge could not find. The judge requested paper copies of the cases and that's when the lawyer handed over some dubious documents. The judge then called the lawyer into the court to ask why he submitted fraudulent cases and why he shouldn't have his law licence revoked. The lawyer fessed up that he asked ChatGPT to write the brief and didn't check the citations. When the judge asked for the cases, the lawyer went back to ask ChatGPT for them, and it generated the cases...but they were clearly not real. So much so that the defendants names would change throughout the case, the judges who ruled on the cases were from different districts, and they were all about a page long when real case rulings tend to be dozens of pages.

I think it would be quite reasonable for any lawyer who files something that includes references to case law that doesn’t exist to simply be disbarred.

The courts are backed up enough without having to deal with this bullshit. And it shows clear lack of concern for properly representing their client

LLMs are basically extremely complex text autocomplete systems.

Most smartphones these days have such systems learn from yourself, personally (and of course use all of your vocab data to make a profile of you and sell it to marketers, law enforcement, whoever is buying)... but LLMs learn from... a little bit of everything, all of the time.

Chatbots are text completion models, improv machines basically, so they don’t really have that ability. You could look at logprobs I guess (aka is it guessing a bunch of words pretty evenly?), but that’s unreliable. Even adding a “I don’t know” token wouldn’t work because that’s not really trainable into text datasets: they don't know when they don’t know, it’s all just modeling what next word is most likely.

Some non-autoregressive architectures would be better, but unfortunately “cutting edge” models people interact with like ChatGPT are way more conservatively developed than you’d think. Like, they’ve left tons of innovations unpicked.

Is there even any suitable "confidence" measure within the LLM that it could use to know when it needs to emit an "I don't know" response? I wonder whether there's even any consistent and measurable difference between times when it seems to know what it's talking about and times when it is talking BS. That might be something that exists in our own cognition but has no counterpart in the workings of an LLM. So it may not even be feasible to engineer it to say "I don't know" when it doesn't know. It can't just straightforwardly look at how many sources it has for an answer and how good they were, because LLMs have typically worked in a more holistic way: each item of training data nudges the behaviour of the whole system, but it doesn't leave behind any sign that says "I did this," or any particular piece of knowledge or behaviour that can be ascribed to that training item.

hehheheh