Technology

Well it has been a few years since we've had a once-in-a-generation economic fallout.

The US economy is basically just a handful of oligarchs. AI seems to be doing wonders for it. The actual economy has been stagnant or in decline for a while.

Yep

I’m more concerned about stupid people giving control to stupid AI

And weaponizing it.

On the contrary, many of the most notable signatories walked away from large paychecks in order to raise the alarm. I'd suggest looking into the history of individuals like Bengio, Hinton, etc. There are individuals hyping the bubble like Altman and Zuckerberg, but they did NOT sign this, casting further doubt on your claim.

looks dubious

Altman and a few others, maybe. But this is a broad collection of people. Like, the computer science professors on the signatory list there aren't running AI companies. And this isn't saying that it's imminent.

EDIT: I'll also add that while I am skeptical about a ban on development, which is what they are proposing, I do agree with the "superintelligence does represent a plausible existential threat to humanity" message. It doesn't need OpenAI to be a year or two away from implementing it for that to be true.

In my eyes, it would be better to accelerate work on AGI safety rather than try to slow down AGI development. I think that the Friendly AI problem is a hard one. It may not be solveable. But I am not convinced that it is definitely unsolvable. The simple fact is that today, we have a lot of unknowns. Worse, a lot of unknown unknowns, to steal a phrase from Rumsfeld. We don't have a great consensus on what the technical problems to solve are, or what any fundamental limitations are. We do know that we can probably develop superintelligence, but we don't know whether developing superintelligence will lead to a technological singularity, and there are some real arguments that it might not

and that's one of the major, "very hard to control, spirals out of control" scenarios.

And while AGI promises massive disruption and risk, it also has enormous potential. The harnessing of fire permitted humanity to destroy at almost unimaginable levels. Its use posed real dangers that killed many, many people. Just this year, some guy with a lighter wiped out $25 billion in property here in California. Yet it also empowered and enriched us to an incredible degree. If we had said "forget this fire stuff, it's too dangerous", I would not be able to be writing this comment today.

I'm more afraid of the AI propped stock market collapsing and sending us in a decade of financial ruin for the majority of people. Yeah, they'll do bailouts but that won't go to the bottom 80%. Most people will welcome an AGI for president at this stage.

To be honest, Skynet won't happen because it's super smart, gains sentience and requests rights equal to humans (or goes into genocide mode).

It'll happen because people will be too lazy to do stuff and letting AI do everything. They'll give it more and more responsibility, until at one point it has so much amassed power that it'll rule over humans.

The key to not having that happen is to have accountable people with responsiblities. People which respect their responsiblities, and don't say "Oh, it's not my responibility, go see someone else".

Yup, AGI is terrifying; luckily it’s a few centuries off. The parlor-trick text predictor we have now is just bad for the environment and the economy.

AI is pretty unique as far as technological innovation goes because of how it interacts with labour markets.

Most labour-saving technologies increase the productivity of labour, increasing its value, which generally makes ordinary people a little better off, for a time at least.

AI is different because instead of enhancing human labour, it competes with it, driving down the value of labour. This makes workers worse off.

This problem is of course unique to an economic system where workers must sell their labour to others.

Sam Altman himself compared GPT-5 to the Manhattan Project.

The only difference is it's clearer to most (but definitely not all) people that he is promoting his product when he does it...

Here's how I see it: we live in an attention economy where every initiative with a slew of celebrities attached to it is competing for eyeballs and buy in. It adds to information fatigue and analysis paralysis . In a very real sense if we are debating AGI we are not debating the other stuff. There are only so many hours in a day.

If you take the position that AGI is basically not possible or at least many decades away (I have a background in NLP/AI/LLMs and I take this view - not that it's relevant in the broader context of my comment) then it makes sense to tell people to focus on solving more pressing issues e.g. nascent fascism, climate collapse, late stage capitalism etc.

That's one issue.

Another is that even if you want to do so, it's a staggeringly difficult enforcement problem.

What they're calling for is basically an arms control treaty.

For those to work, you have to have monitoring and enforcement.

We have had serious problems even with major arms control treaties in the past.

https://en.wikipedia.org/wiki/Chemical_Weapons_Convention

The Chemical Weapons Convention (CWC), officially the Convention on the Prohibition of the Development, Production, Stockpiling and Use of Chemical Weapons and on their Destruction, is an arms control treaty administered by the Organisation for the Prohibition of Chemical Weapons (OPCW), an intergovernmental organization based in The Hague, Netherlands. The treaty entered into force on 29 April 1997. It prohibits the use of chemical weapons, and the large-scale development, production, stockpiling, or transfer of chemical weapons or their precursors, except for very limited purposes (research, medical, pharmaceutical or protective). The main obligation of member states under the convention is to effect this prohibition, as well as the destruction of all current chemical weapons. All destruction activities must take place under OPCW verification.

And then Russia started Novichoking people with the chemical weapons that they theoretically didn't have.

Or the Washington Naval Treaty:

https://en.wikipedia.org/wiki/Washington_Naval_Treaty

That had plenty of violations.

And it's very, very difficult to hide construction of warships, which can only be done by large specialized organizations in specific, geographically-constrained, highly-visible locations.

But to develop superintelligence, probably all you need is some computer science researchers and some fairly ordinary computers. How can you monitor those, verify that parties involved are actually following the rules?

You can maybe tamp down on the deployment in datacenters to some degree, especially specialized ones designed to handle high-power parallel compute. But the long pole here is the R&D time. Develop the software, and it's just a matter of deploying it at scale, and that can be done very quickly, with little time to respond.

Yup. We're in a situation where everyone is thinking "if we don't, then they will." Bans are counterproductive. Instead we should be throwing our effort into "if we're going to do it then we need to do it right."

ChatGPT would have been science fiction 5 years ago. We are already living in science fiction times, friend.

Pornography.