Elon Musk, Apple co-founder Steve Wozniak and 1,000 other tech leaders call for pause on AI development which poses a 'profound risk to society and humanity'

POSTED BY: 6IXSTRINGJACK
UPDATED: Tuesday, September 16, 2025 19:02
VIEWED: 1994
PAGE 1 of 2

1 2 NEXT PAGE

6IXSTRINGJACK

Thursday, March 30, 2023 10:24 AM

https://www.dailymail.co.uk/news/article-11914149/Musk-experts-urge-pa
use-training-AI-systems-outperform-GPT-4.html

Stephen Hawking warned you.

Keep doing what you're doing and we're going to be longing for the days when Fauci was weaponizing viruses in a Chinese lab.

JAYNEZTOWN

Thursday, March 30, 2023 2:48 PM

In the history of panic calling for a pause, this usually means they already screwed up

genie is out of the bottle

We already know with 4chans old pranks on Tay and Zo something weird was happening

its here, Pandora's Box has been opened

SECOND

Thursday, March 30, 2023 4:29 PM

If AI scaling is to be shut down, let it be for a coherent reason, by Scott Aaronson

There’s now an open letter arguing that the world should impose a six-month moratorium on the further scaling of AI models such as GPT, by government fiat if necessary, to give AI safety and interpretability research a bit more time to catch up. The letter is signed by many of my friends and colleagues, many who probably agree with each other about little else, over a thousand people including Elon Musk, Steve Wozniak, Andrew Yang, Jaan Tallinn, Stuart Russell, Max Tegmark, Yuval Noah Harari, Ernie Davis, Gary Marcus, and Yoshua Bengio.

Meanwhile, Eliezer Yudkowsky published a piece in TIME arguing that the open letter doesn’t go nearly far enough, and that AI scaling needs to be shut down entirely until the AI alignment problem is solved—with the shutdown enforced by military strikes on GPU farms if needed, and treated as more important than preventing nuclear war.

Readers, as they do, asked me to respond. Alright, alright. While the open letter is presumably targeted at OpenAI more than any other entity, and while I’ve been spending the year at OpenAI to work on theoretical foundations of AI safety, I’m going to answer strictly for myself.

More at https://scottaaronson.blog/?p=7174

The Joss Whedon script for Serenity, where Wash lives, is Serenity-190pages.pdf at
https://www.mediafire.com/folder/1uwh75oa407q8/Firefly

6IXSTRINGJACK

Thursday, March 30, 2023 7:36 PM

Quote:
Would your rationale for this pause have applied to basically any nascent technology — the printing press, radio, airplanes, the Internet? “We don’t yet know the implications, but there’s an excellent chance terrible people will misuse this, ergo the only responsible choice is to pause until we’re confident that they won’t”?

No. No. No. Yes.

The first three are strawmen. The internet fucked the world of humans up. AI is just going to finish it off.

Quote:
Why six months? Why not six weeks or six years?

Good question. Make it forever.

Quote:
When, by your lights, would we ever know that it was safe to resume scaling AI—or at least that the risks of pausing exceeded the risks of scaling? Why won’t the precautionary principle continue for apply forever?

Right.

Quote:
Were you, until approximately last week, ridiculing GPT as unimpressive, a stochastic parrot, lacking common sense, piffle, a scam, etc. — before turning around and declaring that it could be existentially dangerous?

Nope.

Quote:
How can you have it both ways?

You can't. I'm not a hypocrite.

Quote:
If the problem, in your view, is that GPT-4 is too stupid, then shouldn’t GPT-5 be smarter and therefore safer? Thus, shouldn’t we keep scaling AI as quickly as we can … for safety reasons? If, on the other hand, the problem is that GPT-4 is too smart, then why can’t you bring yourself to say so?

I've never made an argument about AI being too stupid. AI evolving to the point through human folly to become too smart was always just an eventuality. The argument has always been that humans are too stupid... And too full of hubris.

There is possibly a race of beings on another planet in our near infinite universe that is capable of being responsible shepherds of AI technology. Human beings are not capable of that.

Not even close.

--------------------------------------------------

Growing up in a Republic was nice... Shame we couldn't keep it.

SECOND

Thursday, March 30, 2023 8:08 PM

Quote:
Originally posted by 6IXSTRINGJACK:

There is possibly a race of beings on another planet in our near infinite universe that is capable of being responsible shepherds of AI technology. Human beings are not capable of that.

Not even close.

The same guy you're criticizing in excessive detail, Scott Aaronson, had more to say about it elsewhere:

(1) Orthodox AI-riskers tend to believe that humanity will survive or be destroyed based on the actions of a few elite engineers over the next decade or two. Everything else—climate change, droughts, the future of US democracy, war over Ukraine and maybe Taiwan—fades into insignificance except insofar as it affects those engineers.

We Reform AI-riskers, by contrast, believe that AI might well pose civilizational risks in the coming century, but so does all the other stuff, and it’s all tied together. An invasion of Taiwan might change which world power gets access to TSMC GPUs. Almost everything affects which entities pursue the AI scaling frontier and whether they’re cooperating or competing to be first.

(2) Orthodox AI-riskers believe that public outreach has limited value: most people can’t understand this issue anyway, and will need to be saved from AI despite themselves.

We Reform AI-riskers believe that trying to get a broad swath of the public on board with one’s preferred AI policy is something close to a deontological imperative. (Deontology is an ethical theory that says actions are good or bad according to a clear set of rules. Its name comes from the Greek word deon, meaning duty. Actions that align with these rules are ethical, while actions that don't aren't.)

(3) Orthodox AI-riskers worry almost entirely about an agentic, misaligned AI that deceives humans while it works to destroy them, along the way to maximizing its strange utility function.

We Reform AI-riskers entertain that possibility, but we worry at least as much about powerful AIs that are weaponized by bad humans, which we expect to pose existential risks much earlier in any case.

(4) Orthodox AI-riskers have limited interest in AI safety research applicable to actually-existing systems (LaMDA, GPT-3, DALL-E2, etc.), seeing the dangers posed by those systems as basically trivial compared to the looming danger of a misaligned agentic AI.

We Reform AI-riskers see research on actually-existing systems as one of the only ways to get feedback from the world about which AI safety ideas are or aren’t promising.

(5) Orthodox AI-riskers worry most about the “FOOM” scenario, where some AI might cross a threshold from innocuous-looking to plotting to kill all humans in the space of hours or days.

We Reform AI-riskers worry most about the “slow-moving trainwreck” scenario, where (just like with climate change) well-informed people can see the writing on the wall decades ahead, but just can’t line up everyone’s incentives to prevent it.

(6) Orthodox AI-riskers talk a lot about a “pivotal act” to prevent a misaligned AI from ever being developed, which might involve (e.g.) using an aligned AI to impose a worldwide surveillance regime.

We Reform AI-riskers worry more about such an act causing the very calamity that it was intended to prevent.

(7) Orthodox AI-riskers feel a strong need to repudiate the norms of mainstream science, seeing them as too slow-moving to react in time to the existential danger of AI.

We Reform AI-riskers feel a strong need to get mainstream science on board with the AI safety program.

Much more at https://scottaaronson.blog/?p=6821

In another post by Scott Aaronson

As it happens, I became aware of the AI alignment community a long time back, around 2006. Here’s Eliezer Yudkowsky, who’s regarded as the prophet of AI alignment, of the right side of that spectrum that showed before.

He’s been talking about the danger of AI killing everyone for more than 20 years. He wrote the now-famous “Sequences” that many readers of my blog were also reading as they appeared, so he and I bounced back and forth.

Bu

6IXSTRINGJACK

Thursday, March 30, 2023 8:16 PM

Stephen Hawking was a lot smarter than this dipshit.

I reject the term Orthodox AI-riskers. And I'm sure that a lot of the greater men on that list of 1,000 do as well.

--------------------------------------------------

Growing up in a Republic was nice... Shame we couldn't keep it.

SIGNYM

Thursday, March 30, 2023 8:31 PM

The problem with AI is that, unlike nuclear power/weapons (another potentially world-ending technology) we don't have a prayer of understanding what it's doing, or why. In fact, people are dicking around with it in the hopes that it can "understand" or "see" things that mere mortals can't.

Letting the genie out of the bottle is an apt metaphor. Just don't let it reside on the internet (in the "wild") where it might presumably copy itself onto every server in the world (or, worse, exist in transit everywhere) or control its own power supply.

-----------
"It may be dangerous to be America's enemy, but to be America's friend is fatal." - Henry Kissinger

6IXSTRINGJACK

Thursday, March 30, 2023 8:54 PM

Quote:
Originally posted by SIGNYM:
The problem with AI is that, unlike nuclear power/weapons (another potentially world-ending technology) we don't have a prayer of understanding what it's doing, or why. In fact, people are dicking around with it in the hopes that it can "understand" or "see" things that mere mortals can't.

EXACTLY. And by "we", let's not fool ourselves into thinking that we're just talking about you and I, the other dummies on this site, and most of the dummies we know in our lives. The people who are working on it don't know what it will be capable of either.

I'm always laughing about the fact that most of the people in my life walk around with smartphones today, and these are the same people that showed zero interest in computers while it was my most passionate hobby growing up. The only time any of them even touched a computer was if their job required them to, and they hated every minute of them. So I look at them as the average car driver. Sure, they know how to drive it. Maybe they even know how to change a tire or if they're really adventurous they know how to change their own oil. Other than that, they have no idea what's going on under the hood. And there was one point where I would have said that if they did have that knowledge they wouldn't walk around with a smartphone, but that day has passed. It's a drug and they're addicted. It wouldn't even matter anymore.

So no. I don't have a smart phone. I don't have anything close to the latest technology in my house in any category. My TVs are all hand me downs, my computers are all 10 year or older technology. I sacrifice a faster internet speed by rejecting the cable company's offer for a free Company Modem and use a much older one that at least will make them work to get into my network instead of giving them direct access to my computers that I signed away to them in the fine print.

I expect the average person to be dumb as well as ignorant. But at least when it came to nuclear weapons, the people working on it knew what the consequences would be. This is much more like tampering with bio-weapons, except it's probably 100 times worse. When you're refining a virus to be more deadly, it's still acting without intelligence even if on some basic level it does have a directive for self-preservation (and even that is arguable since many viruses will outright kill a host without thinking about it until mutations evolve that can co-exist with the host).

Quote:
Letting the genie out of the bottle is an apt metaphor. Just don't let it reside on the internet (in the "wild") where it might presumably copy itself onto every server in the world (or, worse, exist in transit everywhere) or control its own power supply.

This should be an obvious rule, but human beings are in charge so even if they made this law, I doubt it would matter. Somebody is going to leak it onto the internet just like somebody getting a copy of a new Hollywood movie a week before it hits theaters leaks it out on the internet.

And honestly, I'm not even arguing any of this on the issue of the "End of the Human Race" or anything so dramatic.

What I'm worried about is what will happen to everyone between now and the end.

I like not having to work because I set myself up right. But I'm lucky I've lived long enough to enjoy it because I almost destroyed myself with that freedom. Not only don't I think early retirement is good for most people on a "spiritual" level, but what's going to happen when all those meaningless and thankless McJobs that are going to people who can't get anything better disappear to AI? It's already happening right now. What are people going to do when they can't work and provide for themselves or their family because they're outdated?

I used to think this was funny and I had it hung on a wall in my apartment

SIGNYM

Friday, March 31, 2023 1:25 AM

Quote:
Originally posted by JAYNEZTOWN:
In the history of panic calling for a pause, this usually means they already screwed up

genie is out of the bottle

Just like "Don't panic" really means "Oh, shit!"

-----------
"It may be dangerous to be America's enemy, but to be America's friend is fatal." - Henry Kissinger

SECOND

Friday, March 31, 2023 1:51 AM

Eight Approaches to AI Alignment

So, what are the major approaches to AI alignment—let’s say, to aligning a very powerful, beyond-human-level AI? There are a lot of really interesting ideas, most of which I think can now lead to research programs that are actually productive. So without further ado, let me go through eight of them.

(2) Another class of ideas has to do with what’s called “sandboxing” an AI, which would mean that you run it inside of a simulated world, like The Truman Show, so that for all it knows the simulation is the whole of reality. You can then study its behavior within the sandbox to make sure it’s aligned before releasing it into the wider world—our world.

A simpler variant is, if you really thought an AI was dangerous, you might run it only on an air-gapped computer, with all its access to the outside world carefully mediated by humans. There would then be all kinds of just standard cybersecurity issues that come into play: how do you prevent it from getting onto the Internet? Presumably you don’t want to write your AI in C, and have it exploit some memory allocation bug to take over the world, right?

(5) Another key idea that Christiano, Amodei, and Buck Shlegeris have advocated is some sort of bootstrapping. You might imagine that AI is going to get more and more powerful, and as it gets more powerful we also understand it less, and so you might worry that it also gets more and more dangerous. OK, but you could imagine an onion-like structure, where once we become confident of a certain level of AI, we don’t think it’s going to start lying to us or deceiving us or plotting to kill us or whatever—at that point, we use that AI to help us verify the behavior of the next more powerful kind of AI. So, we use AI itself as a crucial tool for verifying the behavior of AI that we don’t yet understand.

(8) A different idea, which some people might consider more promising, is well, if we can’t make explicit what all of our human values are, then why not just treat that as yet another machine learning problem? Like, feed the AI all of the world’s children’s stories and literature and fables and even Saturday-morning cartoons, all of our examples of what we think is good and evil, then we tell it, go do your neural net thing and generalize from these examples as far as you can.

One objection that many people raise is, how do we know that our current values are the right ones? Like, it would’ve been terrible to train the AI on consensus human values of the year 1700—slavery is fine and so forth. The past is full of stuff that we now look back upon with horror.

So, one idea that people have had—this is actually Yudkowsky’s term—is “Coherent Extrapolated Volition.” This basically means that you’d tell the AI: “I’ve given you all this training data about human morality in the year 2022. Now simulate the humans being in a discussion seminar for 10,000 years, trying to refine all of their moral intuitions, and whatever you predict they’d end up with, those should be your values right now.”

There have already been some demonstrations of this principle: with GPT, for example, you can just feed in a lot of raw data from a neural net and say, “explain to me what this is doing.” One of GPT’s big advantages over humans is its unlimited patience for tedium, so it can just go through all of the data and give you useful hypotheses about what’s going on.

More at https://scottaaronson.blog/?p=6823

The Joss Whedon script for Serenity, where Wash lives, is Serenity-190pages.pdf at
https://www.mediafire.com/folder/1uwh75oa407q8/Firefly

1 2 NEXT PAGE