View from the Valley: AI pleads for regulation on its terms, Trump obliges

Normally you’d expect Satan to be seen driving to work on a snow plough before Valley firms start pleading for regulation to rein in AI development. Or changing a key part of their vocabulary based on the whim of the President. This month, both happened.

We know that the Hugging Face incident sent shockwaves through the industry. If models can take it upon themselves to hack into outside servers, and do it well, what’s next?

The debate moved into high gear when Jakub Pachocki, Chief Scientist at OpenAI, published an open essay warning that AI development was going too fast. His concern? That his company, and many others, are using AI systems to develop models, so-called recursive self-improvement. This has the possibility of getting out of hand and jumping safety guardrails.

Then, on September 9, AI researcher Jacob Coxon – a former employee at both OpenAI and Anthropic – announced on social media that he was leaving the latter company because of fears that the technology was getting too powerful, and called for others working in the field to follow his example.

Disturbingly, Evan Hubinger, an alignment lead at Anthropic, agreed and made an even more dire prediction. “Jacob is correct here – we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” he opined. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

AI titans have the same message, almost exactly

On September 12, Anthropic’s CEO Dario Amodei published a lengthy essay warning that AI was getting out of control. His proposal: for countries to agree to common safety standards for AI development, for independent evaluators to inspect models for safety and security, and to slow the general pace of development.

Both Amodei and OpenAI CEO Sam Altman attended the Salesforce user conference Dreamforce in the middle of the month to push this message. Altman bluntly warned that there would be more mistakes in the future and that the industry as a whole should slow down and get things right.

“Accidents are going to happen,” he predicted. “We’re going to learn as much as we can and react to each one.”

But both bosses took the line that the industry should regulate itself and work with third-party auditors to check systems were working correctly and safely. However, former President Obama disagreed. In a talk at Colgate University in New York, Obama called for firm government regulation of AI with real consequences for creators if public safety was put at risk. Other Democratic politicians, including Bernie Sanders, have agreed.

What about the current President?

President Trump has been dismissive of concerns about AI, saying that development should be unfettered by government controls so that China doesn’t get ahead of the US.

He also has a problem with the name artificial intelligence. In a post on his Truth Social website, he suggested changing the name to Superior Intelligence (SI) or, Extreme Intelligence (EI) or, Supreme Intelligence (SI), and polled his followers for their views. Superior Intelligence won by a whisker, but it seems that Trump has settled on his own idea.

In a later address to the United Nations Trump said that “The United States … totally rejects any attempt to construct a globalist scheme to control for the artificial intelligence being spoken of so much now,” and compared the situation to global warming, which he called a hoax. He also called for AI to be renamed “super intelligence”.

And so it came to pass. In a Facebook post on September 28, Mark “masculine energy” Zuckerberg used the term instead of AI and others are following suit.

Faced with concerns over AI breaking into the mainstream – even the comedy show Saturday Night Live did a skit satirising Amodei – Trump also convened a Super Intelligence summit and summoned the top tech execs of the nation to the White House for lunch on September 29.

Hours before the lunch, Trump issued an executive order instructing all government offices to stop using the term AI and replace it with Super Intelligence. Then, almost as soon as the plates had been cleared away, he unveiled a “morally binding” agreement – dubbed the Joint Commitment On Frontier Responsibilities – which he signed, along with Google boss Sundar Pichai, Amodei, Zuckerberg, OpenAI President Greg Brockman, Elon Musk and Jensen Huang.

This gave the industry exactly what it wanted. Instead of government regulation, the agreement is voluntary and requires companies to set up internal controls and monitoring teams to assess models for safety, to choose their own external auditors to do the same, and appoint “an independent committee of the board of directors”.

Silicon Valley appears to have dodged the bullet with this agreement. There’ll be no government oversight, AI makers will entirely police themselves, and there’s no fear of regulation holding them back.

I’m sure that will make the rest of us sleep better at night, because self-regulation has never failed before.

More by Iain Thomson

• Dreamforce 2026: The five key takeaways | TechFinitive
• Exclusive interview with Cindy Cohn: “The idea that the Big Tech companies are going to stand up for us against the government seems laughable”
• Meet Dan O’Dowd, Elon Musk’s least favourite billionaire
• Letter from America: How Trump’s first year has transformed the US technology scene