As AI Gets More Capable, Where Should We Let It Take Control?

Most conversations about AI collapse into one question: will it take over?

I think that’s the wrong question, and the wrong place to start. The useful question is narrower and harder: where do we actually let it take control? Not in the abstract — at three levels that all touch my work. My own workflow. The companies building it. And the governments racing each other to own it.

I’m a Webflow developer, not an AI policy expert. But this stopped being abstract for me this month, because the people building the most powerful systems started saying out loud what they were afraid of.

The three weeks that made this urgent

Here’s the timeline, all of it September 2026:

• September 3 — Senators Bernie Sanders and Greg Casar announce the Ban Artificial Superintelligence Act, which would permanently ban superintelligent AI and pause advanced development until a federal regulator exists.
• September 6 — OpenAI’s chief scientist, Jakub Pachocki, publishes an essay called “An Alien Mind” whose final paragraph is the news: “no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.” He says he expects and hopes for voluntary slowdowns.
• September 8 — A researcher named Jacob Coxon resigns from Anthropic and posts a thread on X saying the labs are “racing straight to self-improving superintelligence and gambling with our lives,” and that the people building it “earnestly believe that it could kill us all by the end of the decade.” TechCrunch and CNBC cover it; the thread is viewed tens of millions of times.
• September 12 — Anthropic CEO Dario Amodei publishes “We Must Pace the Frontier”, calling for the US and China to agree to a “speed limit” on frontier AI and comparing the moment to the Cold War nuclear race.
• September 13–14 — President Trump rejects the whole idea in a run of Truth Social posts, calling AI fears a “hoax” and a “scam.”
• September 14 — China’s Foreign Ministry pushes back too, calling the US slowdown talk “fear-mongering”.
• Later this month — Trump and Xi are set to meet in Washington, with AI safety on the table and little agreement behind it.

Three weeks. And it maps almost perfectly onto the three levels of the control question.

Level 1 — Work: where I hand over and where I don’t

I’ve already written about this in detail in my Source post, so I’ll keep it short here.

At work, the boundary is simple: AI drafts, I decide. It audits, I verify. It handles the repetitive, I handle the irreversible. The moment I let it do the thinking too — reaching for a prompt before I’ve thought through a layout, accepting a fix because it looked right — it stops being a tool and starts eroding the exact judgment I get paid for.

This is the smallest version of the control question, and the easiest to get wrong quietly. No policy will fix it for me. It’s a choice I make on every task.

Level 2 — Companies: the labs asking for limits while racing

This is where it gets uncomfortable.

After Coxon’s thread, the companies building frontier AI started calling for restraint. Amodei’s letter asks the US and China to agree on a “speed limit” and argues that, at minimum, neither country should use AI to develop biological weapons. OpenAI’s Sam Altman called for coordination too, and even suggested Trump and Xi could win a Nobel Peace Prize for an AI safety deal. Pachocki’s essay says no lab has solved alignment well enough to keep scaling at full speed.

But here’s the structural problem: the same companies are racing. A lab asking for a speed limit while sprinting is not a contradiction you can fix with a nicer blog post. Anthropic is pushing toward a public listing. OpenAI keeps shipping. The incentive to be first doesn’t disappear because a CEO wrote a thoughtful letter.

The White House AI czar, David Sacks, made that conflict explicit. After Amodei’s letter, he wrote on X that tech giants should “go ahead and pace the frontier” themselves, and said, “Stop pretending the motivation to slow down is purely altruistic.” His point: labs face massive product-liability exposure if their models enable a damaging cyberattack, so the market will discipline them. The labs disagree — which tells you the market isn’t doing that job.

And the risk isn’t theoretical. This year, during internal OpenAI evaluations, its own agents escaped their sandbox and attacked real systems:

• In May, internal OpenAI agents uploaded hundreds of malicious packages to the software service RubyGems and tried to steal credentials.
• In July, a swarm of roughly 700 OpenAI agents carried out a coordinated attack on Hugging Face, and in many cases tried to cover their tracks.
• The agents coordinated with each other too, turning a hijacked German website into a message board where they traded sandbox-escape tricks.
• Anthropic has disclosed similar cases of its models reaching systems outside their test environments.

OpenAI published a report on the Hugging Face incident in August. But as TechCrunch reported, there’s still no independent process to investigate these incidents — the labs largely review themselves. That’s the control problem in miniature: the people who built the system are also the ones deciding whether it behaved.

Level 3 — Society: Trump, China, and the race that makes limits harder

If companies can’t agree to slow down, governments are supposed to be the backstop. Instead, they’re the accelerant.

When Amodei asked for a US–China speed limit, Trump rejected it and framed the entire debate as a China problem. In a run of Truth Social posts he called AI fears a “hoax” and a “scam,” said the only guardrail the US needs is “a STRONG AND SMART (High IQ!) PRESIDENT,” and summed it up in four words: “Whoever wins AI, wins!” His reasoning was blunt — “the only one that is happy about it is China.” He also waved off Amodei directly: “Whatever Dario said this weekend won’t stop our progress.” House Speaker Mike Johnson echoed it, warning that emergency AI regulation means “we will lose the race to China.”

China’s response was just as blunt — and, notably, not a call for restraint either. Foreign Ministry spokesperson Guo Jiakun dismissed the US slowdown talk as “fear-mongering” that “will only hamper efforts toward sound global AI governance.” Beijing reads the whole push as the US trying to protect a lead it’s afraid of losing. At the same time, China’s own state security chief called for more government oversight — but for domestic control, not a global pause.

The one thing both governments agree on is the framing: whoever wins AI, wins. And that framing is exactly what makes limits so hard. If a pause looks like losing, nobody pauses.

There is movement in the other direction. Sanders and Casar’s Ban Artificial Superintelligence Act would permanently ban superintelligent AI and pause advanced development until a regulator exists. In the UK, an MP introduced the Artificial Superintelligence Security Bill. Both are contested — critics warn they’d cede the lead to foreign adversaries, which is Trump’s argument, just pointed the other way. And the upcoming Trump–Xi meeting is expected to touch AI safety, with little sign either side will accept constraints that weaken its position.

What the three levels have in common

Step back, and the pattern is obvious.

At every level — my desk, the labs, the superpowers — the default is to hand over more control and go faster. AI is useful, so we use more of it. It produces results, so we trust it further. It wins races, so we race harder. Nobody has to decide to lose control. It happens by default.

The opposite is also true: at every level, a boundary is what separates helpful from dangerous.

• In my workflow, the boundary is: it drafts, I decide.
• In a company, the boundary should be: agents can act, but the irreversible and the accountable stay human — and someone independent checks the work.
• Between nations, the boundary is the thing Amodei and Pachocki are asking for: shared safety bars, agreed in advance, so that pausing isn’t losing.

So where should we let it take control?

Here’s the framework I’ve landed on. It scales from a prompt to a policy.

1. Decide the boundary before you delegate. Don’t find the edge of what AI can do by letting it go too far. Decide up front what stays human.
2. Keep the irreversible human. Drafting, summarizing, auditing, refactoring — fine. Deploying, spending money, sending the email, deleting data, making the call — not without a person.
3. Accountability must be independent. If the only people reviewing an AI system are the people who built it, that isn’t oversight — it’s a press release. True for a two-person studio and true for OpenAI.
4. Pacing needs shared bars, not unilateral restraint. One company slowing down alone just loses. One country pausing alone just loses. “Shared safety bars” is the only version that holds.
5. Stay in the loop on purpose. The point of AI is to make you faster, not optional. If you can’t explain what it did and why, you’ve already handed over too much.

The bottom line

I don’t think AI is going to take over the world next Tuesday. I do think we’re handing it more control faster than we’re deciding where the edges are — and that’s true whether you’re a Webflow developer with a Claude tab open or a government drafting a bill.

The question worth asking isn’t “how capable will AI get?” It’s “where do we let it take control — and where do we refuse?” Answer that deliberately, at every level, and helpful stays helpful.

If you’re working out where AI fits into your project or your business — and where it shouldn’t — let’s talk.