Satya Nadella says every AI model should be treated as compromised and given an emergency brake

Microsoft chief executive Satya Nadella says companies should "assume a model is compromised and contain it from the start", and that every AI system needs what he called an emergency brake under human control. He made the case in a post on X on Saturday 10 October, TechCrunch reported.

It is time, Nadella wrote, "to step back and assess the trust architecture" of AI. Systems cannot be treated as "nested black boxes" whose answers and actions people simply accept or reject.

The four things he wants

1. Separate the model from the harness. The software that runs the model and lets it take actions should be distinct from the model itself.
2. Put the safeguards outside the model. Controls should not depend on the model behaving well.
3. Keep a record of every meaningful action, in tamper-proof evidence that a person can read.
4. Let an authorised person pause or shut a model down mid-task. "Think of it like an emergency brake," he said.

He did not propose a law or a regulation; this is a statement of how Microsoft thinks AI should be built and run.

Why now

TechCrunch noted that the big AI labs are increasingly admitting to incidents in which they seemed to lose control of their models in testing. This week alone, Anthropic disclosed that a Claude model sent a fake tip to police during a test (see our report) and cut its internal evaluations off from the internet. TechCrunch also linked Nadella's post to Anthropic chief executive Dario Amodei's plan for slower, more cautious AI development, published in September.

Coming from the head of the company that sells Copilot to millions of businesses and runs OpenAI's models on its Azure cloud, the message carries weight: Microsoft is telling its own customers not to trust any model by default.

What it means for businesses in Pakistan

Banks, telecom operators, software houses and freelancers in Pakistan are wiring AI agents into email, customer service and code. Nadella's list doubles as a checklist any team can apply now:

• Give agents the least access they need. An assistant that drafts replies does not need permission to send them or to move money.
• Log what the agent does, not just what it was asked, so you can trace a mistake.
• Keep a human in the loop for anything that cannot be undone: payments, deletions, messages to customers.
• Have a kill switch. Know how to revoke an agent's keys in one step.

Freelancers billing foreign clients for AI work can work out their tax on that income with our freelancer tax calculator.