Slowing AI Is Not the Same as Governing It

An open letter to the people deciding what comes next, and to the rest of us who will live with it

When I read that King Charles was bringing leaders from the major AI companies together, I was glad the meeting was happening.

I felt something similar when Dario Amodei called for the development of the most advanced AI systems to be deliberately paced. If the people closest to these systems believe that safety is falling behind, we should take that seriously.

Amodei has proposed outside evaluators inside AI companies, shared safety standards, limits tied to dangerous capabilities, and some degree of cooperation between countries. These are serious ideas. They recognize a problem that is becoming hard to ignore.

AI systems are getting better at acting through software, networks, and organizations. At the same time, the people responsible for them are still trying to understand what they can do and how they may fail.

I think slowing down may be necessary.

I also think we may be asking the idea of slowing down to carry more weight than it can.

Suppose the industry succeeds. Suppose we gain another year, perhaps two. What exactly will we have gained?

That is the question I would ask the people gathering now to consider.

My deeper concern is not only how quickly AI develops. It is whether people and institutions will keep a real ability to understand what is happening, step in when needed, recover when things go wrong, and choose a different path when conditions change.

We often imagine losing control as something a machine might do to us. But control can also disappear through choices we make ourselves.

We can keep the formal right to stop a system while slowly losing the practical ability to do it.

What would slowing down be for?

More time could make a real difference.

Researchers could study strange or dangerous behavior, improve security, and find better ways to understand systems that remain hard to explain, even to their creators. Governments could build rules and public bodies with a chance of keeping up. The public might have room to examine important choices before those choices become part of daily life.

Time can create an opportunity. It cannot decide what we do with it.

An extra year could strengthen outside oversight. It could also allow a small number of companies to gain even more control over the field.

Organizations might use the time to protect human judgment. Or they might rebuild themselves around AI so deeply that working without it becomes nearly impossible.

The question is not simply whether development slows. It is what changes during the time gained.

Some of that time should be used for testing. Researchers need to know whether systems can deceive people, carry out cyberattacks, produce dangerous scientific information, or escape the limits placed on them. Outside evaluators should have enough access to examine those questions without relying only on what a company chooses to show them.

Yet testing has limits.

A model is tested under certain conditions, with certain tools, instructions, and limits. Real-world use changes those conditions. The system enters an organization with its own goals, pressures, habits, and weak points. It receives new information. People use it in ways its creators did not expect.

An error that seems manageable in a lab can be repeated across thousands of real decisions.

I do not say this because testing is pointless. I say it because a test should not be asked to prove more than it can.

The same caution applies to human oversight.

It is easy to promise that a person will remain involved. It is harder to make sure that person has enough time, information, skill, and authority to use their judgment. They may also need protection from pressure to approve whatever the AI recommends.

The real question is whether someone can say no in a way that changes what happens.

If the time gained through slowing down is used well, it should help make that ability real.

Outside evaluators should be able to publish findings that companies would rather keep quiet. Workers and professionals should be able to question AI recommendations. Someone should have clear authority to stop a system that begins to cause harm.

We should also prepare for failures that no test will predict.

Records should make it possible to understand what happened. People affected by important decisions should have somewhere to appeal. Hospitals, public agencies, and other essential services should keep enough human knowledge and skill to continue when an AI system is removed.

I worry that this ability will disappear quietly.

Staff will be reduced. Skills will fade. Workflows will change. Other systems will begin to rely on AI-generated results. Eventually, switching the system off may still be technically possible while becoming unthinkable in practice.

By then, the loss of control will not look like a machine refusing a command.

It will look like an organization discovering that it no longer knows how to work without one.

This is a distinction we need to hold onto:

Having control on paper is not the same as having a real choice.

An off switch matters only if using it remains a realistic choice. Human oversight matters only if the human can change the result. An appeal matters only if someone has the power and skill to reconsider the decision. Public input matters only while important choices are still open.

Slowing down could help us protect those abilities. But only if we recognize dependence as part of the safety problem.

Governing the world we are already in

There is another limit to what slowing down can achieve.

The largest AI companies have great influence over the most advanced systems. Training those systems still requires huge amounts of computing power, money, energy, and technical skill. This gives governments and the public some clear places to apply rules and oversight.

What these companies do matters. Stronger duties at that level are justified.

But the largest labs do not contain the whole technology.

Knowledge is spreading. Models can be copied, changed, and connected to new tools. Research moves between organizations and across borders. Smaller groups and individuals can do important or dangerous things with systems they did not train themselves.

This is where comparisons between AI and nuclear technology stop being very helpful.

Nuclear weapons can be constrained in large part through safeguards focused on nuclear material and related facilities. AI has physical limits too, especially advanced chips and large data centers. But software and knowledge do not behave like nuclear fuel. Once they have spread, they cannot simply be gathered up again.

Pandora’s box is already open.

I do not mean that as a reason to give up. I also do not mean that regulation is pointless.

The most powerful actors can and should face demanding standards. Some capabilities and uses may need firm limits.

But governance must begin with the world we are actually in.

No government will see every use of AI. No agreement among a few labs will bind every group that can build on their work. No international body will prevent every harmful use.

I do not know what a complete system for governing something this widespread would look like. I am increasingly sure it cannot be built around the hope of complete control.

That changes where we need to look.

A system’s intelligence is only one part of its power. Its power also depends on what it can reach, what it is allowed to change, how many people it can affect, and how deeply people or organizations depend on it.

A system becomes more powerful when it can reach private or important information. It becomes more powerful when its decisions affect many lives. It becomes more powerful when people cannot easily question its results. It becomes more powerful when removing it would cause major disruption.

This gives us a different set of questions:

• What can the system reach?
• What is it allowed to decide or change?
• How many people can it affect?
• Who relies on it?
• Who can overrule it?
• What happens if it disappears?

A model does not need to be the smartest in the world to affect someone’s job, education, health, freedom, or access to an essential service. It only needs access to the right information and power inside the organization making the decision.

Many of those choices are already being made far from the leading AI labs.

A company uses AI to evaluate workers. A hospital uses it to shape treatment decisions. A government office adds it to a process that decides whether someone receives a service.

The model may seem ordinary compared with the newest systems. Its place inside the organization gives it power.

Slowing the frontier will not govern those choices for us.

We need rules for the companies building the most powerful systems. We also need clear responsibility for the organizations putting AI into people’s lives.

We need ways to spot harm, respond to it, and keep services running when prevention fails.

This is less satisfying than the idea of complete control. It is also closer to the problem we have.

What is safety meant to protect?

The difficulty of controlling AI may lead governments and companies to watch its use more closely.

Some monitoring may be necessary. Certain threats cannot be addressed without investigation, records, and enforcement. Some powerful capabilities should not be available without safeguards.

Still, I worry about how quickly privacy becomes negotiable when safety and security are invoked.

A government trying to find dangerous AI activity may want to know who is using advanced systems, what they are doing, which computing resources they have, and what information their systems are processing.

A company may argue that it needs detailed records of user activity to detect misuse.

Each step can be defended on its own. Together, they can create a system able to watch far more than dangerous behavior.

Identity checks may reduce some forms of abuse while removing legitimate anonymity. Detailed logs may help explain an incident while creating large stores of private information. That information can be stolen, misused, or handed to a future government with different goals.

Weakening encryption may help one investigation while making private communication less secure for everyone.

A system built to find misuse can become a system for watching people.

We cannot protect people from AI by making them completely visible to the governments and companies governing it. Privacy is not separate from safety. It is one of the things safety should protect.

I do not know where every boundary should be drawn.

Different risks may require different levels of oversight. Different societies will not make all of those choices in the same way.

But governments and companies should have to show that monitoring is necessary and no broader than the risk requires. Information should not be collected simply because it might be useful later. Powers created for an emergency should not quietly become a permanent part of everyday life.

The machinery built to govern AI will create power of its own.

That machinery does not sit outside the AI risk. It changes the risk.

Identity systems, monitoring networks, required logs, access controls, and emergency powers may reduce some dangers. They can also create new stores of information and new centers of power. Those systems may remain long after the threat used to justify them has changed.

They may pass into the hands of future governments, executives, or agencies that were not part of the original plan.

Governing AI therefore creates a second problem: how do we stop the control system itself from becoming something people can no longer question or change?

This connects to a wider concern.

The decisions being discussed at this meeting will travel far beyond the room. AI will enter workplaces, schools, hospitals, public offices, financial systems, and homes.

Many people affected by it will not understand how the systems work. Some will not know AI was involved. Others will know and have no practical way to refuse it.

Public consultation cannot mean explaining decisions after they have already been made.

I am not suggesting that every technical question should be settled by public vote. Expertise matters. Some information must remain private because releasing it would create real danger. Governments have duties that cannot be handed over to opinion polls.

But complexity cannot become a reason to remove important choices from public life.

Workers should have a real voice in how AI changes their work. Communities should be able to question its use in schools, hospitals, policing, and public services. People affected by automated decisions should be able to reach someone with the power to reconsider the result.

Countries outside the main centers of AI development should not be treated only as markets, sources of data and labor, or places expected to follow rules written elsewhere.

The effects of AI will cross borders. Its benefits, risks, and governing power will not be shared evenly.

This future concerns all of us. Responsibility for it does not fall equally on all of us.

The people building these systems, funding them, giving them access, and choosing where they will be used have more power to shape the outcome. They should carry more responsibility for it.

That is why I hope this gathering leads to more than a set of principles.

I hope it produces a clear account of what time gained through slowing down will be used to achieve. I hope outside evaluation means more than companies choosing their own examiners and controlling what the public can learn. I hope the conversation reaches beyond the most advanced models to the organizations already giving AI power over parts of human life.

And I hope safety does not become a word that ends the argument about privacy.

No meeting can settle humanity’s relationship with advanced intelligence. No company, government, or generation should be trusted to settle it forever.

The technology will change. Our institutions will change with it. Some safeguards will fail. New risks will appear, including risks created by the systems meant to protect us.

We need rules and institutions that can change when the evidence changes.

This may be one of the defining tasks of our age. Not because AI is the only challenge we face, but because it may shape how we understand and respond to almost every other challenge.

It may change who can act, how far their choices travel, and how easily those choices can be questioned.

The ways we have governed technology before will still matter. We will need laws, testing, standards, treaties, professional judgment, and public institutions.

But familiar tools will not be enough if we never question the ideas behind them.

We will have to look more closely at where power is growing, when oversight has become little more than a performance, and which human skills are disappearing because of dependence.

We will have to govern the systems of control as seriously as the systems they are meant to control.

We will also need humility.

No one knows exactly what is coming, including the people closest to the technology.

Uncertainty is not a reason to do nothing. It is a reason to protect our ability to respond.

Pandora’s box is open, but the future is not settled.

We cannot return to a world in which this knowledge does not exist. We can still decide what these systems are allowed to reach, what power they receive, how deeply we depend on them, and which parts of human life should remain beyond their grasp.

So slow down where slowing down gives us a better chance to understand, think, and prepare.

Then use that time to make sure we can still see what is happening, question it, stop it when needed, repair what goes wrong, and choose another path.

The box cannot be closed. Our responsibility is to keep the future open.

---