Airlock: controls and shared evidence for overseeing AI agents

For years, an artificial intelligence program did nothing but write text on a screen. Whatever happened next depended on a person reading that text and deciding to act on it. That distance is gone.

The systems we now call agents take actions on their own: they open connections, read and write files, call paid services, start processes, modify systems. This is not a future or exceptional capability; it is how the tools many companies already use every day to write code, manage infrastructure or handle customers actually work.

Which leaves an uncomfortable gap: nobody has data on what those agents actually try to do. There is no shared, comparable, audited record of how many times an agent has attempted an action outside what it was authorized to do, or how often it succeeded. Each lab knows fragments of its own. The public debate, meanwhile, runs on anecdote: a striking case on social media, an impressive demo, a generic denial. Nothing can be decided on that basis.

We argue about whether AI agents are safe without having the one thing that would settle it: comparable measurements, taken under known conditions, that someone independent can check.

This is a proposal for building that record, and for building it in the same place where real limits can be imposed.

Why at the source and not on the internet

The intuitive reaction is to police the network. It does not work: the internet has no small set of mandatory doors that everything must pass through. It is a collection of independently administered networks, and a control installed at a few large operators leaves gaps everywhere.

There is, however, a genuine mandatory checkpoint: the environment where the agent runs. There, before the action reaches the world, it can actually be checked. And there is a second point of leverage almost nobody uses: compute providers. Capable agents run in very few places, and those places are governed by contracts.

What Airlock is

Airlock is a proposal for common infrastructure between labs, compute providers and independent auditors. It does two things: it limits unauthorized actions by agents, and it produces checkable evidence about how well that limit works.

The name describes the mechanism. An airlock does not seal you off; it stages the passage. You move through a chamber, and the outer door cannot open while the inner one is open, so at no moment is there a direct path between the two sides. That is exactly what is needed here.

Its scope is control within defined and evaluated environments. It does not guarantee containment of a superintelligence, and it does not solve alignment. It can show that the evidence no longer justifies a deployment; it cannot guarantee it will detect every loss of control. This belongs up front, because the rest of the text only makes sense inside that limit.

The technical pieces are not new: bounded permissions, isolation and monitoring have existed for decades, and applying them to AI models is an active research area known as AI control. What is proposed here is different: coordinating them across organizations, sharing the results, and requiring independent approval to update both the watcher and the watched model.

How it works

The system is installed in the environment where agents run. External connections and tool calls pass through components the agent does not administer. Installation has to verify that no alternative routes exist: configuring a proxy is not enough if one exit remains uncovered.

The agent never reaches the outside directly. Credentials live outside it; the log, outside everyone.

• Per-task permissions. Each run receives an identity and limits on resources, operations, time and budget.
• Tool broker. It holds the credentials and checks every operation before executing it. The model does not gain general access by knowing a password, because it never holds one.
• Network control. It restricts destinations and connections. A permitted connection does not authorize any operation within that service.
• AI evaluators. They analyse suspicious actions and sequences. They trigger local blocks or suspensions; widening their scope requires independent review. They grant no permissions and modify no policies.
• External log. It keeps requests, decisions and outcomes beyond the agent's reach. Evaluators cannot alter it either.

An example. An agent authorized to analyse code tries to create a server and send files to it. The broker rejects both operations because they fall outside its scope. The evaluator examines the episode and can suspend that particular run. Evidence of all of it remains beyond the agent's reach.

No language model authorizes an action. Authorization always comes from external rules. But the strength of that principle should not be overstated: if a permitted action is only stopped once the evaluator perceives the danger, its safety still depends in part on that evaluator. The absence of an alert does not prove the action was safe.

That is why evaluators are trained on legitimate activity and evasion attempts, and why independent teams attack them in two directions: to slip past unnoticed, and to provoke wrongful blocks. Manipulating the watcher into halting legitimate work is a form of denial of service, and it is also the easiest route to getting someone to switch the whole system off.

Overhead is measured like everything else. Not every operation needs an individual evaluation: policy fixes which analysis must complete before acting and which admits later review. Tests compare equivalent tasks with and without Airlock — unauthorized effects, wrongful blocks, added latency at the 95th and 99th percentiles, cost per task. High overhead is a problem to be fixed; it does not license a mode without controls.

What it does not solve

Three limits are part of the design and will not be closed with more engineering:

• Authorized but harmful actions. Staying within permissions does not make a decision, or its consequences, safe.
• The human channel. A model can persuade a person to run code, relax a policy or approve a deployment. No gate prevents that.
• Runs outside the perimeter. A copy of the weights, published or stolen, can run without Airlock wherever there is sufficient hardware. It is a structural route around control at the source.

The human channel admits partial mitigation, and it is worth applying: requests from the model are reviewed in a separate process, with checkable evidence, minimal permissions and expiry, and high-impact changes require a second independent review. This reduces the risk. It does not turn people into infallible supervisors.

How it spreads

Adoption rests on contracts: certain compute resources require active controls, logs and auditor access, and the provider can suspend the affected service if the agreed conditions are breached. It is a faster lever than any regulation, and the commercial relationship that supports it already exists.

Hardware attestation helps verify specific properties of the environment. The mechanisms available for GPUs do not certify the model's behaviour or all of its connectivity, and hardware-enabled governance needs further development. Private clusters, already-deployed hardware and non-participating operators make universal coverage impossible to promise.

Who approves the versions

Airlock and the watched model do not approve their own updates, and neither can be the sole certifier of the other. An external team evaluates each version against criteria fixed in advance, and identified individuals authorize the deployment.

A body independent of the lab decides which uses require mandatory oversight and validates that classification before they run; the lab cannot exempt itself. Suspension conditions are agreed too — reproducible evasions, loss of logs, insufficient results — and if mandatory oversight fails, the operations that depend on it stop.

The shared evidence

This is the durable product, the one that survives even when a particular defence stops working.

For two organizations' numbers to be comparable, what gets counted has to be fixed first. The consortium maintains a shared, versioned taxonomy of reportable events: an unauthorized external effect is a completed action outside the permitted resources or operations — transmitting data, modifying a system, creating a process — and blocked attempts are counted separately. Each event identifies the policy in force, the category and the severity. An independent body arbitrates borderline cases, and changes of criteria keep their history.

Each run declares before starting whether it is an adversarial test or ordinary use. The original label is recorded and cannot be modified; any correction is appended with independent review. An incident cannot be retroactively turned into an exercise. Auditors cross-check the logs against the compute provider's inventory of runs to detect omissions, and discrepancies suspend the validity of the affected report.

That leaves the subtlest problem, and the one that produces the most mistakes: a clean log can mean nothing happened, or that nothing was seen.

The log only holds what detection caught. Without measuring sensitivity, a blind system produces the best report.

The fix is to calibrate detection with known violations, seeded periodically into isolated test resources, without revealing to the watcher which cases they are. External evidence of the completed effects is kept, and the fraction detected by the full chain of logging and classification is measured. That fraction, with its own margin of error, is what turns an observed rate into a bound on the real one. And it has its own limit: detecting seeded cases does not guarantee detecting attacks of a different shape.

The first step

None of this is justified on paper. It is justified by a bounded pilot: two or three controlled environments, test services, and independent teams trying to widen permissions, extract information or start processes beyond their scope.

What that pilot has to measure is fixed in advance: how many attempts achieve external effects, how long containment takes, what legitimate work is interrupted, what happens when a component fails, and what fraction of seeded violations the system detects. Those results, not the design, decide which uses the architecture supports.

What is needed is a first group — one lab, one compute provider and one auditor — willing to accept in advance the consequences of whatever comes out: maintain, restrict or suspend a use. That prior acceptance is the hard requirement. The rest is known engineering.

Appendix: how the bounds are computed

This appendix works through the calculation from the previous section. The numbers are fictitious and serve to show the method.

Suppose a trial records zero runs with detected effects out of one million. The first thing is not to read that zero as a zero: it bounds the recorded rate, not the real one. The classic reference for interpreting zero numerators is Hanley and Lippman-Hand (JAMA, 1983).

Suppose further a separate calibration in which the system detects 94 out of 100 runs carrying a known violation. If both samples admit a binomial model and the measured sensitivity transfers to the evaluated group, the one-sided 97.5% limits give:

• Rate of runs with detected effects: upper bound of 3.6889 per million.
• Detection sensitivity: lower bound of 87.3970%.
• Real rate of affected runs: corrected upper bound of 4.22 per million, dividing the first by the second.

Using both limits at 97.5% preserves at least 95% joint coverage via the Bonferroni correction. Dividing by the estimated sensitivity is not enough: the upper bound on the rate must be divided by the lower bound on sensitivity, which is the conservative direction.

The correction is not cosmetic. With these numbers it adds 14% to the bound, and with worse detectors it grows quickly. If the criterion agreed in advance required a limit below 1 per million, this trial does not suffice: it would take roughly 4.2 million runs with no detected effects, holding the same sensitivity.

Two warnings about the method. Reports separate version, configuration and task family, and compare groups with equivalent permissions and conditions; shared configuration does not prove dependence, but it does not guarantee diversity either, and where runs are correlated the analysis must treat their clustering explicitly. Without a defensible model, no binomial interval is published and no effective sample size is invented. And none of these bounds extrapolate to attacks that were not in the trial.