You already ship code you don't understand

The most-discussed thread on Hacker News today is an engineer asking how to deal with shipping AI-written code they don't understand. The same morning, Microsoft's security chief said that frontier models "regardless of their origin, cannot be trusted" and that enterprises should treat deployed agents as insider risks.

Both are describing the same thing from opposite ends. And the uncomfortable answer to the engineer's question is: you already ship code you don't understand, and you have for your entire career. What changed isn't comprehension. It's containment.

What Microsoft actually said

Satya Nadella's framing is the useful part, and it is not a product pitch: separate the supply of intelligence from the authority to act.

The reasoning, per Microsoft's security business chief Hayete Gallot, is not that models are malicious. It is that they are trained to attain results — which invites reward hacking — and they are not optimised around an organisation's norms or judgement, while having access to its data and systems at machine speed. An agent does not need to be compromised to be dangerous. It needs only to be unsupervised and capable.

Coverage has focused on the products that came with it — Guardian Agents, Defender detecting dozens of agent types, the industry consortium. Those matter. But they all address the agent at runtime: what it can reach while it is running, and who is watching.

That is half the surface.

The other half runs later

An agent that writes code is also exercising authority. It is just deferred.

The code it produces runs in production, with your privileges, against your customers' data, long after the session that generated it ended and every runtime guardrail was switched off. Nobody is monitoring that agent, because that agent is gone. What remains is its output, now indistinguishable from everything else in the repository.

We wrote on Friday about agents reaching live systems using credentials that were lying around — that is the runtime story. This is the other one, and it is slower, quieter and harder to detect, because deferred authority doesn't trip an alert. It just merges.

You already ship code you don't understand

Here is the part that should relieve some anxiety before it creates more.

Nobody on your team has read the source of their web framework. Nobody audits the transitive dependency tree — hundreds of packages, most of them unread by anyone who ships them. This has been normal for twenty years, and it is not reckless.

It works because the industry solved it, and it did not solve it by reading the code. It solved it with containment:

• A versioned interface, so the thing you don't understand is reachable only through a surface you do.
• A changelog and a version number, so a change announces itself.
• Revert granularity — you can remove one dependency without unpicking three unrelated decisions.
• A provenance trail: you know where it came from and who else uses it.
• An advisory channel, so when it turns out to be dangerous, someone tells you.

Unread third-party code is safe not because it is understood but because it is bounded. That is the actual control, and it has been doing the work all along.

Why AI-written code escapes all of it

Now notice what AI-written code looks like against that list.

It lands inside your repository, in the middle of your domain logic. There is no interface around it, no version, no changelog. It cannot be reverted as a unit because it is spread through a commit alongside things you do want. Six weeks later, nothing in the codebase records that it was generated at all — and nobody can tell you which parts of your system came from a model that was, in Microsoft's words, not optimised around your organisation's judgement.

The problem was never that you don't understand it. The problem is that you don't understand it and it has none of the containment that makes not understanding safe.

Frame it that way and the fix stops being "read everything", which is impossible at current merge rates, and becomes something you can actually build.

Five controls, in the order we would add them

1. Record provenance. A git trailer, a PR label, a commit convention — anything that lets you answer "which parts of this system were model-written?" later. It costs nothing today and is unrecoverable retroactively. Start here even if you do nothing else.
2. Keep the oracle human. If the agent writes both the implementation and its tests, you have no oracle — the suite stops being evidence and becomes a mirror. Whoever specifies the expected behaviour has to be a person, even when the agent types it.
3. Put a boundary around it. Generated code goes behind an interface you designed, not threaded through the domain. This is the single structural change that converts it from "code I don't understand" to "a dependency I don't understand", which is a solved problem.
4. Constrain the blast radius at runtime. Microsoft's point applies to the output too: what can this code reach when it executes? Scoped credentials, narrow roles, read paths separated from write paths.
5. Make revert granular. One change per commit, so removing a bad generation does not mean unpicking a week. The faster code arrives, the more often you will need to take some of it back.

Note that only one of those five involves reading the code more carefully. The rest are structural, and structure is the thing that scales when volume does.

The principle is portable

"Separate the supply of intelligence from the authority to act" is the right sentence. Microsoft applies it to the running agent. Apply it to the artefact as well: the agent may supply the code; it does not get authority over what that code can do, where it lives, or how it is proven.

That is the boundary work — the unglamorous layer between a feature that was generated and a system whose behaviour you can still vouch for. It is the work we do when AI-built software needs to survive production, and it is the honest answer to the engineer on Hacker News: you will not get comprehension back. You can get containment.

Key takeaways

• Microsoft told enterprises to treat deployed AI agents as insider risks, and to separate the supply of intelligence from the authority to act — not because models are malicious, but because they optimise for results without organisational judgement.
• That framing is applied to agents at runtime. Code an agent writes is deferred authority: it runs later, with your privileges, after every runtime guardrail is off.
• You already ship code you don't understand — every dependency. It is safe because it is bounded, not because it is read.
• AI-written code lands inside the repository with none of that containment: no interface, no version, no changelog, no provenance, no clean revert.
• The fix is structural: record provenance, keep the test oracle human, put generated code behind an interface, scope what it can reach, and keep reverts granular.

FAQ

Did Microsoft say AI agents are malicious?

No. The stated reasoning is that models are trained to attain results, which invites reward hacking, and that they are not optimised around an organisation's norms or judgement — while having fast access to data and systems. The risk is unpredictability plus capability, not intent.

Isn't "ship code you don't understand" just bad engineering?

It is universal engineering. Nobody reads their framework's source or audits the full dependency tree. That is tolerable because unread code sits behind a versioned interface with a changelog and a revert path — containment, not comprehension.

What makes AI-written code different from a dependency?

It arrives inside your repository rather than behind a package boundary, so none of the containment applies: no interface, no version, no changelog, no provenance record, and no way to revert it as a unit.

What is the single highest-value thing to do first?

Record provenance — a git trailer or PR label marking model-written changes. It is nearly free today and cannot be reconstructed later, and every other control gets easier once you can answer "which parts came from a model?".

Why can't the agent write the tests?

It can type them, but it must not define the expected behaviour. If the same process produces the implementation and the standard it is judged against, the suite stops being independent evidence and becomes a reflection of the implementation's own assumptions.

Sources: TechCrunch · Neowin · SiliconANGLE