Agents need boundaries, not just intelligence

When I first started using coding and operational agents, I wanted the autonomy to become almost perfect. My instinct was: give the system enough freedom to understand what is happening, spot what is going well or badly, and make sensible decisions on my behalf.

My view became more conservative as the agents became more capable.

That sounds backwards. Better models can plan more effectively, use more tools and recover from more mistakes, so it is tempting to assume that the natural destination is simply more autonomy.

In practice, capability and authority are different design problems.

The thing I underestimated was how much useful context still lived in my head. An agent could have access to the repository, the task and the tools and still not know why a particular compromise existed, what I considered acceptable, or which apparently sensible action crossed a boundary I cared about. That missing tacit context is one reason I became more deliberate about human review and explicit stop conditions.

An agent may be perfectly capable of editing a repository, changing a database, publishing content or submitting a form. That does not mean it should have permission to do all of those things whenever its reasoning concludes they are useful.

The principle I use now is simple: capability is not permission.

The dangerous part is not always bad reasoning

When people talk about agent risk, the conversation often starts with hallucination. That matters, but it is only one failure mode.

A competent agent can create problems without hallucinating at all. It can correctly understand the task, choose a reasonable action and still cross a boundary that the organisation never intended to delegate.

Imagine an agent analysing an underperforming page. It concludes that the title, copy and structure should change. The recommendation may be excellent. The agent may also have access to the codebase and deployment tooling.

Should “I have a good recommendation” imply “I am authorised to change production”?

I do not think so.

The same distinction exists in normal organisations. A finance analyst can identify an investment opportunity without having authority to release the funds. A strategy team can recommend a market exit without having permission to terminate contracts. A product manager can define a requirement without unilaterally changing a production database.

Agentic systems need the same separation between insight, recommendation and action.

A plan is not permission

This became an explicit rule in the AI-assisted operating model around FundRobin.

As I introduced more specialised agents and more connected tools, I found that the system needed clear operating contracts. An agent should know its role, the context it can rely on, the tools it can access, the evidence it must produce and the point at which it has to stop.

It is less dramatic than “fully autonomous agents”, but it is much closer to how useful work gets controlled.

A planning agent can investigate and recommend.

An implementation agent can make bounded changes in an authorised workspace.

A reviewer can challenge the implementation without quietly becoming the implementer.

A production action can require an explicit human decision even when every preceding stage is automated.

These are organisational design choices expressed in software.

Approval should describe the action being approved

One of the weaker patterns in human-in-the-loop systems is the generic approval button.

approved = true

That looks like governance, but it may tell the execution system almost nothing.

Approved what? Which change? In which repository? Against which version? With which side effects? Can the agent interpret that approval as permission to make related changes? What happens if execution fails and needs to retry tomorrow?

In the Organic Growth operating model we designed for FundRobin, the stronger pattern was to treat approval as a scoped execution contract. The recommendation could define the intended handler, the owning system and the allowed side effects. Human approval could narrow that authority but not widen it. Workers were designed to fail closed if the approval and execution contract did not match.

That distinction matters because “the human approved something earlier” is not a durable security model.

An approval should be legible at the point of action.

Autonomy should increase with reversibility

I find it useful to think about agent actions on a spectrum rather than make a binary choice between manual and autonomous.

At the lower-risk end:

• read information;
• search and retrieve evidence;
• classify or summarise;
• propose a plan;
• generate a draft;
• run tests in an isolated environment.

Further along:

• edit a reversible artifact;
• update a non-production record;
• open a pull request;
• publish to a controlled draft state;
• trigger an existing deterministic workflow.

At the consequential end:

• deploy to production;
• change permissions;
• mutate critical data;
• spend material money;
• communicate externally as the organisation;
• submit an application or contractual commitment.

The right boundary depends on consequence, reversibility and the quality of the evidence available to the reviewer.

This means the most effective agentic system is not necessarily the one with the highest percentage of autonomous actions. It is the one that allocates autonomy where it creates leverage and keeps accountability explicit where the downside matters.

Deterministic paths are underrated

Not every action an agent can perform should continue to require agent reasoning.

If a task is well understood and the rules are stable, I increasingly prefer a deterministic handler.

The agent can be valuable in discovering the path, interpreting an unfamiliar situation or deciding among ambiguous options. Once the action becomes routine, a predictable workflow is often easier to test, audit and recover.

This is particularly useful in systems with side effects. The more consequential the action, the more I want to know exactly which conditions cause it to run.

Agent reasoning should be spent where judgement creates value, not used as a universal replacement for ordinary software.

Retries are also an authority question

Retries sound like reliability plumbing, but they have a governance dimension.

Suppose a human approves an action and the execution fails. Is an agent authorised to retry indefinitely? Is the approval still valid after the underlying data changes? What if a new version of the recommendation exists? What if the failure occurred because a safety check blocked the action?

In FundRobin’s control-plane design, failed consequential execution required explicit re-authorisation rather than treating the original approval as an unlimited token.

That creates friction. Deliberately.

The goal of an AI operating model is not to remove every pause. It is to remove unnecessary human effort while preserving the pauses that carry real decision rights.

Observability is part of control

Authority also becomes meaningless if nobody can reconstruct what happened.

For agentic work, I want a record of the recommendation, the authorised action, the execution attempt, the result and any subsequent retry or escalation. Append-only events and durable state let a human operator reconstruct whether the system behaved within its mandate.

This becomes more important as agents span multiple systems.

A useful agent may retrieve context from one source, reason in another service, update a repository and trigger a workflow elsewhere. Without a clear record of identity, authority and state, the system can become operationally opaque even when each component works correctly.

The more distributed the execution becomes, the more explicit the control plane needs to be.

Better models do not remove the governance problem

A common response to these controls is that models will improve. They will need fewer corrections, reason more reliably and use tools more safely.

I expect all of that to happen.

It does not remove the authority question.

A world-class employee still has delegated limits. A highly competent finance system still requires controls. A reliable API still has permissions. We do not design governance only for incompetence; we design it because organisations need intentional decision rights.

Agents are no different.

In fact, increased capability makes the question more important. A weak agent with broad permissions is unreliable. A strong agent with broad permissions can be extremely effective at doing something the organisation did not mean to authorise.

Machines are confident in a different way from people

The management analogy also has a limit.

An agent can produce something polished and then behave as though the job is complete. That confidence can be misleading. The artifact may compile, the page may render, the workflow may pass its happy path and the underlying decision can still be wrong.

This is where I find human experience most useful. I am often less interested in asking “does this work?” than “what assumption is this built on?” and “where will this fail?” A machine is very good at completing the task it can see. A human who understands the customer, the history and the consequences is better placed to challenge the task itself.

Review needs to be more than a ceremonial quality check. The reviewer should interrogate foundations, edge cases and hidden assumptions. Otherwise a confident agent and a rushed human can reinforce each other’s blind spots.

The goal is bounded agency, not timid automation

I do not want agentic systems that ask a human to approve every trivial step. That simply turns the human into a slow API.

The ambition should be higher: give agents enough context, capability and room to complete meaningful units of work independently, but make the boundary of that unit explicit.

Within the boundary, let the system move quickly.

At the boundary, make the evidence clear and the decision intentional.

The operating model I trust more is bounded agency: capable systems with explicit authority, observable state, meaningful human gates and a clear answer to the question, “Who owns this decision?”