The Hidden Danger Isn’t a Rogue AI
When technologies converge, dependency can grow faster than our ability to understand, afford and govern it.
Imagine this situation. Nothing goes rogue. The agent follows its instructions. The software passes its assigned tests. The supplier delivers the contracted service.
Yet the organisation cannot explain a decision, reproduce the result or afford to change providers. Everyone has delivered their part. Nobody can demonstrate control of the whole.
That is a hypothetical scenario, but a useful test of what we mean by control. It requires no malicious machine. It requires dependencies to grow faster than our understanding of them.
Convergence also concentrates dependency
Models, agents, generated software, cloud services and business processes can now be assembled into a single workflow. That creates useful possibilities. It also creates more places where responsibility can fall between the pieces.
The model supplier owns one layer. A software vendor owns another. The infrastructure provider manages capacity. The customer remains responsible for the service received by an employee or a member of the public.
Each participant may have a perfectly reasonable account of its own performance. The person affected still needs an answer about the complete outcome.
A familiar parallel is a supply chain. Every delivery can meet its local target while the overall operation becomes dependent on one irreplaceable supplier. Visibility of individual transactions does not automatically reveal the vulnerability of the system.
Who runs out of room first?
It is tempting to ask which AI company will run out of cash first. A credible answer needs available cash, cash consumption, financing terms and future commitments. A valuation or a revenue run rate cannot supply that answer.
Anthropic’s May 2026 funding announcement, for example, reported a $65 billion raise and annualised revenue exceeding $47 billion. Those are substantial numbers. They are not, by themselves, a statement of free cash flow or proof of sustainable profitability.
For a customer, the more useful question is conditional: what happens to our operation if a supplier must raise prices, withdraw a service or reduce its support?
Costs need not rise everywhere. Better hardware, more efficient models and competition may reduce some costs. But an organisation can still become more expensive to run if it increases usage, adds review work or loses the practical ability to switch.
The dependency deserves a budget as well as the subscription. Include migration, testing a replacement, retaining evidence and keeping enough human capability to recover.
The code can become part of the trap
Consider an agent that generates a service, another that writes its tests and a third that reviews the result. The checks pass. The release is approved.
Now ask someone who did not participate in the original conversation to change a business rule. Can they identify what code is actually deployed, explain its assumptions and make the change safely?
If that takes days of reconstruction, some of the apparent productivity gain was borrowed from future maintenance. If implementation and tests share the same mistaken assumption, their agreement may conceal the problem.
Human review helps when the reviewer has the evidence, time, understanding and authority to challenge the work. A name beside an approval button does not establish those conditions.
Standards are becoming useful—but their scope matters
There are concrete foundations to build on. They address different problems.
The Model Context Protocol’s published authorisation specification sets out access controls for HTTP connections. It includes requirements to validate that tokens were issued for the intended server and prohibits passing a received token straight through to an upstream service. Those are meaningful security boundaries. They do not establish whether a permitted business action is appropriate.
Agent2Agent provides a common protocol for communication and task exchange between agent systems, including security mechanisms. It helps systems work together. It does not, on its own, prove that their combined decisions meet an organisation’s requirements.
ISO/IEC 42001 is a published international standard for an AI management system: how an organisation establishes, maintains and improves its approach to AI. Its scope is broader than a connection protocol. It is still not a guarantee that every individual output is correct.
NIST’s AI Agent Standards Initiative is developing work around interoperability and secure agent systems. Its identity and authorisation work is particularly relevant to delegated action. An initiative and its developing guidance should not be presented as a completed universal safety certificate.
We need to ask which version applies, what has actually been implemented and what evidence demonstrates that the control works in this deployment.
Control has to survive a change
My interest in Agent Control and the Digital Labour Exchange starts with the job: what was requested, what could be spent, what authority was granted and what would count as an accepted outcome. I use that as a practical design lens here, not independent proof that an industry-wide problem has been solved.
A useful record would connect the accepted result to the model, tools, deployed code, source evidence and rules in force at the time. It would preserve uncertainty and failed attempts, rather than keeping only the successful answer.
That history matters when requirements change. Work accepted six weeks ago may have been correct under the rule then in force. We need to distinguish a new requirement from an old failure, and identify which existing decisions now need review.
For buyers, I would ask for a demonstration of five things: explain the outcome; challenge its evidence; stop further action; repair a failure; and replace a dependency without losing the essential record.
For the people affected, there must also be an accessible way to question a decision and reach someone with authority to remedy it. Protection cannot end at the purchasing organisation’s boundary.
The hidden danger is that capability expands while understanding, affordability and intervention fall behind.
The danger may not be a machine that refuses to obey. It may be an organisation that no longer knows what it has authorised—and cannot afford to find out.
Primary sources
Anthropic funding disclosure, 28 May 2026
MCP authorisation specification, 18 June 2025 version
Agent2Agent protocol specification
NIST AI Agent Standards Initiative
NIST agent identity and authorisation work
Also published on Medium ↗.
This article reflects the evidence and development status at its original publication date.