💼 Labor & AI
One Container Held Every Credential for a 4-Node AI Fleet. That Night, 4 Doors, 0 Reachable.
One credential container served four SSH doors; its outage left all four nodes unreachable. Plan 9 namespaces are the analogy, and the math is illustrative.
Four doors, one container, zero reachable. On the night this fleet needed a machine, an incident reconstructed from operator logs, NODE-A through NODE-D sat on a 192.168.7.x LAN with every SSH port listening and every login failing: all four nodes drew credentials from a single gateway container, and the container was down. A container of this kind does not fail per node; it fails totally, and recovery ran manual, slow, and avoidable with early-1990s protocol work.
In the fleet examined here, a coordinator reaches machines holding the real things (storage, sensors, LAN devices) through per-host SSH configs, keys parked in a Docker container, and per-task port forwarders. Two protocols standardize the conversation: the Model Context Protocol specifies how clients and servers exchange tools, resources, prompts, and discovery, and A2A how remote agents advertise capabilities, send messages, and run task lifecycles. Neither moves a machine's ground truth into the coordinator's namespace, so the fleet forwarded ports.
Pricing the Middleman
Run the numbers under one stated model: node and container states are independent, availability stationary, and credentials fail exactly when the container fails. Shared power, one LAN, and one deployment pipeline violate that independence, so treat every figure as illustrative structure.
Give four doors and the container 99.5 percent uptime each. Behind the container, a coordinator reaches one specific node 0.995 × 0.995 = 99.0 percent of the time, and reaches all four at once 0.9955 = 97.5 percent of the time. Node-local credentials make the same figures 99.5 and 0.9954 = 98.0 percent. Everyday reachability barely moves.
Total darkness is where the designs part ways. With node-local credentials, all four doors down at once has probability 0.0054 = 6.25 × 10−10, about 19.7 milliseconds a year. With the container in series, the fleet is dark whenever the container is: 0.005 × 8,760 = 43.8 expected hours a year from the container alone, an eight-million-fold difference in annual darkness; at 99.9 percent container uptime the figure falls to 8.76 hours a year, about 1.6 million times the 99.5-percent node-local estimate, and matched at 99.9 everywhere the ratio approaches three hundred million. (Exactly computed, the serial total 1 − 0.995 × (1 − 0.0054) adds only milliseconds.)
Two quieter failures deserve equal billing. A forwarded port can listen and still fail authentication downstream, a result exit-code-watching coordinators record as success, while a mount either appears in the namespace or throws. Blast radius belongs to this implementation, not the category: a single-instance container holding long-lived, baked-in keys concentrates theft, whereas a stateless broker over a dynamic secret manager, or a highly available service issuing short-lived credentials, changes both theft math and outage math in the direction its defenders promise. A fifth node meant a fifth key, a fifth port map, a fifth entry, each hand-maintained.
| Design | Standardizes | Ground truth lives | Blast radius | Auth |
|---|---|---|---|---|
| Plan 9 / 9P (early 1990s) | The namespace: files, devices, processes over one protocol | On the serving machine, mounted where needed | One mount, one connection | Authenticate on attach, then per-file permissions |
| MCP (spec 2026-07-28) | Client–server context exchange: tools, resources, prompts, discovery; topology out of scope | In server processes, local or remote; no machine namespace defined | One server's trust scope; fleets may run many servers or one remote server for many clients | Follows transport and server policy; stdio is a local transport, not a trust model |
| A2A | Agent discovery, messages, task lifecycles, artifacts between remote agents; machine state out of scope | With each agent; an implementation may expose live state as a capability | One agent's trust boundary | Deployment-dependent; the agent's card declares its requirements |
| Single-instance credential container (this incident) | Key storage in one place | Long-lived copies baked in or injected at provisioning | The whole fleet, per event | Long-lived keys checked out of one store |
What Plan 9 Actually Did
Bell Labs met this problem when a fleet meant machines, and its answer kept the transport, attach, and authorization machinery while moving where resources appear. In The Use of Name Spaces in Plan 9, Rob Pike, Dave Presotto, Ken Thompson, Howard Trickey, and Phil Winterbottom describe per-process namespaces joined by 9P: a process attaches a remote tree at a chosen name and reads devices and process tables as files. Its authors are candid that addressing, authentication, and connection control do not map cleanly into the filesystem; Plan 9 moved the crossing into the namespace, not away from it.
The protocol survived. Linux ships the v9fs client in mainline, QEMU documents a 9P backend for host directories, and Microsoft's WSL team has explained how Windows reaches Linux files through a 9P file server inside WSL 2.
Namespaces Instead of Doors
Treat this as an untested hypothesis, not a prescription. Each node's agent would serve its local ground truth as a 9P namespace over a WireGuard mesh, whose design paper reports under 4,000 lines excluding cryptographic primitives, with a tailnet providing addressing; a coordinator would mount NODE-C's namespace and read from the node that owns the answer. Mounting is the easy half: production still owes each node a distinct identity, a least-privilege export, a revocation path, audit semantics, and defined behavior when a mount fails mid-task. Treating a capability file as executable by whoever mounts it is a default to design away, not a feature, and nothing here measures whether serving names beats forwarding ports; that benchmark is the experiment.
What This Analysis Did Not Prove
Read the piece with its scope attached. The incident is a reconstruction from the operator's session logs, its identifiers anonymized; nobody independent audited it. Availability figures are assumptions under an independence model; correlated failures, one power strip, one router, one identity outage, raise darkness for both designs and hit the serial one harder. That row prices a single-instance, static-secret implementation; rotation, audit, and high availability are absent here by construction. Mounted-namespace consistency under partitions is unmeasured, and node-local secrets can make rotation and revocation harder. MCP and A2A are moving targets cited at their current specifications; either could absorb this problem.
The Strongest Case Against This Design
Strongest of all is the operational objection. Managed secret infrastructure, the Vault class of brokers, SSH certificate authorities, Tailscale SSH, and SPIFFE-style attestation, ships rotation, audit, policy, least privilege, and high availability today, while a bespoke 9P-over-WireGuard plane adds an attack surface, correlated-failure modes in the mesh and namespace, consistency costs, and support burdens a research protocol never matches. On that reading the container failed not because centralization is wrong but because a homemade, single-instance version lacked the broker's virtues. That objection is mostly correct and sets the bar: it prices the real attack paths, credential theft at rest, export misconfiguration, and mesh-peer compromise, all answered today by short-lived broker credentials, and a namespace design must answer the same paths with per-node identity and least-privilege exports. What it cannot answer is naming: Vault hardens a basket, a certificate authority shortens a door's lifetime, and neither gives a coordinator one namespace where another machine's truth appears as a readable file. Where reliability and audit dominate, the broker wins today.
The Playbook
Three moves, ordered by cost. First, put keys where the work runs, on the node that uses them, and if policy forces centralization, monitor the container like the dependency it is. Second, run the missing experiment once, as a lab exercise rather than a migration: serve one node's file tree with the v9fs client or QEMU's 9P backend linked above, mount it from a coordinator over WireGuard, and measure read latency, failure detection, and recovery against SSH. Third, if you build agent frameworks, treat what a machine sees as a permissioned, mountable path with identity and revocation; the demo that matters is a node joining with nothing but a mount.
The Bottom Line
For anyone running a multi-machine agent fleet, the boring fix comes first: centralize secrets only with broker-grade rotation, audit, and high availability, or keep them where the work runs and monitor what you centralize. Every figure above is illustrative, not measured, and prices one single-instance design, not the category in general. One question is older than the outage: whether a coordinator should read another machine's state by mounting it instead of tunneling to it. Bell Labs named it in the 1990s, Linux and WSL carry the protocol, and the namespace remains unshipped, not refuted.