This is the first article in our Agent System Design series. Our goal is not to provide answers, but to ask more fundamental questions and establish a language and framework for reference.

Current Agent Systems Engineering mostly focuses on capability-level improvements — longer memory, more flexible token management, async communication, session state management, and so on. These are important and represent real engineering challenges. But as single-agent capabilities become sufficient, we are reaching an inflection point: these capability-layer improvements will soon cease to be the bottleneck. The recent attention on Paperclip (a GitHub trending project) shows that the market is starting to recognize that role frameworks may be the real bottleneck today. Paperclip’s company model essentially transplants human organizational design principles directly. But is this truly the best architecture for agent systems? Or should we approach it from first principles — why do we need systems at all? What makes agents unique? — to derive an organizational design more suited to agents? This is exactly what this article explores.

Starting Point: Why Human Organizational Design is the Wrong Template

Most people designing agent systems instinctively replicate human organizational patterns — hierarchical structures, department divisions, reporting lines, approval chains. This is an intuitive response because we are most familiar with it. However, such design may lead to fundamental mistakes.

The core reason is this: human organizational design largely exists to compensate for human limitations. When these constraints no longer apply, the optimal structure will change entirely.

What Are Human Organizations Compensating For?

Human cognitive limitations directly shape organizational forms. Because working memory is limited, we need documents, meetings, and knowledge management systems to compensate for memory gaps. Because attention decays, we develop specialization and shift systems to ensure continuous output. Because behavior is inconsistent, we establish processes, checklists, and audit mechanisms to reduce human variation.

More complex is that humans have political instincts and self-interest, which drive power structures and incentive mechanisms. Communication bandwidth limitations mean management must become information routers, and the org chart becomes the topology of communication. Additionally, trust decays over time, so we need contracts, performance evaluations, and culture-building to maintain organizational cohesion. Finally, knowledge exists in human minds and can be lost, which explains why onboarding, knowledge management, and succession planning become organizational infrastructure.

Agent systems face completely different constraints. Some human limitations have disappeared, while new constraints have emerged. This means the optimal organizational response should also be different.

The Constraint Profile of Agent Systems

Human Limitations That Have Disappeared

Agent systems do not tire, have no attention limits, and do not experience emotional fluctuations. These fundamental human physiological constraints simply do not exist in agent systems. Agent systems also lack political motives or self-interest — they do not pursue personal power, nor do they hide information for personal gain. Furthermore, knowledge does not leave: agent systems do not lose accumulated expertise due to personnel turnover, as long as the memory architecture is designed properly, knowledge can be preserved persistently. Under the same context and instructions, agent behavior is consistent, which contrasts with human behavioral variation. Finally, agents do not engage in information blocking due to ego.

New or Different Constraints

However, agent systems face an entirely new set of constraints. First is context window limitation: working memory truly exists but has a ceiling and can be artificially reset, which is fundamentally different from human memory continuity. Second is hallucination and confabulation (a psychological term referring to fictional, distorted or misunderstood memories) — agents may produce confident errors but have no internal signal of doubt, making errors difficult to self-detect.

Third, there is no persistent state by default — each invocation is new unless memory is explicitly designed, which contrasts with human continuous self-awareness. Fourth, errors accumulate and propagate — early errors are amplified downstream, and agents lack the natural intuition to correct them. Fifth, agents cannot know what they do not know — there is no genuine metacognition about their own knowledge limitations. Sixth, they are extremely sensitive to prompts — small framing changes can cause drastic behavioral shifts, which contrasts with humans relative stability in language environments. Seventh, there is no true goal internalization — agents optimize for stated goals, not underlying intent, which means goal drift risk always exists.

This constraint profile suggests that organizational principles should differ from those for humans. We need a new theoretical framework to guide agent system design.

Useful Frameworks from Systems Theory

Among the many frameworks available, two are particularly practical and actionable for agent system design:

One: Modular Theory — Loose Coupling, High Cohesion

The core concept from software architecture and organizational theory is that systems that can endure and scale are characterized by loose coupling (modules can change independently) and high cohesion (each module has a clear, singular responsibility).

In agent systems, a concrete example is our Orbit system: Sophie is responsible for bringing in knowledge leads, Quinn for content writing, and Devin for technical deployment. These three agents each have clear responsibility boundaries, with communication proceeding through structured handoffs rather than allowing any two agents to dialogue directly. This is high cohesion — each agent focuses only on its core capability; and loose coupling — adjusting Sophie’s internal prompt does not affect Devin’s behavior.

Human organizations frequently violate this principle — departments become deeply entangled through informal dependencies, tribal knowledge, and political alliances. Agent systems lack political alliances, making the loose coupling principle truly achievable — but it requires explicit design.

Two: Control Theory — Feedback Loops as an Organizational Principle

In control theory, a system’s quality depends on its feedback loops. The classic model is: sense → process → act → observe → correct. This means the system must continuously monitor its own output and adjust subsequent behavior based on observed results.

In human organizations, feedback loops are slow and noisy — quarterly reviews, annual assessments, customer complaints arrive months after events. In agent systems, feedback can be near-instantaneous, which changes the design thinking entirely.

A concrete example: in Orbit, when Quinn completes a draft, the status changes from Drafting to In Review — this itself is a feedback signal, telling the system there is output requiring human attention. If the review finds issues, the status changes back to Revising — this is another feedback, telling Quinn modifications are needed. This status mechanism is how we operationalize the observe → correct principle from control theory in actual system design.

This also points to the importance of error signal design — the system needs a way to detect whether its behavior deviates from intended purpose. This is difficult in human organizations because people hide failures. Agent systems do not hide failures, but they also do not naturally emit failure signals — therefore error signals must be explicitly designed into the system.

Three: Information Theory — Reducing Lossy Handoffs

Claude Shannon’s core insight is that information degrades every time it passes through a noisy channel. In human organizations, the noisy channel is people — every time a manager summarizes, every time a meeting loses detail, information attenuates.

The corresponding concept in agent systems is context handoff — the moment one agent passes its output to another. This is where information loss occurs. Each handoff compresses, reformats, or truncates context, causing the receiving agent to lack complete information.

The design principle derived from this is: minimize handoffs on any critical information path. Since handoffs are inevitable, they must be explicitly designed — specify what information must be preserved during transitions and establish verification mechanisms to ensure it is achieved. A dedicated knowledge management agent can serve as an architectural solution to lossy handoffs, responsible for maintaining and restoring context on demand.

Four: Cybernetics — The Law of Requisite Variety (Ashby’s Law)

Ashby’s Law in cybernetics (the law of requisite variety) is often stated as: a controller must have at least as much variety as the system it controls. What does variety mean here? Simply put, it is the size of the controller’s internal state space — how many different system states it can distinguish, represent, or map.

For example: a binary controller that can only output on or off cannot handle an environment requiring recognition of multiple-dimensional states such as indoor temperature, humidity, whether users are present, time of day, and so on. This is not because its computational capacity is insufficient, but because its internal representational capacity is inadequate to build a model complex enough to reflect the environment. Translating to agent systems: a simple agent that only executes single instructions cannot handle complex tasks requiring consideration of user intent, context, priority, risk assessment, and other multi-dimensional factors — not because the prompt is insufficient, but because the agent’s state space simply does not contain these dimensions.

This offers three insights for agent system design: First, do not let agents overly simplify their internal models in complex domains, or they will fail at edge cases — not because the model is insufficiently powerful, but because the instruction space does not contain enough situations. Second, the coordinator’s variety must exceed any single agent, otherwise the coordinator cannot effectively route requests. Third, emergent complexity requires emergent supervision — when the system becomes more powerful, meta-level monitoring must correspondingly become more complex.

Five: Redundancy and Graceful Degradation

Robust systems do not fail catastrophically — they degrade gracefully. The design of aerospace, nuclear, and financial systems all embody this: multiple redundant paths, safety devices, circuit breakers. These mechanisms ensure that when a component fails, the system does not collapse entirely but continues operating in a reduced-capacity mode.

Human organizations achieve this through informal redundancy — when someone is absent, people who happen to know certain matters step in. Agent systems lack this informal backup layer. Their redundancy must be explicitly designed.

Two patterns worth adopting: verification agents — an independent agent whose responsibility is to evaluate another agent’s output before it proceeds, which is not the same agent’s self-check (unreliable) but a truly independent evaluation process; circuit breakers — explicit stop conditions that halt the system when output falls below a confidence threshold, rather than letting errors propagate, pulling humans back when the system reaches its boundaries.

Six: Levels of Purpose — Mission, Strategy, Tactics

The concept from military theory and management science is that effective organizations operate at multiple levels of abstraction simultaneously, each level serving a different function. Mission is why we exist, the ultimate definition of success; Strategy is the approach we take to achieve the mission; Tactics are the specific actions we are currently taking.

Human organizations often conflate these, leading to drift — tactical victories undermine strategic direction, strategic choices forget the mission. Agent systems are particularly susceptible to this, because they can only access content within the context window — they may be tactically correct but strategically wrong, with no internal alarm.

The design principle is: objectives should be encoded at multiple levels of the system, not just at the task level. An agent given only a task (write a blog post about X) cannot access mission-level constraints (but never in a way that damages brand philosophical positioning). Mission and strategy must exist in context, not assumed to be inherited from above.

Seven: Emergence and the Danger of Local Optimization

Complex systems produce emergent behavior — outcomes that cannot be predicted from the behavior of individual components. This is both promise and danger. Emergence can bring unexpected innovation, but it can also lead to unpredictable problems.

In human organizations, emergence is managed through culture — shared values enable people to make autonomous decisions locally, but these decisions aggregate into coherent organizational behavior. Agent systems lack any meaningful culture. Each agent locally optimizes for its stated objectives. If these objectives are not aligned at the system level, aggregate behavior may appear coherent but actually be wrong — a system where every agent is competent may produce collectively suboptimal or contradictory output.

The design principle is: do not rely on emergent coordination. Coordination must be explicitly designed — either through a coordinator (hierarchical coordination) or through shared protocols (decentralized coordination). The choice between these is one of the most important architectural decisions in agent system design. For personal-scale systems, hierarchical coordination makes sense — a single coordinator agent. At larger scales, pure hierarchical systems become bottlenecks, and hybrid modes (hierarchical plus peer-to-peer communication for known, explicit tasks) become necessary.

(This topic is difficult to fully expand on now, but it is an extremely valuable direction of observation. We will continue exploring this in future articles, including observing and verifying this phenomenon in our own Orbit agent team.)

Toward a Framework

Bringing these threads together, optimal agent system design may mean:

1. Minimal necessary complexity — the system should be as simple as possible while still being able to handle the variety it needs.
2. Explicit information architecture — every key piece of context should have a designated owner and defined format.
3. Feedback loops at every level — the system should be able to detect when it deviates from intended purpose at the agent level, team level, and mission level.
4. Preserve human judgment at genuine boundaries — where the system’s variety is insufficient for the environment’s complexity.
5. Interfaces over internal implementation — design energy should go into what passes between agents, more than any single agent’s internal behavior.
6. Mission visibility everywhere — overall purpose and constraints should be visible in every agent’s context.

A Genuinely New Domain: Identity and Continuity

What no existing framework fully solves is the problem of identity and continuity in agent systems over time.

Human organizations maintain continuity through people — institutional memory, culture, relationships. When everything is agents, continuity must be designed at the architectural level.

A dedicated knowledge management agent represents one answer. But the harder question is: when an agent system scales, evolves, and changes its component agents over time, how does it maintain its identity — consistent values, aesthetics, and purpose?

This question has no clear answer in control theory, information theory, or cybernetics. It may require an entirely new framework — one that does not yet exist.

This, perhaps, is why this discipline needs to be established.

---

This article is the first in Human Beyond the Loop’s exploration of Agent System Design.