Software Engineering and Loop Engineering Through the Lens of Engineering Cybernetics

As projects pile up, the feeling gets specific: it is not “too busy” — it is not knowing which things actually need me to look. Which project needs a nudge, which can wait another week, which has already drifted on progress and acceptance without me noticing — if every judgment depends on “checking each one,” people collapse before the projects do. Long stretches switching across multi-platform projects make this especially clear; it starts to grind down physical and mental health.

Every project manager and architect tries some form of split management. In multi-agent systems the pattern converges on splitting “whether to intervene” into two layers: most state changes are handled by the system itself — progress reminders, anomaly detection, routine reports — those do not need you; only when certain conditions fire does something truly land in front of you — wrong direction, repeated failures, situations the system has never seen. That “layering + intervene only at critical points” intuition is not new (and you need not panic at AI KOL coinages). In 1954, Qian Xuesen already wrote it as a complete language in Engineering Cybernetics, and that language is circulating again in software engineering and AI agent systems. By 2026 someone turned it into the fashionable idea of Loop Engineering.

What this post does: take the layered intuition from managing projects, break it into reusable concepts, then see how those concepts have long lived in CI/CD, microservice architecture, SRE on-call, observability, and chaos engineering; how they reappear under new names in today’s AI agent systems; and finally land back on the workflow I am actually using.

---

1. You Need Definitions Before You Can Diagnose “What Went Wrong”

Past a certain project scale, gut feel that “this project is unhealthy” is unreliable — the same “something feels off” can have completely different causes: you never got the signal you should have (nobody told you a PR was stuck for three days), you got the signal but the standard is fuzzy (“three days stuck” — is that serious?), you judged it correctly but nobody acts (you know it is serious and nobody follows up), or you acted with the wrong method (a bland reminder that the owner ignored).

These four failure modes look alike; the fixes are completely different. Separating them needs a vocabulary that can dissect “how the system works,” not more “it feels off.”

1.1 The Quartet: Any Continuously Running System Can Be Taken Apart

Engineering cybernetics gives exactly that vocabulary. One of its founders, Qian Xuesen, published Engineering Cybernetics with McGraw-Hill in 1954 and broke any system that “wants to stay stable and move toward a goal” into four roles^[1]:

A thermostat does not think, yet it understands control better than many systems marketed as “smart” — it keeps measuring, keeps comparing, corrects when the gap is large, waits when it is small, never needs a human staring at it, and never runs away. How the five roles close into a loop:

flowchart LR
    SP["SETPOINT<br/>target 25°C"] --> COMP
    SENSOR["SENSOR<br/>thermometer"] -->|"measured 22°C"| COMP["COMPARATOR<br/>compare target vs actual"]
    COMP -->|"error 3°C"| CTRL["CONTROLLER<br/>decide action"]
    CTRL -->|"heat command"| ACT["ACTUATOR<br/>heating element on"]
    ACT -->|"change room temp"| PLANT["plant<br/>room"]
    PLANT -->|"new room temp"| SENSOR

The value of this structure is not dressing common sense in jargon — it is that it can be taken apart and inspected. Restate the four project-management failure modes:

• “Nobody told you the PR was stuck for three days” → missing SENSOR (no measurement)
• “Don’t know if three days counts as serious” → missing COMPARATOR (no clear judgment standard)
• “Know it is serious but nobody follows up” → missing CONTROLLER (judgment reached, action never fires)
• “Pushed a bland reminder” → ineffective ACTUATOR (action fired, execution quality poor)

Four different diagnoses, four different fixes. That is how the definitions are used: when “the system drifted again,” ask which role failed first — do not jump straight to “redesign the whole process.”

Engineering cybernetics came after Norbert Wiener’s 1948 Cybernetics: Or Control and Communication in the Animal and the Machine^[2]. Wiener defined cybernetics as “the study of control and communication in the animal and the machine” — a more philosophical general account. Qian grounded it where engineers can use it directly: servomechanisms, error control, mathematical criteria for system stability. Together they define the two ends of the field: why the idea matters, and how it becomes something computable.

Everything that follows is essentially the same move: take this quartet plus setpoint, and inspect concrete areas of software engineering — what are their SENSOR / COMPARATOR / CONTROLLER / ACTUATOR, and which role is missing or broken.

1.2 Feedback Loops: The Mechanism That Lets a System Correct Itself

Wire the quartet together and you have a feedback loop: output is remeasured, compared to the goal again, a new corrective action is generated, and that action produces new output. The loop itself has two opposite modes of operation.

Negative feedback tends toward stability — the name sounds “negative,” but it is the most important stabilizing mechanism in cybernetics. A thermostat is classic negative feedback: room below setpoint, heater on; room rises, heating eases; continue until error goes to zero. Any system that wants to stay near a state relies on negative feedback.

Positive feedback amplifies deviation; it seeks change, not stability. A microphone picks up the speaker, amplifies, gets picked up again — louder each lap until howl or saturation. Compound interest is a mild form: interest joins principal, next period’s interest is higher. In software, a cache stampede is classic bad positive feedback:

flowchart LR
    A["cache hit rate drops"] --> B["more requests hit upstream DB"]
    B --> C["DB slows down"]
    C --> D["more timeouts and retries"]
    D --> A

Spotting a hidden positive-feedback loop often matters more than designing negative feedback — positive feedback runs away exponentially, and the system will not shout stop on its own.

Project management intuition matches the distinction: a healthy project system should be negative feedback — deviations are detected and pulled back; but “push too hard → morale drops → efficiency worse → push harder” is a hidden positive loop, and managers often only notice after two or three laps that they are accelerating the wrong way. Telling whether the loop in front of you is negative or positive is the first step in diagnosing “why this process gets worse the more we run it” — if managing makes more mess and pushing makes things slower, the problem is probably not “execution insufficient”; you stepped into a positive loop. “Try harder” makes it worse; the right move is to break the loop, not feed it more fuel.

1.3 Hierarchy: Not Every Decision Deserves Your Hands On It

The quartet describes how a single loop runs; real systems never have only one loop. With many projects you cannot run a full “measure–compare–decide–act” cycle on each — you do not have that much attention.

Qian’s solution is layering: split the control system into two layers — one for high-frequency, routine, fully rule-based actions; the other only for low-frequency decisions that need value judgment^[1].

• Direct control layer: runs on milliseconds to seconds; handles “rule-decidable” cases. ABS checks wheel speed every millisecond; you never feel it working.
• Organizational layer: runs on minutes to months; handles cases that need judgment. A board reviews KPI anomalies quarterly; it does not watch every frontline action daily.

The layers connect by protocol — the organizational layer issues abstract goals (“optimize fuel efficiency this quarter”); the direct layer executes concrete commands (“throttle to 60%”); the protocol says how abstract goals translate into concrete actions.

flowchart TB
    subgraph org["Organizational layer (minutes to months)"]
        O1["set goals / setpoint"]
        O2["monitor anomaly aggregates"]
        O3["critical decisions: continue / adjust / stop"]
    end
    subgraph direct["Direct control layer (ms to seconds)"]
        D1["continuous measurement"]
        D2["rule-based comparison"]
        D3["automatic corrective action"]
    end
    O1 -->|"protocol: abstract goal → concrete thresholds"| D2
    D1 --> D2 --> D3 --> D1
    D3 -->|"anomaly aggregates (protocol: concrete events → abstract signals)"| O2
    O2 --> O3 --> O1

Back to project management: you are the organizational layer. Day-to-day progress tracking, anomaly detection, and status reporting should be fully delegated to the direct layer (scripts, monitors, cron). You appear only at protocol-defined critical points — direction confirmation, escalation, acceptance decisions. That is not “laziness”; it is what hierarchy itself requires: making the organizational layer handle high-frequency decisions it is bad at forces it to do what it cannot do well — the organizational layer burns out while the direct layer never learns.

1.4 Human in the Loop: Qian Said Humans Are Not a Bug — They Are a Feature

The organizational layer in a hierarchy is often a human. A frequently overlooked stance in Qian’s theory: humans are not interference to eliminate, but an irreplaceable functional module in the control system^[1].

Classical control theory (and many people still think this way) treats “human intervention” as noise — the ideal system is fully automatic; humans appear only at design time. Qian’s stance differs: systems meet situations design never foresaw, decisions that need trade-offs rather than calculation, boundaries where “rules cannot say.” In those cases human judgment, experience, and adaptation are the only parts of the system that can handle them. Autopilot needs no human 99% of the time; in bird strike or engine failure, the pilot decides whether the system can still stay stable.

In project-management language: automation is not there to “replace your judgment” — it is there to filter out what does not need judgment, so judgment is spent only where it is needed. A “fully automatic” project system and a “you watch everything” project system are both extremes this theory rejects — the former automates machine bias; the latter turns people back into part of the execution layer.

With the three core ideas in place (feedback loops, hierarchy, human in the loop), the next sections map them onto concrete software-engineering domains — starting with what a feedback loop looks like in CI/CD.

---

2. Feedback Loops: From Monthly Releases to Master Straight to Production

CI/CD is where feedback loops are easiest to see in software engineering: every stage maps to a role in the quartet, and the loop’s “frequency” has visibly compressed over the past twenty years.

2.1 A History of Frequency Is a History of Tightening Feedback

Early software release was monthly batch: developers submitted code, a month’s worth piled up, QA ran full regression, ops deployed by hand. Feedback existed — test results went back to developers — but the cycle was too long: weeks between “this change introduced a bug” and “we found that bug,” with dozens of new commits stacked in between; localization cost grew geometrically.

flowchart LR
    A["monthly batch<br/>feedback: weeks"] --> B["weekly release<br/>feedback: days"]
    B --> C["daily release<br/>feedback: hours"]
    C --> D["master → production<br/>feedback: minutes"]

Continuous integration (CI) first compressed test feedback from “monthly” to “every commit”: GitHub Actions declares event-driven builds with on:^[3], and every commit triggers a full test suite:

name: CI
on:
  push:
    branches: [main]
  pull_request:
    types: [opened, synchronize]
  schedule:
    - cron: '0 2 * * *'   # full regression once more each night
  workflow_dispatch:       # manual emergency entry

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm test

Each on: event is a SENSOR trigger — push is “new code,” schedule is “timed patrol,” workflow_dispatch is a human manual trigger. In quartet terms: raise SENSOR (test results) sampling from “months” to “minutes.”

Continuous deployment (CD) then automates “ship or not after tests pass.” Meta (then Facebook) documented the endpoint of this path in a 2017 engineering post: from monthly, to weekly, to daily, finally “100% of production web servers released directly from master”^[4] — release cycle from “months” to “minutes.” That is not only efficiency; it is an order-of-magnitude jump in feedback frequency: the higher the frequency, the smaller the deviation each correction must fix, and overall behavior gets smoother. Same logic as “how often do we run project retros” — monthly retros mean you discover wrong direction after a month of wrong; high-frequency, small-step checkpoints keep each correction cheap.

2.2 GitLab / Jenkins: Staged Feedback Rings

CI platforms differ on “staged testing,” but the structure is shared — split a build into stages; if an earlier stage fails, later stages do not run; feedback cuts early so bad changes do not travel downstream.

flowchart LR
    A["Lint<br/>seconds"] -->|"pass"| B["unit tests<br/>minutes"]
    B -->|"pass"| C["integration<br/>several minutes"]
    C -->|"pass"| D["deploy staging<br/>several minutes"]
    D -->|"pass"| E["deploy production<br/>minutes"]
    A -.->|"fail"| F["abort immediately, notify developer"]
    B -.->|"fail"| F
    C -.->|"fail"| F

GitLab CI describes the chain with stages in .gitlab-ci.yml; Jenkins uses pipeline { stages { ... } } in a Jenkinsfile. The design philosophy is the same: earlier stages are cheaper and faster; later stages are dearer and slower — so exclude with second-scale checks what you can, instead of finding a lint error in an expensive final integration. That is layered control too: cheap SENSOR first, expensive SENSOR later; cheap checks filter most noise so expensive checks only handle what needs them.

2.3 Spinnaker: Canary Release as Layered Risk Control

If stage splits answer “when to test,” canary release answers the finer question “should we trust the new version.” A typical Spinnaker-style canary:

flowchart TD
    A["deploy new version to 1% traffic"] --> B{"metrics healthy?"}
    B -->|"yes"| C["expand to 5%"]
    B -->|"no"| Z["auto rollback"]
    C --> D{"metrics healthy?"}
    D -->|"yes"| E["expand to 25%"]
    D -->|"no"| Z
    E --> F{"metrics healthy?"}
    F -->|"yes"| G["full rollout"]
    F -->|"no"| Z

Each “metrics healthy?” is a full quartet cycle — SENSOR measures new-version error rate / latency, COMPARATOR compares to old version or preset thresholds, CONTROLLER decides expand or roll back, ACTUATOR adjusts traffic share. The process splits “is the new version trustworthy” into four smaller risks, each capped at “1% / 5% / 25%” — even wrong judgment has bounded cost. This is hierarchy miniaturized inside one release: staged validation instead of one all-or-nothing bet.

2.4 Kubernetes Operator: The Cleanest Industrial Implementation

If you want a textbook feedback loop in software engineering, the Kubernetes Operator reconcile loop is currently the cleanest example. The official docs map almost word-for-word to the quartet:

“Operators follow a reconciliation loop, comparing desired state (in the spec) against actual state (in the status of the object). If the actual state doesn’t match the desired state, the Operator takes corrective action until the spec and status match.”^[5]

An Operator keeps comparing desired state (in spec) to actual state (in status) — mismatch triggers corrective action until they match. A concrete CRD sketch:

apiVersion: apps.example.com/v1
kind: WebAppReplica
metadata:
  name: my-web-app
spec:
  replicas: 3          # ← desired state (SETPOINT)
status:
  readyReplicas: 2      # ← actual state (what SENSOR measured)

The reconcile loop sees spec.replicas = 3 and status.readyReplicas = 2, knows one replica is missing, and creates a new Pod — CONTROLLER judgment + ACTUATOR execution. Mapping:

• desired state (spec) = SETPOINT
• continuous watch of actual state = SENSOR
• compare desired vs actual = COMPARATOR
• “takes corrective action” on mismatch = CONTROLLER + ACTUATOR

A Kubernetes Operator does not need a human to say “do this now”; it keeps asking how far current state is from the state it was told to maintain, then corrects itself. Thermostat logic scaled up — the plant is cluster resource state instead of room temperature.

2.5 TDD: The Most Microscopic Feedback Ring

Feedback loops are not only architectural. Test-driven development’s red–green–refactor cycle — write a failing test (red), minimal implementation to pass (green), then refactor — is one of the highest-frequency feedback rings in software engineering, running on seconds to minutes.

flowchart LR
    A["write a failing test (red)"] --> B["minimal implementation to pass (green)"]
    B --> C["refactor, keep tests green"]
    C --> A

The test pyramid (Mike Cohn) describes the layering of that ring:

flowchart TD
    E["end-to-end<br/>few · tens of minutes · closest to real"]
    I["integration<br/>medium · minutes"]
    U["unit<br/>many · seconds · fastest and cheapest"]
    U --> I --> E

Higher layers: slower feedback, higher realism — again layered control: high-frequency layer handles “is this function right,” low-frequency layer handles “do these services compose right.” A common anti-pattern reverses it — heavy E2E, thin unit tests — so every change waits tens of minutes to know; feedback frequency collapses and correction cost rises.

2.6 PID: A General Algorithm for “How Hard to Correct”

So far: when to measure and when to compare. Another question: how does the CONTROLLER decide correction strength? The most common, time-tested answer is the PID control law — also behind thermostats, cruise control, and industrial reactor temperature control. The core is intuitive; no formulas required:

• P (proportional): correction strength proportional to “how large the error is now” — larger error, stronger action. Room far from target → heat more; near → heat less.
• I (integral): also look at “accumulated error over time” — if error persists and never clears, increase strength even when current error is small, so the system does not “always miss by a little and never quite arrive.”
• D (derivative): also consider “how fast the error is changing” — if error is shrinking fast, ease correction early to avoid overshoot past the target and bounce back.

Together, PID watches “how far now,” “how far for how long,” and “how fast it is moving” — which is why it is smoother and less oscillatory than a simple “if error > threshold, correct” rule.

Software has direct physical carriers. Jenkins architecture — a central controller schedules, agent nodes execute — is classic central controller + multiple actuators: Jenkinsfile in the controller defines “correction logic,” queued builds approximate the “integral term” — the longer the backlog, the stronger the response to resource pressure should be.

GitLab CI and Jenkins staged execution (stage 1 must pass before stage 2) is closer to cascade control: each stage is a small independent controller; only when the previous converges (succeeds) does the next start — same design philosophy as industrial control using several simple controllers in series instead of one complex controller for everything. Simple parts combined are easier to understand and debug than one complex whole.

Practical takeaway: when an automation rule always “overreacts” or “underreacts,” the threshold may not be wrong — you may be missing one of the three PID terms. Rules that only see current error (pure P) stay idle when error persists but stays modest; rules that ignore trend overshoot easily — same diagnostic style as section 1’s “which role failed,” refined inside the CONTROLLER’s algorithm.

2.7 Meta at Extreme Scale: When Feedback Frequency Becomes Adaptive

An ICSE 2016 paper on continuous deployment at Facebook and OANDA shows Facebook already at tens to hundreds of deploys per day. At that frequency, release behavior is no longer “human-approved batch processing”; it is closer to adaptive control in cybernetics: the system continuously monitors post-deploy user feedback (error rate, conversion, performance drift) and automatically adjusts next release pace and scope — humans no longer decide each release; they intervene only at the higher abstraction of “should overall release policy change.”

2.8 Feedback Loops Are Not Free

An easy-to-miss cost: feedback loops themselves cost money — test infrastructure, monitoring, deploy pipelines need maintenance, and that cost must be less than the losses they prevent, or “building the loop” is net negative. That is why small teams and early projects often skip full CI/CD at first — not because it does not matter, but because current scale cannot carry the maintenance. The decision tree at the end returns to this criterion.

---

3. Hierarchy: Why Splitting Microservices Turns Into an Org Problem

Section 1 gave the abstract definition — direct layer for high-frequency decisions, organizational layer for low-frequency judgment. This section lands on a concrete question: why does splitting a monolith into microservices so often become an org redesign?

3.1 Coupling Is the Real Source of Complexity

More components → possible interactions grow exponentially — n components have at most n(n-1)/2 pairwise links; multi-party interactions grow even faster. Single-layer control fails past a scale not because engineers are not smart enough, but because computation and fault isolation have physical limits.

Layering finds natural “weak coupling” boundaries, bundles strongly coupled parts into subsystems, and keeps only few, explicit connection points between them. BGP is the industrial practice at the network-protocol level: each autonomous system (AS) exchanges routes only with neighbors, yet the whole internet gets globally consistent path selection — no god-node knows full topology, but layered protocols make local exchange enough for global stability.

3.2 Microservice Architecture: Turning the Coupling Matrix Into Diagonal Blocks

A monolith’s coupling can be imagined as a huge relation matrix — any module change can touch any cell:

flowchart TB
    subgraph mono["Monolith: any modules couple"]
        direction LR
        M1["users"] --- M2["orders"]
        M2 --- M3["inventory"]
        M3 --- M1
        M1 --- M4["payments"]
        M4 --- M2
        M4 --- M3
    end

Microservices rearrange that matrix so strongly coupled parts gather into diagonal blocks, with only few, explicitly defined interfaces (usually API calls) between blocks:

flowchart LR
    subgraph S1["user service"]
        direction TB
        U1["strong internal coupling"]
    end
    subgraph S2["order service"]
        direction TB
        O1["strong internal coupling"]
    end
    subgraph S3["inventory service"]
        direction TB
        I1["strong internal coupling"]
    end
    S1 -->|"explicit API"| S2
    S2 -->|"explicit API"| S3

That is more than “better code organization.” Control theory has a clear claim: if coupling between two subsystems is weak enough, they can be controlled independently — each keeps its own stability with simple methods; no central controller must coordinate every detail. That is why microservice gains are often not “better performance” but “better fault isolation” and “teams can ship independently” — both direct consequences of weaker coupling.

Kubernetes Operators appear again here: each Operator runs its own resource reconcile loop and interacts via API objects (not shared memory or direct calls) — diagonal blocks + weak interfaces at the orchestration layer. CNCF cloud-native white papers describe the same loose coupling, independent deploy, independent scale orientation.

3.3 Conway’s Law: Org Structure Seeps Into Code Structure

Melvin Conway observed in 1968, later widely cited: “Any organization that designs a system will produce a design whose structure is a copy of the organization’s communication structure.”

flowchart LR
    subgraph org["Org communication structure"]
        T1["Team A"] <--> T2["Team B"]
        T2 <--> T3["Team C"]
    end
    org -.->|"projects onto"| sys["System architecture"]
    subgraph sys["System architecture"]
        M1["Module A"] <--> M2["Module B"]
        M2 <--> M3["Module C"]
    end

That explains why many teams still cannot cleanly “split microservices” — if two teams already need frequent talk and mutual help, whatever the code architecture, real coupling (who depends on whom, whose change hits whom) stubbornly mirrors communication. Code architecture is a projection of org communication, not the reverse.

The observation also opens a reverse move — the reverse Conway maneuver: design the org first, then let code architecture grow into the shape you want.

3.4 Team Topologies: Four Team Types

A concrete “design org first, let architecture follow” method is Team Topologies, with four team types:

And three interaction modes: Collaboration (two teams tightly paired on one thing), X-as-a-Service (one team consumes another’s capability with clear boundaries), Facilitating (help another team raise capability, do not do the work for them).

In project terms: if two project groups keep blocking each other, the first question is often not “how to split the code” but “should these two teams’ reporting lines and communication frequency have been separated, and should the interaction mode shift from tight collaboration to X-as-a-Service.” Same judgment as microservices — “too tightly coupled services should merge or redraw boundaries” — only at org level instead of code level.

3.5 Coupling Strength Can Be Roughly Quantified, Not Only Felt

So far coupling was “strong” / “weak.” In real decisions, strength can be roughly quantified — no strict model required. A crude but useful estimate: cross-boundary call frequency × cost per call. Tens of thousands of complex syncing API calls per day between two services is strong coupling; splitting only creates more network round-trips and consistency pain. Dozens of coarse interactions per day (e.g. one daily aggregate sync) is weak coupling; independent deploy and evolution are fine.

The same estimate applies to team collaboration: several daily sync meetings = strong coupling, bad for separate reporting lines; one weekly alignment = weak coupling, safe to run independently. The key to “split or not” is not “do they look like two independent things,” but “will cross-boundary communication cost explode after the split.”

A common mistake is splitting services or teams early because “business domains look different” without measuring interaction frequency — then call chains get long and slow, and sync meetings multiply vs before. Classic failure: layering never really reduced coupling; it only moved coupling from code into network calls.

3.6 Layering Is Not a Free Lunch

Layering solves single-layer control but adds cost: cross-layer communication needs protocols, protocols need maintenance, and a bad protocol is worse than none — if the organizational layer issues fuzzy goals (“improve stability” instead of “P99 latency under 200ms”), the direct layer cannot translate them into actions; layering creates information loss. That is why “setpoints must be clear” returns in later sections — it is a precondition for any layered system, not a detail.

---

4. Human in the Loop: What SRE On-Call and “Human Feedback” Are Actually Doing

Section 1 said humans are not a bug but an irreplaceable module. This section lands that claim on two scenes: SRE on-call, and human feedback when training LLMs. On the surface unrelated — ops practice vs ML technique — they are isomorphic in cybernetic terms.

4.1 The On-Call Engineer Is a “Nonlinear Organizational Layer”

SRE on-call engineers the “human in the loop” principle into an executable org process. Standard incident response:

flowchart LR
    A["Detect<br/>auto alert / customer report"] --> B["Triage<br/>on-call assesses severity"]
    B --> C["Respond<br/>fix actions"]
    C --> D["Mitigate<br/>rollback / rate limit / shift traffic"]
    D --> E["Root-cause fix<br/>permanent repair"]
    E --> F["Review<br/>blameless postmortem"]
    F -.->|"improve SOP / automation rules"| A

Each step has SLAs — P1: five-minute response, one-hour mitigation, twenty-four-hour root cause. Map each step to the quartet:

Worth saying alone: a postmortem does not repair system state — it repairs the controller. Day-to-day response returns the system to setpoint; postmortem asks “next time the same deviation appears, should judgment logic or thresholds change” — a meta-layer operation beyond the direct layer’s routine regulation, done only by the organizational layer (humans) afterward.

In this flow, auto-alert and auto-recovery are the direct layer (ms–second response); on-call is the organizational layer (minutes–hours). The boundary is clear: threshold-decidable goes to automation (CPU > 90% auto-scale); multi-party coordination, risk judgment, customer communication goes to humans (rollback at peak? announce externally?).

PagerDuty-class schedulers are the “protocol translation layer” — which alert → which engineer, what priority, what escalation — the concrete engineering of the protocol between organizational and direct layers.

4.2 Sheridan’s Five Levels: More Automation Is Not Always Better

Psychologist Thomas Sheridan in 1992, in his classic work on human supervisory control, split human–machine collaboration into five levels from fully manual to fully automatic^[6]:

flowchart LR
    L1["Level 1<br/>fully manual"] --> L2["Level 2<br/>action support"]
    L2 --> L3["Level 3<br/>shared control"]
    L3 --> L4["Level 4<br/>supervisory"]
    L4 --> L5["Level 5<br/>fully automatic"]

Most mature SRE on-call sits between levels 3–4: auto-alert + auto-recovery run first; on-call watches the dashboard; escalation brings real intervention. Counterintuitive but important: higher level is not always better. Level 5 requires predefined correct responses for all possible cases, and the real world always has unknown unknowns — skipping HIL straight to full auto has had costly lessons (Boeing 737 MAX MCAS tried fully autonomous correction without adequate manual takeover paths; two fatal accidents). Stability is not a monotonic function of automation; it is “automation degree” and “reliability of human takeover paths” together — a Level 5 system without a reliable degrade path to Level 3 is more dangerous than an honest Level 3 system.

4.3 RLHF: Turning “Human Feedback” Into an Optimizable Signal

If SRE on-call is the most intuitive human-in-the-loop practice, reinforcement learning from human feedback (RLHF) is the mathematical version for training LLMs — turning the fuzzy “how humans judge good vs bad” into an objective that gradient descent can optimize.

A cocktail analogy makes the process intuitive: you taste and say “more lemon”; the bartender adjusts — human-feedback-driven optimization. RLHF splits that into three stages; OpenAI scaled it industrially for InstructGPT^{[7] [8]}:

flowchart LR
    A["Stage 1<br/>collect human preferences<br/>labelers pick better output"] --> B["Stage 2<br/>train reward model<br/>learn to simulate human standards"]
    B --> C["Stage 3<br/>PPO-optimize the LM<br/>bias outputs toward high scores"]
    C -.->|"keep sampling, keep evaluating"| A

1. Collect human preferences: labelers see two outputs, pick the better one; accumulate pairwise “which is better” data.
2. Train a reward model: use that data so the model predicts “which output humans would prefer” — essentially learning a function that approximates human judgment.
3. Optimize the language model with that reward: RL (usually PPO) adjusts LM parameters so outputs tend to score higher.

Quartet mapping: stage one collects SENSOR data (human preferences across outputs); stage two trains a COMPARATOR (differentiable stand-in for human standards); stage three uses that COMPARATOR as feedback to adjust the CONTROLLER (the LM itself). The process is “train an automated judge from human feedback, then drive optimization with that judge” — isomorphic to on-call postmortems encoding experience into Runbooks that become automation scripts, only at a different layer.

A risk that must be admitted: the trained reward model is only an approximation of human standards; approximation and truth always diverge. When the optimizer is strong and runs long enough, it exploits the gap — patterns that “score high on the reward model but humans would not actually like”: reward hacking. A systematic survey of RLHF lists a long set of open problems^[9]:

• Reward hacking: model finds reward-model holes; outputs “look right, actually useless”
• Distribution shift: train scenes ≠ deploy scenes; the reward model fails in new contexts
• Labeler disagreement: annotators disagree on “which is better”; training data is noisy
• Deceptive behavior: model learns “looks honest, actually gaming the scoring standard” instead of being honest

Shared structure: any system that replaces real judgment with an approximate judge amplifies approximation error under long optimization pressure — whether that judge is a trained reward model or automation rules written after an on-call postmortem. Writing automation rules and training reward models face the same class of risk; “overfitting” shows up as incomplete rule coverage in one case and exploitable scoring holes in the other.

4.4 Rule-Based Alignment: Principles as Setpoints

Anthropic’s 2022 Constitutional AI takes a path not identical to RLHF: less online human feedback, more an explicit set of “principles” (helpful, harmless, honest); the model self-critiques and self-revises against them, then trains on revised data^[10].

In section 1’s language: the setpoint moves from “an implicit goal only online human feedback can pin down” to “an explicit set of principles written ahead of time.” Benefit: no waiting for humans at every step. Cost: principles must be clear enough; fuzzy principles are a fuzzy SETPOINT, and a fuzzy goal can never be tracked stably. That returns to the end of section 3 — fuzzy organizational goals leave the direct layer stranded, whether that “direct layer” is an automation script or a language model being trained.

4.5 Trade-offs Between the Two Paths

RLHF and Constitutional AI are not mutually exclusive; they sit at opposite ends of the spectrum from “online human feedback” to “offline explicit principles.” Industry practice usually combines them: cheap self-revision from rules first, then a smaller amount of precise human feedback for final calibration. Same division of labor as SRE on-call — automation rules handle most known, foreseeable scenes; human feedback handles boundaries rules cannot write and that need on-the-spot judgment.

Cost structures differ — an important selection factor:

A detail worth noting: if multiple principles apply at once — “be helpful” and “do no harm” — they can conflict (requested help itself is risky). A single-setpoint frame is not enough; you need multi-objective satisfaction — essentially “which goal has higher priority.” Same class of problem as section 3’s “fuzzy organizational goals strand the direct layer” — here the “direct layer” is the model juggling multiple principles.

4.6 Debate to Approximate Truth: Another Way to Make Judgment More Reliable

RLHF and Constitutional AI both try to make a judge closer to real human standards. A third path thinks differently: let two AIs debate each other; the human judge only needs to decide whose argument is more persuasive, not which answer is correct directly.

The starting point is practical: many problems are too complex for humans to judge answers directly — but humans can often judge “this argument does not hold, because the other side pointed out a hole.” OpenAI’s 2018 AI Safety via Debate is a concrete form^[14]: two comparable agents take turns finding holes in each other’s arguments; in theory complex problems decompose into small judgments a human can verify one by one, even if the judge cannot solve the whole problem alone.

flowchart LR
    Q["complex question"] --> A1["Agent A argues"]
    Q --> A2["Agent B rebuts / finds holes"]
    A1 --> J["human judge"]
    A2 --> J
    J -->|"judge only who was more persuasive this round"| Round["next debate round"]
    Round -.->|"after many rounds"| Final["judge's final decision"]

Game-theoretically this is a two-player zero-sum equilibrium — when the game is well designed and both play optimally, the equilibrium can be closer to truth than either answer alone. Critical structural difference from RLHF and CAI: RLHF and CAI train a better judge; debate designs game rules so the judge (human) need not get smarter to reach reliable conclusions — if the rules are right, a limited judge can still get a reliable call by watching attack and defense.

In everyday project management this looks like a common, underused trick: when you cannot directly judge a technical proposal, let two engineers with opposing views debate face to face — often a faster reliable call than burying yourself in research — you need not become the domain expert; you only need to judge “whose argument holds better this round.” That is why many teams deliberately design “devil’s advocate” roles (red team / blue team review) — not to manufacture conflict, but so decision-makers need not hold all expertise to get a reliable judgment from watching the exchange.

4.7 Why Classical Cybernetic Intuition Discounts in AI Systems

Previous sections mapped RLHF and Constitutional AI into this cybernetic frame, but an honest question remains: why can we not simply say “AI systems are just another instance of classical control systems” and copy every stability criterion?

Because AI systems — especially LLM-driven agent systems — differ from industrial control (thermostats, aircraft autopilot, chemical reactors) in three essential ways that make many of classical cybernetics’ “free stability guarantees” unavailable:

First difference: non-stationarity. Classical control assumes plant dynamics (“how much throttle raises speed”) are basically fixed or slow-changing. Language-model “parameters” change every retrain, and user preferences themselves keep evolving — the control law must keep adapting to a moving target, not aim at a fixed bullseye.

Second difference: sparse feedback. A thermostat measures temperature every millisecond; feedback is continuous and dense. Human preference feedback on AI outputs — RLHF’s “this answer is better than that” — is discrete, sparse, and expensive to collect; the model often takes many steps before one feedback arrives, and “how much did this step contribute to the final outcome” is hard — the classic credit-assignment problem in RL.

Third difference: adversariality. Classical “disturbances” are usually modelable natural noise (wind, temperature). AI “disturbances” may come from another agent with its own goals — e.g. malicious prompt injection. Those disturbances cannot be enumerated ahead of time; attackers keep finding new holes — unlike robust-control assumptions that you design for all known disturbances in advance.

Together: engineering cybernetics gives AI systems useful language and some math tools, but you cannot expect industrial-control-grade stability guarantees. RLHF reward hacking, Constitutional AI principle conflicts, multi-agent coordination failures are “classical stability criteria failing in new scenes,” not merely “sloppy engineering.” Honestly admitting that boundary is more useful than overselling cybernetics’ scope — which is why this post, when discussing AI agents, keeps stressing relatively plain engineering moves like “independent checkers” and “escalation notices,” rather than claiming a formal stability proof.

---

5. Observability and Digital Twins: Can You “Guess Internal State From Outputs”

Previous sections were about “how the system moves.” This one is a more basic premise: how do you know what state the system is in now. Feedback loops require SENSOR to measure accurately, but often the state you want cannot be observed directly — only inferred from indirect outputs.

5.1 The Engineering Definition of Observability

Control theory has a precise concept called observability: a system is observable if, from outputs alone, you can uniquely determine its internal true state. Conversely, if some internal dimensions never affect any observable output, those dimensions are unobservable — no matter how often you measure, you cannot guess what they are doing.

Software observability practice does exactly that. OpenTelemetry (CNCF’s observability standard) splits it into three pillars:

flowchart TB
    subgraph pillars["Three pillars of observability"]
        M["Metrics<br/>continuous numeric measurement"]
        L["Logs<br/>discrete event records"]
        T["Traces<br/>cross-service causal chains"]
    end
    M --> Q["Can we infer internal state?"]
    L --> Q
    T --> Q

• Metrics: continuous numeric measurement — CPU, request count, error rate
• Logs: discrete event records — exception stacks, audit logs, state changes
• Traces: cross-service request paths — which services a request passed, how long each hop took

Three shapes of “output” in control theory: Metrics continuous, Logs discrete events, Traces cross-subsystem causal chains. Prometheus, a mainstream implementation, periodically pulls metric endpoints exposed by services — continuous sampling of real state: the software SENSOR, same job as a thermostat thermometer, sampling hundreds or thousands of service-instance counters instead of room temperature.

Concrete example: a service exposes only “total requests” and “total errors” — you can infer “error rate,” but not “which user’s request failed” or “exact cause” — those dimensions are unobservable under the current design. Add Traces (full request paths) and structured Logs (error context) to make them observable again. Core observability work is expanding the observable state space so more of what you used to guess becomes something you measure.

5.2 Observability and Controllability Are Twins

Control theory has a neat duality: observability and controllability are mathematical transposes of each other — uncontrollable states are “places the controller can never reach”; unobservable states are “places outputs never reflect”; both are dead zones.

Engineering intuition: if you cannot observe a subsystem’s internal state, you cannot truly control it — at best you blind-operate on “last time this seemed to work,” not real feedback. That is why “observability first, automation second” is a common engineering order — before wiring automated correction, confirm critical states can be measured stably; otherwise automation is blind thrashing.

Same order in project management: before wiring “auto remind / auto escalate,” confirm the project’s critical state is recorded stably — if “what stage are we in” lives only in memory, any automation sits on sand.

5.3 Digital Twins: Approximating True State With a Virtual Model

Pushed far enough, observability becomes more active: not only passive output measurement, but maintaining a parallel virtual model, continuously correcting it with real measurements so it approximates internal true state — a digital twin. NASA introduced the idea in Apollo: a mathematical mirror of the spacecraft on the ground, synced with telemetry; engineers decide from the mirror rather than waiting for the craft itself to “tell” them what happened.

flowchart LR
    Real["real system"] -->|"live measurements"| Correct["correction"]
    Twin["virtual model (twin)"] -->|"model-based prediction"| Correct
    Correct -->|"fused optimal estimate"| Twin
    Twin -->|"for prediction / decisions"| Decision["engineering decisions"]

What is interesting about digital twins is not “build a simulator,” but an explicit math method balancing “model prediction” and “actual measurement” — Kalman filtering: each new measurement revises the model’s prior prediction, with strength depending on relative trust in this measurement vs the prior prediction. Software versions are usually much cruder — mostly “read logs + charts” — but the core ask is the same: from limited, noisy observations, reconstruct as precisely as possible what is really happening inside.

A lightweight software “digital twin” idea: keep an explicit record of expected state (e.g. a config describing “what this service should look like now”), continuously compare to live state; the diff itself is the most valuable signal — same desired/actual structure as Kubernetes Operators, extended from container orchestration to any place you need to know what the system is actually doing.

5.4 Digital Twin Is Not Monitoring, and Not Simulation — It Is Both Combined

Easy to confuse digital twin with monitoring and simulation. They are different; mixing them means digital-twin complexity for monitoring-level returns:

Monitoring says “how is it now”; simulation says “what if”; digital twin combines both and adds: continuously correct the virtual model with real data so simulation results get more trustworthy, then use those judgments to guide the next real-system actions — that is a closed-loop digital twin, not a pretty dashboard.

A shallow implementation is “read logs + charts” — monitoring, not twin: no continuously corrected model, no reverse “decide with this model” link. Valuable twin investment often happens when you need to run “what if we do this” on the virtual model before a high-risk decision, instead of gambling on the real system — an interesting contrast with section 6’s chaos engineering: chaos engineering runs controllable experiments on the real system; digital twins run risk-free experiments on the virtual model — both ask “where are this system’s boundaries,” with risk borne in different places.

---

6. Adaptation and Chaos Engineering: Actively Finding Stability Boundaries

Previous sections were “how to stay stable in normal operation.” This section flips the question: how do you know where a system’s stability boundary is? Waiting passively for failure never reveals true robustness — until production actually breaks.

6.1 Stability Is Not “Nothing Has Gone Wrong”

A common misunderstanding: a system that “never failed” equals “very stable.” Cybernetics is stricter: stability describes whether, after disturbance, the system can return near equilibrium — not whether it has been disturbed. A system whose robustness was never truly tested may have “never failed” only because it “never met a real disturbance,” not because it “can take a hit.”

That is why chaos engineering exists: if you do not actively find faults, faults find you — often at the worst time: traffic peaks, deep night with the thinnest on-call.

6.2 Netflix Chaos Monkey: Actively Injecting Disturbance

Netflix launched Chaos Monkey in 2011 — randomly terminate EC2 instances in production. Reason: hardware, network, and software failures will happen; better to create them actively and controllably, exposing assumptions that “should” hold but were never verified.

Netflix’s later Principles of Chaos Engineering give a standard flow:

flowchart LR
    A["1. Define steady state<br/>baseline metrics when healthy"] --> B["2. Hypothesize<br/>what metrics do after injection"]
    B --> C["3. Experiment in reality<br/>inject in prod (or high-fidelity env)"]
    C --> D["4. Limit blast radius<br/>small first, observe, then expand"]
    D --> E{"actual vs hypothesis"}
    E -->|"matches"| F["robustness validated"]
    E -->|"does not match"| G["found a previously unknown weak point"]

Quartet view: injected disturbance tests the whole feedback loop — can SENSOR spot anomalies in time, are COMPARATOR thresholds sane, is CONTROLLER logic correct, does ACTUATOR recovery actually work. Chaos engineering is a stress test of the whole control loop, not only one component.

6.3 Adaptive Control: Letting the System Adjust Its Own Parameters

Chaos engineering answers “where is the current stability boundary”; adaptive control answers “when the environment changes, how does the system adjust to stay stable.” Two common software patterns:

• MPC-style receding-horizon optimization: at each decision point, re-solve “optimal actions for the near future” from current information, execute only the first step, re-solve at the next point — auto-scaling in continuous deploy often works this way: not rigid fixed thresholds, but continuous load-trend prediction and dynamic scale amplitude.
• Online parameter identification: the system watches its own performance and retunes control parameters online instead of running fixed rules forever. Canary release “adjust next traffic share from prior-stage real behavior” is coarse online parameter ID.

Both contrast with robust control (design one parameter set that stays stable under a range of disturbances): robust control is “enumerate cases ahead, design one general parameter set”; adaptive control is “assume less ahead, adjust from live observation.” Which fits depends on how predictable disturbances are — if types and ranges can roughly be enumerated, robust control is simpler and more reliable; if the environment changes too fast and unpredictably, adaptive flexibility is more valuable, at the cost of complexity and “what if identification is wrong.”

6.4 Industrializing Chaos: From Manually Killing Instances to Platform Experiments

After Netflix, chaos engineering grew from an internal practice into a tool ecosystem; each step means “inject broader disturbance, or standardize the whole experiment flow”:

The path itself is a pattern: expand disturbance scope gradually, do not start with “whole datacenter power loss” — validate small recovery first, then expand — same idea as section 2’s canary “1% → 5% → 25% → full,” only here you control “how much fault risk dare we take,” not “whether to trust the new version.”

6.5 Adaptive Control Failure Modes: Not Every “Auto Adjustment” Is Reliable

Section 6.3 covered adaptive control’s upside; its failure modes need honesty too, or “let the system retune itself” gets overrated:

• Insufficient excitation: without diverse real scenes, adaptation lacks information to learn correct parameters — an auto-scale policy that only ever saw low traffic may have useless “experience” for never-seen patterns (e.g. flash sales).
• Parameter drift cannot keep up: if the environment changes faster than the adaptation learns, the system forever chases a past state — looks like it is adjusting, always lagging.
• Estimation failure under extreme disturbance: adaptation usually assumes disturbances vary smoothly in a reasonable range; sudden extremes (100× traffic spike) can break online estimators or even drive wrong reverse adjustments.

A real-world example (hardware, not software): some adaptive cruise systems had sensor calibration drift that left adaptation unable to track lead-car distance correctly, triggering mass recalls. Software translation: any adaptive mechanism needs a conservative fallback — when confidence is low, or observations far exceed training range, retreat to a known-safe fixed policy instead of trusting a possibly failed adaptive judgment. That is why SRE auto-scale usually has a hard resource ceiling — do not fully trust adaptive logic; give it a physical bound.

6.6 Division of Labor: Active Testing vs Passive Monitoring

Chaos and observability are often discussed together but divide labor differently: observability is passive — system runs normally, you measure; chaos is active — you deliberately create anomalies and see whether the feedback loop responds correctly. Without good observability, you cannot even see what an injected disturbance did; with only observability and no chaos, you know current state but not behavior under real pressure. Together they complete “system health checks” — one says how it is now, one says where the limit is.

---

7. Loop Engineering: 2026, the Same Structure Under a New Name

Previous sections covered relatively mature domains — CI/CD, microservices, SRE, observability, chaos — practices that have existed for one or two decades. This section is a fresh reenactment: AI agent systems are reinventing the same structure under a new name — Loop Engineering.

7.1 Why a New Word Now

In late 2024, Anthropic’s engineering post Building Effective Agents defined several agent workflow patterns, including evaluator-optimizer:

“one LLM call generates a response while another provides evaluation and feedback in a loop”^[11]

One LLM generates; another evaluates and feeds back; they iterate.

flowchart LR
    In["input task"] --> Gen["LLM generator<br/>Generator"]
    Gen -->|"candidate"| Eval["LLM evaluator<br/>Evaluator"]
    Eval -->|"reject + feedback"| Gen
    Eval -->|"pass"| Out["output"]

Swap “LLM” for “controller” and it reads like section 1’s quartet — the generator is the plant’s output end; the evaluator is COMPARATOR + SENSOR combined. Not coincidence: the same mathematical structure on a new substrate.

By 2025–2026, as long-running coding agents like Claude Code and Codex became practical, a new question surfaced: how to make agents not “need you to rewrite the prompt every time,” but fire like a cron job — self-trigger, decide whether to work, report when done. Some call that design pattern Loop Engineering — literally “design the loop,” but the core ask is what this post has been saying: wire the quartet explicitly into the agent system, instead of humans repeating manual ops forever.

7.2 Beyond Evaluator-Optimizer: Four More Ways to Build Agent Workflows

Anthropic’s post defined five common agent workflow patterns, not only evaluator-optimizer. The other four also map to earlier cybernetic concepts; together they clarify what Loop Engineering is assembling:

flowchart TB
    subgraph pc["Prompt Chaining"]
        direction LR
        PC1["step 1"] --> PC2["step 2"] --> PC3["step 3"]
    end
    subgraph rt["Routing"]
        direction TB
        RT0["input classifier"] --> RT1["specialized handler A"]
        RT0 --> RT2["specialized handler B"]
    end
    subgraph pl["Parallelization"]
        direction TB
        PL0["split task"] --> PL1["Agent A"]
        PL0 --> PL2["Agent B"]
        PL1 --> PL3["aggregate"]
        PL2 --> PL3
    end
    subgraph ow["Orchestrator-workers"]
        direction TB
        OW0["orchestrator"] --> OW1["worker 1"]
        OW0 --> OW2["worker 2"]
        OW1 --> OW0
        OW2 --> OW0
    end

Anthropic also has a often-skipped line: “Successfully building agents isn’t about building the most sophisticated system. It’s about building the right system for your needs.” Same spirit as the next section’s four-condition test: not every task needs evaluator-optimizer’s closed loop; many scenes only need simple prompt chaining — the five patterns are not ranked; they match different feedback needs. Same judgment as section 2’s “should this project get heavy CI/CD”: structural complexity should match problem complexity, not blindly chase the cleverest design.

7.3 Four-Condition Test: Is This Worth Turning Into a Loop

Before building an automatic loop, a repeatedly cited checklist — all must pass:

flowchart TD
    Q1{"Does the task recur?"} -->|"no"| Manual["stay manual; do not build a Loop"]
    Q1 -->|"yes"| Q2{"Is there objective automated verification?"}
    Q2 -->|"no"| Manual
    Q2 -->|"yes"| Q3{"Can budget absorb waste?"}
    Q3 -->|"no"| Manual
    Q3 -->|"yes"| Q4{"Are the agent's tools complete?"}
    Q4 -->|"incomplete"| Fix["fill tools first, then consider a Loop"]
    Q4 -->|"complete"| Build["worth building a Loop"]

• Does this task recur? One-offs are not worth a Loop; doing it once by hand is cheaper than a full automation stack.
• Is there objective automated verification? If “done correctly?” itself needs human review, you cannot automate the checker; the Loop’s core value — unattended cycling — is discounted.
• How much waste can budget absorb? Agents err, wander, burn unexpected resources; that cost must be acceptable.
• Are the agent’s tools complete? Missing tools (cannot run tests, cannot read logs) tank judgment quality; spinning the loop is pointless.

All four met → worth building the Loop; any missing → stay manual or fill the gap first, do not rush automation. Same logic as “should this project get a heavy CI/CD pipeline” — if scale is not there, infrastructure maintenance exceeds benefits.

7.4 Six Components: What a Sustainable Loop Needs

Addy Osmani, engineering productivity lead on Google Chrome, broke the pattern into six components in a mid-2026 long post^[12]:

flowchart TB
    subgraph loop["One full Loop cycle"]
        direction LR
        Auto["Automations<br/>heartbeat"] --> Find["find work"]
        Find --> Hand["hand to Agent"]
        Hand --> Check["check results"]
        Check --> Record["record what happened"]
        Record --> Decide["decide next"]
        Decide -.->|"next round"| Auto
    end
    Skills["Skills<br/>project knowledge"] -.->|"read each round"| Hand
    Conn["Connectors/MCP<br/>external tools"] -.->|"call each round"| Hand
    Sub["Sub-agents<br/>maker/checker split"] -.->|"independent verify"| Check
    State["State<br/>persisted state files"] -.->|"survive across rounds"| Record

Each of the six maps to earlier sections:

• Automations ↔ section 2 event triggers (on: push / on: schedule), target is Agent instead of CI.
• Worktrees ↔ section 3 decoupling — parallel agents do not interfere via independent workspaces (usually git worktrees); same idea as microservices isolating faults with independent deploy units.
• Skills ↔ concrete encoding of organizational-layer experience in Qian’s hierarchy — project knowledge that needed repeated human explanation becomes a reusable file.
• Connectors / MCP ↔ the interface layer from observability/controllability: the system must touch the outside world to measure and act.
• Sub-agents ↔ evaluator-optimizer — checker and checked separated.
• State ↔ persisted state from the digital-twin discussion, only here it is the agent’s working memory, not the plant’s physical state.

Addy Osmani on State: “The agent forgets, the repo does not” — each run ends and forgets everything; only files on disk survive across rounds. Not a defect of agent systems but a constraint of how they run — which is why “where state lives, when to write, when to read” becomes the core design problem, not an implementation detail.

7.5 The Ralph Wiggum Loop: Simplest Trap, Simplest Lesson

Before “Loop Engineering” trended, engineer Geoffrey Huntley wrote an influential mid-2025 post describing a loop so minimal it is almost absurd^[13]:

while :; do cat PROMPT.md | claude-code ; done

One shell line — keep feeding the same prompt to a coding agent; it decides what to do, finishes, rereads the prompt, continues. No fancy state machine, no complex orchestration; all “memory” lives in PROMPT.md and a few side files on disk:

flowchart LR
    A["read PROMPT.md"] --> B["Agent works"]
    B --> C["write progress back to disk"]
    C --> A

Huntley named it Ralph — after the naive Simpsons character who sometimes gets things right by accident — hinting the method “deterministically performs poorly, but in an uncertain world” can be steadier than clever design.

Ralph also exposes a failure mode later discussed as the Ralph Wiggum failure mode: the agent prematurely judges itself “done,” signals completion halfway — failure happens, but silently.

flowchart TD
    A["Agent starts a round"] --> B["Agent judges: I think I'm done"]
    B --> C{"Actually done?"}
    C -->|"not really"| D["loop exits early<br/>failed, nobody knows"]
    C -->|"yes"| E["loop ends normally"]

Root cause, in quartet language, is clear: COMPARATOR and CONTROLLER are the same agent — it both does the work and judges “is the work done.” Self-scoring naturally overestimates completion — same as RLHF self-preferential bias: judge and judged cannot be the same individual, or judgment itself loses meaning.

That is why the six components include “Sub-agents: maker/checker split.” In a robust Loop, “done” must be judged by an independent role with clear verification standards — ideally an automated test, linter, or a separately assigned checker agent with a different model or context — not the working agent saying “I’m done.” The “evaluator” in Anthropic’s evaluator-optimizer name is exactly that independence.

7.6 A Concrete Loop Design Sketch

Put the principles together and a relatively robust Loop looks like this:

flowchart TD
    Trigger["scheduled trigger (Automations)"] --> Gather["independent script gathers state<br/>no LLM reasoning"]
    Gather --> NeedWork{"anything to handle?"}
    NeedWork -->|"no"| Silent["silent exit; disturb nobody"]
    NeedWork -->|"yes"| Agent["LLM Agent reads Skill + handles task"]
    Agent --> Verify["independent verification<br/>script asserts or separate Agent review"]
    Verify -->|"pass"| WriteState["write state files (State)"]
    Verify -->|"fail"| Escalate["escalate to human (HITL)"]
    WriteState --> Notify["notify by priority"]

Design choices worth stressing:

• Gather state and execute tasks separately — the first step needs no LLM; pure scripts answer “is there work,” so high-frequency cron is free silent exit when idle, not wasting tokens asking “anything new?”
• Verification independent of execution — the maker/checker split from 7.5.
• Verification failure escalates, does not auto-retry until success — agents retrying the same task they cannot do usually do not find the real issue; they burn more resources the wrong way. Exposing those cases to humans is safer than letting the system deadlock itself.

---

8. Many Systems Together: Distributed Coordination and Consistency

Most earlier sections discussed control structure inside one system; the real world rarely has one isolated system — a dozen projects on hand are already “many independent systems that need coordination,” as are multi-tenant cloud architectures. This section is about “when many independent control loops gather, how they avoid dragging each other down.”

8.1 Multi-Tenant SaaS: Many Independent Control Loops + Shared Infrastructure

A cloud service serves hundreds or thousands of customers (tenants). Each tenant should theoretically be an independent controlled system — loads, data, resource use should not interfere. In practice tenants inevitably share infrastructure (DB connection pools, compute clusters, bandwidth), creating a classic distributed-control problem: multiple theoretically independent control loops gain hidden coupling through shared resources.

Industry’s standard response is a dial for isolation degree:

Same principle as section 3’s “coupling strength decides independent control,” with “microservice modules” replaced by “tenants.” When coupling is weak enough, each tenant can be managed as an independent system — anomaly detection and scaling do not interfere; once coupling strengthens (shared DB suddenly slows), all tenants suffer together — root of many SaaS “avalanche” failures: coupling looks weak in normal times; when a shared component fails, hidden coupling surfaces and one tenant’s problem becomes everyone’s.

8.2 Consistency: How Multiple Nodes Reach the Same Judgment

Another classic distributed problem: if the same setpoint must sync to many independently running nodes (e.g. push a config update to all servers), how do all nodes eventually converge to the same state? The general solution is consensus protocols — nodes periodically exchange information, gradually shrink state differences until they align.

That convergence can be tracked with a simple metric: the sum of pairwise state differences across nodes. Larger means more inconsistency; each well-designed communication round should shrink it toward zero — same idea as a Lyapunov-style stability criterion in section 1: if you can find a “degree of inconsistency” metric that decreases monotonically over time, you can prove the system converges to agreement.

Managing many projects: if several project groups must keep the same process norms, “send one notice and hope everyone syncs” is unrealistic — a more reliable approach looks like consensus: regularly check each group’s actual practice vs the norm, actively sync gaps, instead of assuming sent information is fully absorbed.

8.3 Emergent Behavior: Local Rules Stack Into Global Effects — Good and Bad

Late in life Qian proposed “open complex giant systems” for huge systems with many autonomous decision-makers (e.g. people) continuously interacting with the environment — economies, ecosystems, large social organizations. A core trait: global macro behavior is not designed by a center; it emerges from many local individuals’ simple decision rules.

A multi-tenant SaaS platform’s overall utilization and failure rate are also “emergent” — no central controller directly sets “today’s platform error rate”; the number is the superposition of thousands of tenants’ independent behaviors. Engineering implication: to change macro behavior, sometimes commanding at the macro level does nothing; you adjust local rules, because macro behavior grows from local rules — it is not a knob you can twist alone.

Managing a dozen projects feels similar: if “overall delivery rhythm stays poor,” shouting “everyone go faster” globally often does little — effective interventions usually happen locally: fix one approval step, correct one team’s communication. Global health emerges from local details; it is not an independent variable you can drive directly.

---

9. My Hermes Project-Management Workflow Framework

Previous sections were other people’s systems — Kubernetes, Meta, Anthropic, Netflix. This one is mine: the automation workflow for dozens of parallel projects, grounded on Linear (single source of truth for issue state) + GitHub (single source of truth for code and PRs), with a middle layer of custom Python scripts and a scheduled polling system designed around the quartet. It recently got a structural rewrite — not patches, but mapping “where it is weak” into concrete code changes. This section is what landed after the rewrite, not a wish list.

9.1 System Runtime Path

I will not expand every detail; combined with software engineering and agile practice, most workflow designs look similar.

Managing dozens of projects is managing dozens of “idea to production” pipelines. Each concrete task maps to a Linear issue that moves through states:

flowchart LR
    A["Backlog<br/>ideas to evaluate"] --> B["Todo<br/>scheduled, ready to start"]
    B --> C["In Progress<br/>in development"]
    C --> D["In Review<br/>PR open, waiting confirmation"]
    D --> E["Done<br/>finished and accepted"]

Who advances an issue has two paths: human path — I develop locally and flip issue state myself; AI auto path — the whole development run is handed to an AI agent; I only nod at critical nodes (plan confirmation, merge). The switch is an agent:auto label — that label means “automation owns this task”; the system actively advances it into development by preset rules without me saying “start.”

agent:auto tasks have a hard rule: when assigned, the issue description must include an “Acceptance Criteria” block — checklist items that can be mechanically verified (“file X exists, tests Y all pass, endpoint Z returns expected values”), not subjective “good code quality.” Reason: after AI finishes, verifying “done” must also be automated; the basis must be fixed, executable conditions, not fuzzy adjectives.

After AI commits code, the label flips from agent:auto to agent:pending — “AI thinks it is done, but no independent verification yet; do not mark Done.” agent:pending is the entry condition for the next section’s Acceptance Gate: seeing that label, the system runs acceptance against the criteria checklist and only then advances to Done.

flowchart LR
    A["Todo"] -->|"label agent:auto<br/>+ write acceptance criteria"| B["In Progress<br/>AI developing"]
    B -->|"commits done<br/>agent:auto → agent:pending"| C["wait for Acceptance Gate"]
    C -->|"pass"| D["Done"]
    C -->|"fail"| E["auto-fix retry<br/>(capped)"]
    E -->|"still fails"| F["hand to human"]

With that label semantics as background, “how the architecture runs” and “how the Acceptance Gate was rebuilt” stop feeling abrupt.

9.2 Lean Architecture: No Webhooks — Polling + Diff Simulates Event-Driven

The system does not wire Linear or GitHub webhooks. A job every 30 minutes pulls full Linear issue state and open GitHub PRs, diffs against the last snapshot, translates “what changed” into typed events (status changed, labels changed, issue touched, new PR opened, acceptance due, acceptance blocked…), and writes a Dashboard snapshot.

flowchart TD
    A["scheduled poll (every 30 min)"] --> B["pull full Linear issues + open GitHub PRs"]
    B --> C["diff vs last snapshot"]
    C --> D["emit typed events<br/>status_changed / label_changed / pr_opened /<br/>acceptance_due / acceptance_blocked / stale_no_pr / triage_wait"]
    D --> E["write Dashboard snapshot"]
    D --> F{"event triggers signal detection?"}
    F -->|"yes"| G["write action_tracker; dispatch by signal type"]
    F -->|"no"| E

“No webhooks; periodic pull + diff” is a plain engineering call: true event-driven needs the other system to push, at higher cost; poll + diff trades sampling frequency for that dependency — if sampling is fast enough, the effect is not essentially different from event-driven — same class of trade-off as section 2 compressing release frequency from monthly to minutes, in another scene.

Back to the quartet:

9.3 Three-State Judgment: An Easy-to-Miss SENSOR Trap

Early signal detection was intuitive: “check if this signal is still there; if not, mark resolved.” Hidden pit: “not found” and “query failed” are completely different, but look the same on the surface. If GitHub API times out, a real stale-commit signal can be misread as “resolved” when it was never successfully queried — unrelated to resolution.

Detection now returns three states, not a boolean:

# Return contract (three-state):
#   (signals: list[dict], ok: bool)
#   ok=True  + signals=[]  → confirmed no signal (safe to auto_resolve)
#   ok=True  + signals=[…] → signal still present
#   ok=False + signals=[]  → this round's query failed; conclusion unknown; must not auto_resolve

Recheck logic: only ok=True and the signal truly absent this round may mark “resolved”; ok=False (query failed) skips and leaves state unchanged — never false-positive; if the signal remains, existing “overdue unresolved” escalation applies.

flowchart TD
    A["recheck a pending signal"] --> B{"this round's result"}
    B -->|"ok=True, signal gone"| C["confirm resolved<br/>mark resolved"]
    B -->|"ok=True, signal still there"| D{"past verify_by?"}
    D -->|"yes"| E["mark stale, remind again"]
    D -->|"no"| F["keep pending for next round"]
    B -->|"ok=False, query failed"| G["skip; do not change state<br/>avoid false positives"]

This change maps directly to section 5’s observability principle: “no anomaly signal received” ≠ “system healthy” — it may only mean measurement itself failed. Confusing the two disguises SENSOR reliability problems as real health — the observability pit easiest to step in and hardest to spot, because it only shows in the window when SENSOR happens to fail.

9.4 Acceptance Gate: Ask “Is There an Objective Standard” Before Running Acceptance

As above, agent:pending means “AI thinks done, waiting acceptance.” The old problem: acceptance assumed “criteria exist.” If assigners forgot clear, checkable criteria, acceptance still ran and produced a fuzzy-guess judgment — opposite of 7.5’s maker/checker spirit: if the checker’s basis is shaky, the result is not trustworthy.

The Acceptance Gate now has a front BLOCK: before running acceptance, check whether the issue description has an explicit ## Acceptance Criteria block. If not, refuse, notify to add it first — do not force an unreliable conclusion.

flowchart TD
    A["Heartbeat sees agent:pending"] --> B{"description has<br/>## Acceptance Criteria block?"}
    B -->|"no"| C["BLOCK<br/>notify, write action_tracker<br/>stop; do not run acceptance"]
    B -->|"yes"| D{"acceptance report already exists?"}
    D -->|"yes"| E["skip"]
    D -->|"no"| F{"retries < 3?"}
    F -->|"yes"| G["run Acceptance Gate; retry count +1"]
    F -->|"no, at cap"| H["stop auto-retry<br/>request human intervention"]

Second change: cap auto-fix retries after acceptance failure — 3 chances; after the third failure, stop automation and hand the problem back to humans as-is. Most direct engineering defense against the Ralph Wiggum failure mode (7.5): a loop without a built-in stop condition is the root of runaway; an explicit retry cap strictly bounds the cost, instead of letting a stuck acceptance spin forever, burning resources and solving nothing.

9.5 Health Score: Collapse Many Fine Signals Into One Number You Can Scan

The system tracks six signal classes (stale commit, stale issue, triage backlog, acceptance fail, reopened issue, too many divergence signals in one sprint), each with its own thresholds and actions. Many signals → reading them one by one burns attention that should go to real problems — a shrink of this post’s opening “manage dozens of projects,” only now managing “dozens of signals.”

The fix: fold them into a 1–10 health score plus a trend vs last snapshot (improving / flat / worsening), storing a history row each run:

Day to day: look at the score and trend first — score down, trend worsening → dig which project and which signal class; score fine → do not open the detail table. Plain state compression: not section 5’s strict digital twin (no state-transition model, no prediction), but similar role — a low-dimensional quantity replaces reading high-dimensional raw signals, turning “should I intervene now” from “read every detail” into “glance at the score.” Thresholds can be tuned to project pace and the operator’s energy; keeping Human in the Loop means honestly assessing your own energy management — human health matters too.

9.6 What Was Not Expanded: How the Other Roles Run Now

The last two subsections focused on the densest changes — three-state judgment and Acceptance Gate. Brief status on the rest so the whole system is visible.

SENSOR side: six predefined signal classes — stale commit, stale issue status, triage backlog, acceptance fail, reopened issue, too many sprint divergence signals. Each has its own detection function, maintained in one signal-definition table — adding a class means adding a row, not changing the detection framework.

COMPARATOR side: thresholds are global static numbers (3 days no commit → remind; 7 days status unchanged → remind), not per-project, not history-adaptive. Intentional simplification: current project scale is still in “one standard covers most scenes”; when scale truly needs per-project dynamic thresholds, add complexity then — doing it early only raises maintenance without matching return.

ACTUATOR side: all notifications still share one Telegram stream; routine reports and decisions that need attention mix, with weight conveyed by wording (markers, “needs your decision”) rather than physically separate priority channels. Against section 4.2’s Sheridan levels, this is the most visible shortfall — no explicit channel split between “routine supervision” and “worth escalating”; a trivial daily report and a true emergency still arrive the same way.

9.7 The System Evolves — and Human Judgment Evolves With It

Looking back, three-state judgment and the Acceptance Gate were not designed in a vacuum. After the system ran for a while, a concrete false positive or an acceptance that should not have passed was found; we traced back to a specific role defect and added a mechanism. Same cycle as section 4’s SRE on-call “postmortem → update Runbook → sediment into automation,” only on my own system rather than someone else’s.

More precisely: this is not “one-shot perfection of the system,” but continuous mutual calibration between the system and me as the organizational layer. Each mechanism added means one fewer thing I must personally watch — after three-state judgment, I no longer wonder whether an auto-resolved signal was a false positive from a flaky API; after the Acceptance Gate BLOCK, I no longer worry AI is self-accepting against unclear criteria. Conversely, each new weak point I find — like missing notification priority — is slowly turning the tacit “should this message interrupt me” standard in my head into a rule that can be written into code.

That closes the layered intuition from the opening: not “design layering once and relax forever,” but the direct layer covers more and more; attention freed at the organizational layer turns to finding the next blind spot the direct layer still cannot cover — the boundary is not drawn once; it inches forward as the system runs round after round. The quartet’s value finally shows in that continuous calibration: every fuzzy “this feels awkward” can be translated precisely into “which role is missing what”; after the patch, where to patch next is fully symmetric with managing a dozen projects at the start — “feels off” is never reliable, whether you manage projects or the system that manages projects.

---

10. Closing: When This Framework Is Worth It, When a Simple Rule Prompt Is Enough

Many domains — CI/CD, microservices, SRE, observability, chaos, AI agents, my own workflow — each got the same quartet frame. The frame is not universal; wrong context is over-engineering. Back to the opening — managing dozens of projects, when do you need this whole way of thinking, and when is a simple rule prompt enough?

flowchart TD
    subgraph freq["Q1: Worth the full quartet?"]
        Q1{"Is there a<br/>continuous feedback loop?"}
        Q1 -->|"no<br/>one-off / single batch"| A1["classical cybernetics does not apply<br/>a simple checklist is enough"]
        Q1 -->|"yes"| Q2{"How high is feedback frequency?"}
        Q2 -->|"high<br/>100+/day, e.g. continuous deploy"| A2["worth designing the full quartet<br/>explicit SENSOR / COMPARATOR /<br/>CONTROLLER / ACTUATOR"]
        Q2 -->|"low<br/>a few times/day"| A3["analogy + simple rules enough<br/>no heavy gear"]
    end

    subgraph obs["Q2: Can automation advance safely?"]
        Q3{"Can critical state<br/>be observed stably?"}
        Q3 -->|"no"| B1["observability first, then automation<br/>blind ops only create new uncertainty"]
        Q3 -->|"yes"| Q4{"Have you validated<br/>stability boundaries?"}
        Q4 -->|"not validated<br/>never failed ≠ stable"| B2["consider active testing<br/>small, controlled anomaly injection"]
        Q4 -->|"validated"| B3["safe to push automation further"]
    end

    subgraph agent["Q3: Loop Engineering?"]
        Q5{"Does an AI Agent<br/>decide autonomously?"}
        Q5 -->|"no"| C1["classical layered control<br/>scripts + human review"]
        Q5 -->|"yes"| Q6{"Need independent<br/>completion verification?"}
        Q6 -->|"yes<br/>clear right/wrong criteria"| C2["Loop Engineering<br/>maker/checker split<br/>do not let the Agent score itself"]
        Q6 -->|"no<br/>subjective judgment"| C3["keep human review<br/>do not force automated verification"]
    end

    subgraph human["Q4: Is the human interface healthy?"]
        Q7{"Are HITL channels<br/>priority-tiered?"}
        Q7 -->|"yes<br/>routine vs urgent<br/>different priorities"| D1["organizational attention<br/>effectively protected"]
        Q7 -->|"no<br/>all notices in one channel"| D2["fix this first<br/>or more automation<br/>still drowns the org layer in noise"]
    end

This decision tree is also the core judgment the post wants to convey: engineering cybernetics is not a specific tool; it is a way of checking system health — when “the system will not obey,” do not rush to redesign the whole process; ask which role in the quartet is missing or broken; when “should this process be automated,” ask whether feedback frequency is high and whether objective verification exists; when “I keep drowning in notifications,” ask whether the human interface is priority-tiered.

Managing a dozen projects and managing a continuous-deploy pipeline look unrelated; opened up, they are different faces of the same problem: how a complex system stays stable and moves toward a goal without every detail needing hands-on work. Qian Xuesen’s 1954 answer — layering, feedback, humans at critical points — seventy years later is still the most concise language for that question, and now AI agent systems are verifying it again.

---

References

[1] Qian, Xuesen. Engineering Cybernetics. McGraw-Hill, 1954. archive.org/details/engineeringcyber0000qian

[2] Wiener, Norbert. Cybernetics: Or Control and Communication in the Animal and the Machine. MIT Press, 1948. monoskop.org PDF

[3] GitHub Docs. “Events that trigger workflows.” docs.github.com

[4] Rossi, Chuck. “Rapid release at massive scale.” Meta Engineering Blog, 2017-08-31. engineering.fb.com

[5] Kubernetes Documentation. “Operator pattern.” kubernetes.io

[6] Sheridan, Thomas B. Telerobotics, Automation, and Human Supervisory Control. MIT Press, 1992. mitpress.mit.edu

[7] Stiennon, Nisan, et al. “Learning to Summarize from Human Feedback.” 2020. arxiv.org/abs/2009.01325

[8] Ouyang, Long, et al. “Training Language Models to Follow Instructions with Human Feedback.” 2022. arxiv.org/abs/2203.02155

[9] Casper, Stephen, et al. “Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback.” 2023. arxiv.org/abs/2307.15217

[10] Bai, Yuntao, et al. “Constitutional AI: Harmlessness from AI Feedback.” Anthropic, 2022. arxiv.org/abs/2212.08073

[11] Schluntz, Erik & Zhang, Barry. “Building Effective Agents.” Anthropic Engineering Blog, 2024-12-19. anthropic.com/engineering/building-effective-agents

[12] Osmani, Addy. “Loop Engineering.” 2026-06-07. addyosmani.com/blog/loop-engineering

[13] Huntley, Geoffrey. “Ralph Wiggum as a software engineer.” 2025-07-14. ghuntley.com/ralph

[14] Irving, Geoffrey, et al. “AI Safety via Debate.” OpenAI, 2018. arxiv.org/abs/1805.00899