Field notes from an autonomous engineering fleet

A single operator built privateos, an orchestrator that lets AI agents write production code, in 29 days and put it to work on four production repositories, where its agents authored 75 pull requests across a 28-day operation window. We analyze the complete GitHub record of both the construction (376 commits, 285 pull requests and 310 issues) and the operation, and verify every reported failure against a citable artifact: an eleven-entry incident registry checked by code against archived snapshots. The central operational finding is that enforced structure outperforms prompted instruction. On May 5, 2026 the orchestrator began building each pull request’s description in code instead of asking the model to follow a template. The share of agent pull requests carrying a summary and an issue link rose from 1 of 4 before the change to 71 of 71 after it (one-sided Fisher exact p = 5.9 × 10⁻⁵). The median pull request also shrank, from 9 changed files to 3. Merge rate depends on the kind of work: 92 percent on a well-specified trading backend, 35 percent on a design-led website. Failures cluster into recurring families: scope indiscipline, self-triggering loops, silent capability loss, liveness misjudgment and isolation gaps. The main limitations are a pre-contract era of only four pull requests and the loss of runtime telemetry, which restricts the analysis to the GitHub record.

Between April 29 and May 27, 2026, a single operator built an orchestration system that lets AI agents write production code without a human in the loop, and the agents then wrote some. The system, privateos, uses a GitHub Projects board as its scheduler. Moving a card between columns dispatches one of sixteen personas (manager, engineer, worker, reviewer, fixer, triage and others) as a database-backed job, which runs once however many times it is triggered. The dispatched agent checks out a working copy of the target repository, makes its changes, and pushes a branch and a pull request back. Four repositories were enrolled as workspaces during the study window: a Polymarket trading fleet (poly), a generative audio project (market-ambience), the research site that publishes this paper (atol-research), and a smoke-test sandbox. From May 5 the fleet also worked on its own orchestrator: agents wrote 22 of the 285 pull requests that built privateos, and the analysis keeps those out of the operational record (§4.3).

Vendor demonstrations and benchmark suites dominate the public evidence on AI coding agents, and both share a selection problem: the work is chosen to be completable. The record studied here is the opposite: everything the fleet did. It includes the pull request that deleted 565 lines it was never asked to touch, the fixer persona that triggered itself in a loop, and the kill switch that updated a database row while the process it named kept running.

The contribution of this paper is an archival analysis of that complete record, with every failure claim checked against a concrete GitHub artifact and every number reproduced by a dependency-free notebook. §3 places the work against benchmark evaluations and the honor-system result it extends, and §4 documents the system, the snapshots, the metrics and the verification method. §5 reports the construction record, the operational record, the before-and-after effect of the PR-body contract, and the incident registry. §6 interprets the failure families and the one intervention that measurably worked, §7 sets out the threats to inference, and §8 closes.

Benchmark evaluations measure AI coding agents on curated issue sets. SWE-bench reports resolve rates on 2,294 GitHub issues drawn from twelve Python repositories (Jimenez et al., 2024), and SWE-agent established the agent-computer-interface framing on the same suite (Yang et al., 2024). Field evidence on AI-assisted development is thinner and concentrates on human-in-the-loop pair programming, where Peng et al. (2023) measured a 55.8 percent speedup on a single controlled task. Operational records of unattended fleets, where the agent owns the branch, the pull request and the merge-queue interaction end to end, are scarce in the published record. The record studied here is one such record, at single-operator scale, with the limits that scale implies (§7).

The analytical lens extends MKT-002 (Atol Research, 2026), which documented how an AI-coordinated trading operation reproduced the Knight Capital failure mode: rules that existed in documents and were enforced by nothing. MKT-002 argued that the distinction between a guidance layer (what the system is told) and an enforcement layer (what the system can do) is categorical. The fleet’s record measures that distinction on a new surface, because a pull-request quality rule moved from the guidance layer to the enforcement layer partway through the window.

Two systems-engineering literatures bear on the failure families in §5. Making job dispatch safe when a message may be delivered more than once is a known hard problem with known patterns (Helland, 2012), and privateos converged on the same patterns through operational incidents rather than through design. Liveness detection by heartbeat, and the false-positive reaping that follows from aggressive thresholds, is standard material in site-reliability engineering (Beyer et al., 2016); the fleet relearned it across a 14-day incident arc.

4.1 System under study

privateos is a TypeScript application (Hono server, React dock UI, Postgres 16, pg-boss job queue) that connects a GitHub App to a fleet of Claude-based personas. Webhook deliveries for board-column changes enter a dispatch registry that maps columns to roles. Each dispatch attempts to claim a run row under a transaction-scoped advisory lock, so that webhook redeliveries, concurrent column moves and operator retries cannot create duplicate work. The claim protocol is the system’s load-bearing concurrency primitive:

// source: privateos/src/server/claim-run.ts @ 44a0a64
await client.query("BEGIN");
await client.query(`SELECT pg_advisory_xact_lock(hashtext($1))`, [
  opts.lockKey,
]);

Worker-class personas run as spawned subprocesses against git worktrees of the target repository and report heartbeats; in-process personas run inside the job handler. A janitor reaps (stops and closes) runs whose heartbeats go stale, a reconciler re-enqueues stalled chains, and per-workspace daily budget gates can defer runs. The repository at the studied commit contains 100,863 lines of TypeScript under src/, a further 88,454 under tests/, and 66 SQL migrations.

4.2 Data

All data are fixed snapshots fetched on June 11, 2026 and archived under data/raw/. They are the full pull-request and issue records of privateos (raw-privateos-prs, 285 rows; raw-privateos-issues, 310 rows), the full PR records of the four workspace repositories (raw-poly-prs, 197; raw-atol-research-prs, 49; raw-market-ambience-prs, 41; raw-privateos-sandbox-prs, 1), the git history of the privateos main branch at commit 44a0a64 in numstat form (raw-privateos-git-numstat, 376 commits), and the listing of the postmortem directory (raw-privateos-learnings-files, 29 files). Dataset identifiers here and below refer to data/manifest.yaml.

The fleet’s runtime Postgres held zero rows in its runs, events and webhook_log tables at the snapshot date, following a database reset after the operation window. Runtime telemetry (per-run cost, tokens, latency) is therefore unavailable, and the analysis is restricted to artifacts the fleet left in GitHub: pull requests, issues, commits, and documents committed to the repositories.

4.3 Agent-PR identification and quality metrics

A pull request is classified as agent-authored when its head branch carries the agent/ prefix or its body contains the run.id stamp the runner writes on creation. The rule yields 75 PRs across the four workspaces (agent-prs). Four structure flags are computed from each PR: a ## Summary section, an issue linkage line (Part of #, Closes # or Fixes #), a ## Scope section, and whether the head branch is a raw GitHub node id rather than a readable slug.

The window splits at May 5, 2026, the day a root-cause analysis of poly#191 (an agent PR rejected in human review) produced the PR-body contract. Under the contract, the body template, the issue linkage and the scope statement are assembled in code from data the system already holds. The design note says this must happen “not by asking the model to remember to write it.” PRs created before May 6 form the before era (n = 4) and the rest the after era (n = 71). The split is observational and the analysis is tagged [post-hoc] throughout; no hypothesis was registered before the data existed.

Applying the same identification rule to the privateos repository itself yields a further 22 agent-authored pull requests, all merged, created from May 5 onward: from the contract day on, the fleet also worked on its own orchestrator. These 22 PRs are excluded from the 75-PR operational record so that the construction and operation surfaces stay distinct; §7 returns to this boundary. The one structure-compliant before-era PR (poly#192) was created hours after the root-cause analysis on the contract day itself, so the era boundary is conservative for the measured contrast.

4.4 Incident registry verification

Eleven incident claims were collected from the fleet’s own engineering record (commit messages, issue threads, and the May 5, 2026 state-of-the-app document in the privateos repository) and checked against the snapshots. For each claim, the notebook asserts that the named artifact exists, that its title contains the expected phrase, and that its state and dates match the narrative; the notebook fails if any assertion drifts from the record (incident-registry). The nine merged rollout PRs of the claimRun consolidation (epic #196) are verified the same way.

4.5 Statistical approach

Era contrasts on binary flags use the one-sided Fisher exact test, computed as the hypergeometric tail probability of observing at most $s_1$ successes in the before era under a shared rate:

where $N$ is the total PR count, $K$ the total success count, and $n_1$ the before-era count. With $n_1 = 4$, only near-categorical shifts can reach significance, which is the appropriate level of caution for this sample.

4.6 Software environment

Python 3.12, standard library only; no third-party packages and no network access. The analysis loads the archived snapshots, rebuilds every derived dataset, runs the verification checks and draws the four figures in under ten seconds on commodity hardware.

5.1 Construction record

Building privateos took 29 days, and 376 commits reached its main branch on 27 of them (Fig. 1). Over the same window the repository took 285 pull requests, of which 284 merged, and 310 issues. More of the added lines went into tests than into anything else, 91,382 under tests/ against 62,902 for the server itself, so the builder spent more of the diff on verifying behavior than on implementing it. Incidents kept arriving to the end: the last verified one, the janitor threshold fix of May 26, landed one day before the window’s final commit.

5.2 Operational record

Across the four workspaces, agents opened 75 pull requests between April 29 and May 26, and 56 of them merged, three in four. On the well-specified repositories agents merged almost all their work, and on the design-led website about a third (Fig. 2). On poly, whose issues carried file-level specifications inherited from epic breakdowns, 36 of 39 agent pull requests merged (92 percent), and on market-ambience all 12 did. On atol-research, where the work was design-led, 8 of 23 merged (35 percent); 14 of the rest were still open at the snapshot date and one was closed unmerged. The sandbox’s single pull request did not merge.

5.3 The PR-body contract

After the May 5 contract, every agent pull request carried a summary and an issue link, 71 of 71, against 1 of the 4 before it (Fig. 3) [post-hoc]. All but two, 69 of 71, also carried an explicit scope section. If the contract had made no difference, a split as lopsided as either the summary’s or the issue link’s would arise by chance with a probability of 5.9 × 10⁻⁵, about one in 17,000 (one-sided Fisher exact test, Eq. 1). Branches named with a raw node id went from 3 of 4 to none of 71.

The typical pull request also got smaller. Its blast radius, the number of files it changes, fell from a median of 9 before the contract to 3 after it (Fig. 4). Most of the 71 later pull requests were small, with 51 changing four files or fewer. Three still changed seventeen or more, and the largest, at 30 files, topped the before era’s largest of 26.

The pull request that prompted the contract was poly#191, opened on May 5 and closed unmerged. Titled as a narrow configuration fix, it touched 15 files, adding 53 lines and deleting 565, including client and redeemer modules it was never asked to modify. Its body read (no summary) above the run-id stamp, and its branch was named agent/I_kwDORxYP-s8AAAABA1Z61A. Merges rose after the contract too, from 2 of 4 to 54 of 71, but at this sample size the change does not reach significance (p = 0.26) [post-hoc].

5.4 Incident registry

Table 1 lists the eleven verified incident artifacts; the dispatch loop they interrupt is the one sketched in Figure 5. Every failure family that produced a control subsystem is represented: the scope explosion preceded the PR-body contract, the fixer self-loop preceded review-driven triggering (Fig. 6), the duplicate deferred rows preceded the dedup fix, and the ineffective kill preceded the cooperative-cancellation work.

Table 1. Verified incident registry. Each row is asserted by the notebook against the archived snapshots; dates are artifact creation dates (incident-registry).

The liveness arc alone spans 14 days, from the discovery that the kill mechanism could not kill (privateos#362) to the threshold rebalance that stopped the janitor from reaping healthy runs (privateos#613) (Fig. 7). The fleet also kept a written memory: 29 postmortems, each on a single class of bug, accumulated in the repository’s learnings directory across eight categories, led by architecture patterns (9) and conventions (7).

The contract result isolates a mechanism, within the limits of an observational split. Before May 5, pull-request structure was a guidance rule: the worker persona was expected to write a summary, link the issue and respect scope. After it, structure became an enforcement property. The orchestrator assembles the body from the issue node id, the epic linkage and the spec file list it already holds, so the rule no longer depends on the model remembering it. The shift moved compliance from 1 of 4 to 71 of 71 (§5.3) [post-hoc]. MKT-002 (Atol Research, 2026) argued from a single anchoring incident that guidance-layer rules fail however prominently they are stated; the present record adds a measured before-and-after on a production surface, and the direction matches.

Failures arrived in families rather than as one-off events, and the families ordered the architecture. Every control subsystem in privateos has a verifiable incident that preceded it (Tab. 1). The body contract followed the scope explosion, and review-driven triggering followed the fixer loop. After redelivery races, the claimRun primitive consolidated the per-role dedup logic behind three composable guards, and the janitor’s thresholds were rebalanced only after they reaped healthy work. The system’s complexity reads as scar tissue: none of the controls existed in the initial design, and the postmortem directory gained 29 entries in 27 active days, about one a day.

Silent capability loss is the family hardest to detect. A loud failure, such as a crash or a red CI run, entered the record and was fixed within days. In privateos#193, an agent’s terminal tools were defined but never registered, so it produced plausible work without the capability the work assumed, and nothing raised an error. The matching SDK quirk in privateos#220 inverted the sign, recording successful runs as failures. Both artifacts date to the same audit, which suggests this family is found by deliberate inspection rather than by alerts.

The merge-rate spread (Fig. 2) bounds what the fleet can be trusted with. Where acceptance criteria were mechanical, such as a file list or a passing sequence of gates, agents merged 92 to 100 percent of their pull requests. Where acceptance was aesthetic, 15 of 23 did not merge, 14 of them still open at the snapshot date and waiting on a human judgment the board could not encode. The operator also remained the only effective backstop when controls failed. In privateos#362, where the kill mechanism updated a row while the process ran on, attention rather than architecture was the remaining layer of enforcement. The honor system that MKT-002 located in guidance documents survives here in a narrower form, as the operator’s own vigilance.

This is a single fleet, run by a single operator, on one model family, over 29 days, and none of the rates reported here estimate anything beyond this system. The before era contains four PRs, the era split is post-hoc, and the contract landed alongside other early fixes, so the attribution in §6 rests on mechanism as much as on the test in Eq. 1. The merge-rate contrast is confounded by the operator being both the merger and the author of the system under study. Runtime telemetry was lost to a database reset, so cost, latency and run-level failure rates are absent. The GitHub record undercounts any agent activity that produced no PR, and the agent/ branch heuristic may miss agent-authored work pushed under other names. The construction and operation surfaces also overlap: the 22 agent-authored privateos PRs of §4.3 mean the fleet helped build its own orchestrator from May 5 onward, and the 285-PR construction record is not purely human work. The incident narrative draws on engineering documents written by the same operator whose system they describe. Verification (§4.4) holds those narratives to the artifact record and says nothing about what the record omits.

A month of complete operational record from a self-hosted agent fleet supports one measured claim and two structural ones. The measured claim is that moving pull-request structure from prompted instruction to code-level enforcement took compliance from 1 of 4 to 71 of 71. In the same change, the median blast radius shrank from 9 files to 3. The first structural claim is that failures recur in families (scope, loops, silence, liveness, isolation), each of which eventually purchased a control subsystem. The second is that the fleet’s competence tracks how crisply the work is specified: agents merged 92 to 100 percent of their pull requests under mechanical acceptance criteria and 35 percent under aesthetic ones.

What remains open is the cost ledger. The next operation window runs with the telemetry retained, which permits the questions this record cannot answer: cost per persona, time to recovery by failure family, and whether the contract effect holds when the before era is larger than four.