Performance · Published 26 August 2026
How to benchmark a distributed agent runtime without inventing a leaderboard
A reproducible method for measuring admission, worktree creation, tunnel backpressure, recovery, and operator-visible latency.
Benchmark the boundary you operate
A single task-completion score hides the costs that determine whether a distributed runtime survives production: policy admission, filesystem isolation, transport backpressure, process startup, event fan-out, and crash recovery. Measure those stages independently before measuring an end-to-end agent result.
Use a fixed repository snapshot, fixed executables, fixed prompts, and a recorded node policy. Publish the machine profile and reject runs that silently fall back to a different provider or containment mode.
Latency and throughput protocol
Record p50, p95, p99, maximum, and failure count for request admission, detached worktree creation, first terminal byte, diff materialization, and orderly teardown. Throughput tests must include bounded queues and a slow consumer so the protocol proves backpressure rather than buffering until memory exhaustion.
Separate cold starts from warm starts. A Git object cache, an already-running daemon, and an existing tmux server materially change the result.
Adversarial cases belong in the benchmark
Replay duplicates, reorder frames, truncate payloads, disconnect during workspace creation, kill the provider process, and restart the daemon between durable state transitions. A fast happy path with an unmeasured recovery path is not a production result.
The pass criterion is convergence: one authorized workspace, one durable terminal state, no leaked credential, and a receipt that accurately reports what happened.
Comparisons need equivalent boundaries
A cloud workspace and an operator-owned node solve different deployment problems. Compare time-to-first-use, isolation ownership, network reachability, and evidence availability separately. Do not translate a product’s architecture into a blanket “faster” or “safer” claim.
Bernato publishes no synthetic win here. The useful artifact is the protocol and the raw measurements a buyer can rerun.
Production checklist
- Verify Default deny against the deployed environment, not a screenshot.
- Verify No inbound node port against the deployed environment, not a screenshot.
- Verify Bounded execution against the deployed environment, not a screenshot.
- Verify Non-executable memory against the deployed environment, not a screenshot.