Plan — backend cohesion & parity

Keep the backends (kvm, proxmox, ec2, vz) sharing one pipeline, branching only where the mechanism genuinely differs — so functionality and testing stay in parity as the set grows. Delivered as red/green/verify phases with an independent review, like the vz backend and standalone CLI plans.

The decision: centralize at the contract, NOT at Terraform

A tempting way to “unify” would be to route vz through Terraform too. Rejected, because it adds branching rather than removing it:

• No usable/maintained Terraform provider for lima. The other three backends have real providers (dmacvicar/libvirt, bpg/proxmox, hashicorp/aws) that do declarative VM management; lima has none. The only “TF” option is a null_resource + local-exec shell wrapper around limactl — Terraform as a worse Makefile (no plan-diffing, flaky destroy-time provisioners).
• The toolbox can’t run it. Terraform runs inside the Linux toolbox container; limactl needs the host’s macOS Virtualization.framework, which a Linux container can’t reach — even on a Mac. A TF-driven vz would force Terraform onto the host (a new macOS dependency) and split TF execution across two contexts. That’s more branching and breaks the “all tooling in Docker” invariant.

Terraform is a provision mechanism, not the pipeline. The real central pipeline is the Go CLI + the six-operation backend contract (SPEC §4: host-readiness, host-setup, provision, health-gate, endpoint-resolution, orphan-reconciliation). Centralize there and the mechanism becomes one clean branch.

The two legitimate branch points

Everything else is (or becomes) shared; a backend should differ in exactly two places:

1. provision — a Terraform module (kvm/proxmox/ec2) vs limactl (vz, host-side).
2. endpoint shape — a bare IP (virsh lease / guest-agent API / AWS attribute) vs 127.0.0.1:<forwarded-port> (vz). Already unified as a host [port] pair by scripts/guest-endpoint.sh.

Already shared (the parity that exists today)

Config resolution + local-is-platform-native; the ephemeral-key lifecycle; the content of cloud-init (cloud-init/install-agent-stack.sh is canonical); the whole agent lifecycle (auth/skills/run/attach/audit, port-threaded); and the uniform ztd <verb> surface (dispatchLane). This plan closes the remaining gaps: the guest-provisioning delivery still duplicates, and the smoke suites duplicate.

Hard constraints / non-goals

Constraints

• No behavior change to any backend. This is dedup/centralization, not new capability; every backend keeps working and every suite stays green.
• Boot-testing the Terraform backends needs a Linux host. kvm/proxmox/ec2 can’t boot on macOS, so Phase 1’s verify is gated on a Linux/self-hosted run; terraform validate + render inspection is the macOS-side check.
• The only per-backend branches are provision-mechanism + endpoint-shape. A new duplication introduced elsewhere is a regression this plan exists to prevent.

Non-goals

• Routing vz (or any local backend) through Terraform — explicitly rejected above.
• Changing backend behavior, images, or the guest end-state.
• Merging the four smoke suites into one file — they stay per-backend entry points; only the shared assertions are extracted.

Execution model — phased red/green/verify

Each phase is one Opus-orchestrated red/green/verify cycle (red-green-verify skill) closed by an independent adversarial review + a docs-update box. Global rules as in the vz plan (red fails at runtime; green may not weaken tests; verify re-runs the suites + git diffs; the assume-bad review must find nothing; later phases only add).

Test character. Phase 1 has a real behavioral invariant (the guest end-state is identical across backends) that a smoke assertion pins. Phase 2 is a refactor — its gate is “all four suites still green and the shared assertions live in one place,” not a new feature. Phase 3 is Go unit-testable (the interface seam).

Shared commands: task go:test/vet/fmt; TARGET=vz ./ztd test (macOS); ./ztd test + TARGET=proxmox/ec2 ./ztd test (Linux); task build:docs.

Phase map

Phase 1 — Single-source guest provisioning

Today cloud-init/install-agent-stack.sh is canonical, but the delivery duplicates: vz embeds it in a lima provision entry, while the Terraform backends inline the same commands in cloud-init/user-data.yaml.tftpl (kept in sync by hand — the unification deferred in the vz plan).

• Phase 1 complete
  • Red — a smoke assertion (all backends) that the guest reaches the SAME end-state from the SAME source: e.g. a marker the script writes is present, and a check that the tftpl no longer inlines the docker/node/claude commands (they come from the shared script). Fails today for the TF backends (they inline). — Added a static preflight assertion to scripts/smoke-test.sh (step 0) that greps cloud-init/user-data.yaml.tftpl for the inlined install strings (get.docker.com/nodesource/npm install -g @anthropic-ai/claude-code) and fails if found. Confirmed red against the pre-refactor tftpl.
  • Green — refactor user-data.yaml.tftpl to consume cloud-init/install-agent-stack.sh (cloud-init write_files the script via templatefile, then one runcmd runs it) so there is ONE source; vz already embeds it. terraform validate clean; the rendered cloud-init runs the same commands, in order. — main.tf now reads the script via file("${path.module}/cloud-init/install-agent-stack.sh") and passes it to templatefile; the tftpl writes it to /root/install-agent-stack.sh via write_files (content: | + indent(6, ...)) and runcmd executes it with ZTD_GUEST_USER=${username}. The tftpl’s packages: list dropped the ca-certificates/curl/git/rsync/tmux entries (the script installs them) — only qemu-guest-agent stays, since the script deliberately omits it (a Terraform-backend-only concern; vz doesn’t need it). terraform validate clean; a terraform console render + yaml.safe_load confirmed the emitted YAML parses and the script content round-trips byte-for-byte.
  • Verify + review — on a Linux host, ./ztd test (kvm) + TARGET=proxmox/ec2 ./ztd test boot green with the refactored delivery; on macOS, terraform validate + a render diff confirm parity, and TARGET=vz ./ztd test stays green. Review: any command dropped/reordered vs the old inline runcmd? qemu-guest-agent still handled per-backend (TF backends need it; vz omits it)? — ./ztd test (kvm): 26/26 passed, including the new single-source assertion. TARGET=proxmox ./ztd up/TARGET=ec2 ./ztd up: both booted; guest end-state (docker/claude/tmux/git/rsync present, docker run hello-world works, claude --version works) verified over SSH on both, then cleanly down. vz.sh/lima path untouched by this change (only main.tf + cloud-init/user-data.yaml.tftpl + the shared install script’s header comment changed) — no macOS regression risk; TARGET=vz ./ztd test not re-run this session (no macOS host available), but nothing in its code path was touched. qemu-guest-agent: still only in the tftpl’s packages:/runcmd, install script still explicitly omits it (comment preserved).
  • Done when: exactly one file defines the guest agent stack; both delivery vehicles consume it; all four backends boot to the same end-state.
  • Update docs: note the single-source provisioning; drop the “keep the two in sync” caveat from install-agent-stack.sh + CLAUDE.md once unified. — Both updated: install-agent-stack.sh’s header now describes the write_files/runcmd consumption path instead of “keep in sync manually”; CLAUDE.md’s vz-gotchas section replaced the “keep the two in sync” line with the actual mechanism.

Phase 2 — Shared smoke-test library

smoke-test-vz.sh duplicates much of smoke-test.sh (the color/ok/bad/step helpers, the ephemeral-key + foreign-key checks, the cloud-init/docker/node/claude checks, the agent-lifecycle checks). Factor the common parts into scripts/smoke-lib.sh; each suite sources it and provides only its backend-specific probe (endpoint resolution; mount-type assertion; the local-vs-remote bits).

• Phase 2 complete
  • Red — the duplication is the starting condition (measurable: the same assert helpers + checks copy-pasted across smoke-test*.sh). Add a guard that fails if the shared assertions aren’t sourced from one place (e.g. a lint/grep check, or the suites reference smoke-lib.sh functions that don’t exist yet). — Added a preflight assertion to scripts/smoke-test.sh (step 0, alongside Phase 1’s single-source check) that greps all four smoke-test*.sh for smoke-lib.sh; confirmed red (3/4 missing it) before the lib existed.
  • Green — extract scripts/smoke-lib.sh (helpers + backend-agnostic assertion functions parameterized by an injected endpoint/ssh accessor); each of the four suites sources it and keeps ONLY its backend-specific probes (virsh lease vs guest-agent API vs terraform output vs vz.sh endpoint; virtiofs fstype vs lima mount). No assertion changes — pure dedup. — scripts/smoke-lib.sh holds: colors/step/ok/bad/info/die (with an optional DIE_HOOK so proxmox/ec2’s “tear down on abort” behavior survives unchanged, kvm/vz’s “rely on the EXIT trap” behavior also survives unchanged) and smoke_verdict; assert_ephemeral_key_only, assert_cloud_init_done, assert_claude_installed, assert_tmux_installed, assert_guest_docker, assert_guest_internet; and, for the two near-byte-identical remote suites, assert_mount_sync/assert_fetch/assert_run_plumbing/assert_audit_export/ assert_remote_teardown (proxmox/ec2’s mount/fetch/run/audit/teardown blocks were verbatim duplicates — the biggest single win). smoke-test.sh, smoke-test-proxmox.sh, and smoke-test-ec2.sh now source the lib and use all of the above; smoke-test-vz.sh sources it too but deliberately keeps its ephemeral-key and cloud-init checks inline — vz’s key check additionally proves a foreign key is REJECTED (a strictly different assertion, not just a reshaped one) and its cloud-init capture has no cut -d: -f2, so unifying those would have changed what’s asserted, violating “no assertion changes”; vz does use the shared claude/tmux/docker/verdict helpers, which are genuinely identical in shape.
  • Verify + review — every suite still passes unchanged: TARGET=vz ./ztd test (macOS) + ./ztd test / TARGET=proxmox/ec2 ./ztd test (Linux); git diff shows dedup, not behavior change (same pass/fail set). Review: did any suite silently lose a check in the extraction? are the shared assertions truly backend-agnostic (no hidden virsh/libvirt assumption leaking into vz)? — Re-ran all three Linux suites for real: ./ztd test (kvm) 27/27 (26 + the new lib-source assertion), TARGET=proxmox ./ztd test 30/30 (full suite incl. mount/fetch/run/audit), TARGET=ec2 ./ztd test 30/30 — identical pass sets to the pre-refactor runs (accounting for the one added assertion), all backends torn down clean. TARGET=vz ./ztd test could NOT be re-run this session (no macOS host available); mitigated by keeping vz’s non-identical assertions inline untouched and only swapping in helpers whose logic is provably identical to what vz already did — reviewed by inspection, no macOS regression risk expected, but flagged here as unverified-by-execution. No shared assertion references virsh/libvirt/terraform — they all operate on values the caller already captured (strings, exit codes), so nothing backend-specific leaked into the lib.
  • Done when: the shared assertions live in one smoke-lib.sh; each suite is only its backend-specific probes + the shared calls; all four green (vz green by inspection/prior state, not re-executed this session).
  • Update docs: CONTRIBUTING/how-to — “adding a backend” now means: a provision mechanism, an endpoint probe, and a thin smoke-suite over smoke-lib. — Added an “Adding a backend” paragraph to CONTRIBUTING.md’s Testing section.

Phase 3 — Thin Go backend interface (finish the contract)

Formalize the six-op contract so the per-backend code is only the two branch points. Much exists (dispatchLane, guest-endpoint.sh, smokeTestScript, readiness.DefaultChecks(target)); this pass names the seam.

• Phase 3 complete
  • Red — Go tests for a small Backend seam: Provision/Endpoint/ HealthGate differ per backend; everything else (key, cloud-init render, agent verbs, dispatch) is shared and target-independent. Fails against the current scattered isVZ/switch target sites. — cmd/backend_test.go’s TestBackendIsTheOnlySeam parses every non-test .go file in cmd/ with go/parser and fails if the backend-name string literals ("vz"/"proxmox"/"ec2") appear anywhere outside backend.go (AST-level, so it can’t be fooled by a comment or a substring in an unrelated message). Confirmed red: it found the literals scattered across vz.go (isVZ’s own target == "vz") and smoketest.go (the switch target picking the smoke script) — the actual scatter that existed.
  • Green — consolidate the per-backend branches behind the seam (keep isVZ/dispatchLane semantics; just centralize them); the shared pipeline calls the seam. No behavior change; kvm/proxmox/ec2/vz byte-identical dispatch. — New cmd/backend.go holds a Backend{HostLane, SmokeTestScript} struct + a backends table (vz/proxmox/ec2; unlisted targets get the container-lane local-suite default) and backendFor(target). isVZ (in vz.go) and smokeTestScript (in smoketest.go) are now one-line wrappers over backendFor instead of holding their own ==/switch — same names, same call sites, same return values, purely relocated. dispatchLane (vz.go) and readiness.DefaultChecks’s repo-mount branch (readiness.go) are UNCHANGED — they’re the two allowed branch points (provision mechanism, endpoint shape), not scatter to eliminate; backend.go’s doc comment says so explicitly so a future reader doesn’t “fix” them.
  • Verify + review — full Go suite green; a real run per available backend; review that the ONLY places that branch on the backend are provision + endpoint (+ the readiness mount-probe, which follows endpoint). — TestBackendIsTheOnlySeam now passes (0 stray literals); full suite (go build/vet/test, all packages) green in the toolbox. Real run: ./ztd test (kvm) 27/27 unchanged after the refactor — proves smokeTestScript/ isVZ/dispatchLane still resolve identically end to end (this is the code path ztd test/ztd status --watch/ztd tunnel all go through). proxmox/ec2/ vz dispatch logic itself untouched (only the two lookup functions moved), so not independently re-booted for this phase — their Phase 1/2 boot-verification already covers the scripts these functions merely point at. Review: grepped cmd/*.go/readiness/*.go/config/*.go for target ==/case "..." — the only remaining sites are backend.go (the seam), vz.go’s dispatchLane (provision-mechanism branch), readiness.go’s repo-mount check (endpoint-shape branch), and config.go’s LocalBackendFor (a goos, not backend-name, branch — macOS-native local resolution, unrelated to this seam).
  • Done when: adding a backend touches exactly the seam (provision + endpoint), not the shared pipeline. — Registering a new backend now means: add one entry to the backends table in backend.go (name → HostLane + SmokeTestScript), implement its provision/endpoint branch in dispatchLane/readiness.go if it needs a new shape, write its Terraform module or host driver, and add a scripts/smoke-test-<name>.sh over smoke-lib.sh — nothing else in cmd/ needs to change, enforced going forward by TestBackendIsTheOnlySeam.
  • Update docs: the backend-contract reference — the seam + the two branch points, with each backend’s implementation named. — backend.go’s package-level doc comment IS the reference (seam + the two allowed branch points + where each lives); CONTRIBUTING.md’s “Adding a backend” paragraph (added in Phase 2) already points contributors at it.

Definition of done

• One file defines the guest agent stack; both Terraform cloud-init and the lima provision consume it (no hand-synced duplication).
• One scripts/smoke-lib.sh holds the shared assertions; the four suites are thin backend-specific probes over it; all four green (vz on macOS, the rest on Linux). (vz confirmed green by inspection only — no macOS host available this session to re-execute TARGET=vz ./ztd test.)
• The only per-backend branches in the codebase are provision-mechanism and endpoint-shape (+ the readiness mount-probe that follows endpoint) — statically enforced by TestBackendIsTheOnlySeam.
• No backend behavior changed; vz was not routed through Terraform.
• Every code phase landed via red/green/verify + independent review, each closed by its docs-update box.