Plan — the standalone, installable ztd CLI

Turn ztd from “a binary that must run inside a checkout of this repo” into a self-contained CLI you install (Homebrew on macOS, a downloaded binary on Linux) and run from any project directory: brew install ztd → cd my-project → ztd init → ztd up. The binary carries everything it needs; the only config a user ever touches is a single ztd.toml.

Delivered as phases, each an Opus-orchestrated red/green/verify cycle (see the red-green-verify skill) with an independent adversarial review before it closes — the same discipline that built the vz backend.

Linux handoff — finishing the Terraform-backend work

Complete. The handoff described here was carried out: Phase C-verify, D, D1 and E all landed on a Linux host, and kvm/proxmox/ec2 are validated end-to-end from .ztd/ztd.toml with no terraform.tfvars. Kept as the record of how the work moved between machines.

Phases A–C were built and verified on macOS, where the vz backend + ztd init

• the config→env layer are green. What remains needs a Linux host, because kvm/proxmox/ec2 boot Terraform inside the toolbox and cannot be boot-tested on macOS (macOS only runs terraform validate). To move the work over:

Prerequisites on the Linux box

1. Commit + push the branch (currently 0.4.x). The macOS work is committed per phase by the operator; the Linux box clones/pulls it. (At the time, .ztd/ztd.toml was tracked and travelled with the branch. It no longer is — this repo is public and the file held a real homelab topology and public IP, so it is gitignored and .ztd/ztd.toml.example is the committed template. Recreate it on the new host alongside the secrets in step 2.)
2. Recreate secrets — .ztd/secrets/proxmox.env + aws.env are gitignored and do NOT travel: cp examples/proxmox.env.example .ztd/secrets/proxmox.env (+ aws.env) and fill in the endpoint/token / AWS creds.
3. Host prep — Docker + libvirt/qemu/dnsmasq (kvm); then ./ztd check + ./ztd setup (the Docker→libvirt forwarding shim). See Host prerequisites.
4. Sanity — task go:test (deterministic on Linux; local resolves to libvirt there) and TARGET=proxmox ./ztd validate should be green out of the box.

Ordered Linux work (each has a full checklist elsewhere):

1. Phase C-verify (below) — boot kvm + proxmox/ec2 from .ztd/ztd.toml (no terraform.tfvars); relocate Terraform state → .ztd/state/.
2. Backend cohesion Phase 1 — make the Terraform tftpl and the lima provision both consume the one cloud-init/install-agent-stack.sh; boot-test all three TF backends.
3. Phase D (below) — publish the toolbox image; pin it in compose.yaml.
4. Phase E (below) — release + Homebrew; the Linux tarball download is tested here.

Do NOT redo on Linux: the vz backend, ztd init, and the config→env layer are macOS-proven and green — and vz cannot run on Linux (it needs macOS Virtualization.framework). Run ./ztd test (kvm) there, not TARGET=vz ./ztd test.

The two problems this solves

1. Assets aren’t bundled. The binary shells out to repo-relative files at runtime — scripts/*.sh, the Terraform root (*.tf + modules/), cloud-init/, lima/ztd.yaml.tmpl, compose.yaml, docker/Dockerfile, Taskfile.yml. There is no go:embed; they’re read from the current directory, so a brew-installed binary would find none of them.
2. CWD is doing double duty — it’s both the program (source of the assets above) and the project (.ztd/ config + keys + the working tree to share). Those must split: the binary finds its own assets (embedded); the user runs from their project.

The config north star (the user’s directive)

• ztd.toml is the ONLY config a user touches. Every dependency’s config — Terraform variables, the lima instance YAML, cloud-init, the compose env — is generated inside the binary from ztd.toml + baked-in defaults, and is never written as a file the user references or edits.
• Reasonable defaults for everything, so a ztd.toml that says only backend = "vz" (or nothing at all, on a Mac) boots.
• Concretely: terraform.tfvars is deleted (the binary feeds all TF_VAR_* from the resolved config); lima/cloud-init/compose are embedded + rendered to a scratch dir; the *.example files and the host scripts’ tfvar() helper (which reads terraform.tfvars for username/repo_target) go away.

Two settled decisions

• ztd.toml lives under .ztd/ (.ztd/ztd.toml, dropping the hidden .ztd.toml name) so all project state is under one .ztd/ dir.
• Secrets stay OUT of ztd.toml — the Proxmox token / AWS creds remain gitignored env files under .ztd/secrets/ (ztd.toml is meant to be committed and holds only non-secret config).

No repo split. The repo becomes the CLI (approach A below); the root-level source scaffolding is embedded or retired in place as it stops being needed.

Hard constraints / non-goals

Constraints

• Single binary, no runtime repo dependency. After install, nothing the binary needs may live outside the binary except the user’s .ztd/ project dir. Assets are embedded (//go:embed) and extracted to a versioned cache dir.
• go:embed can’t reach .. — embed patterns are relative to the .go file and cannot escape the module tree. This is why the module hoists to the repo root (Phase A): the assets (scripts/, lima/, …) currently sit above the cli/ module and can’t be embedded from there.
• One user config surface (ztd.toml) + defaults. No user-facing dependency config files. The binary is the single source of generated Terraform/lima/cloud-init.
• The other backends must not regress. kvm/proxmox/ec2/vz keep working at every phase; the Go suite stays green.
• Cross-platform. linux + darwin, amd64 + arm64 (goreleaser already targets this). local stays platform-native (vz on macOS, libvirt on Linux).

Non-goals

• Extracting the CLI to a separate repo (explicitly rejected — monorepo).
• A GUI / hosted control plane.
• New backends or backend features (this is packaging, not capability).
• Auto-installing host deps (lima/docker/libvirt) — ztd check guides; the Homebrew formula may declare deps, but ztd doesn’t install them.

Repository layout (target, after Phase A + B)

go.mod                      ⊕ hoisted to root — module gitlab.com/frob/ztd
main.go                     ⊕ package main (was cli/main.go)
cmd/ config/ engine/        ✎ moved up from cli/*; imports rewritten
  readiness/ orchestration/
internal/assets/            ⊕ embedded runtime payload (embed.FS)
  assets.go                 ⊕ //go:embed scripts lima cloud-init tf compose docker taskfile
  scripts/…                 ← moved from repo-root scripts/
  lima/ztd.yaml.tmpl        ← moved from repo-root lima/
  cloud-init/…              ← moved from repo-root cloud-init/
  tf/ (main.tf, variables.tf, outputs.tf, modules/)  ← moved from repo root
  compose.yaml, docker/, Taskfile.yml                ← moved from repo root
ztd / ztd.sh                ✎ dev shim: builds + execs ./bin/ztd (root module now)
.goreleaser.yaml            ✎ drop dir:/gomod.dir (module at root); add brews:
docs/                       Hugo docs (unchanged location)

Runtime dirs (created by the binary, not shipped):

<cache>/ztd/<version>/…     extracted assets (macOS ~/Library/Caches, else $XDG_CACHE_HOME)
<project>/.ztd/ztd.toml     the ONE user config
<project>/.ztd/{keys,state,vz,runs,secrets}/   per-project state (as today)

Execution model — phased red/green/verify

Each phase is one Opus-orchestrated red/green/verify cycle (red-green-verify skill) closed by an independent adversarial review + a docs-update box. Two test characters, as in the vz plan:

• Unit-testable (Go): asset resolution, config→TF_VAR_* mapping, defaults, the platform/dir split — pure Go with table tests.
• Structural / acceptance: the hoist and embedding are proven by the existing suite staying green + a real ztd up on a Mac (vz, the toolbox-free path) and, where possible, a Linux kvm/remote run. A “red” for these is a new assertion (e.g. “assets resolve when CWD is an empty dir”) that fails pre-change.

Shared commands: task go:test / go:vet / go:fmt; task build:docs; TARGET=vz ./ztd test (macOS acceptance, toolbox-free); ./ztd test (kvm, Linux).

Global rules (copy from the vz plan): red fails at runtime; green may not edit tests/fixtures; verify re-runs the suite + a real run and git diffs the tests; a phase isn’t done at green — the assume-bad review must find nothing; later phases only add; close each phase by updating docs (rebuilt with task build:docs).

Milestones (not all phases are needed for the first install)

• MVP — Homebrew vz on macOS: Phases A → B → C(partial) → E. vz needs no Docker toolbox and no Terraform, so a Mac user gets brew install ztd → ztd init → ztd up with only lima. Phase D is deferred for this milestone.
• Full standalone (all backends): add D (published toolbox image) + the Terraform-side of C (state relocation, all TF_VAR_* from config).

Phase map

Phase A — Hoist the Go module to the repo root

• Phase A complete
  • Red/Green — git mv’d go.mod/go.sum/main.go + cli/{cmd,config,engine,readiness,orchestration,tools} to the repo root (history preserved as renames); module path gitlab.com/frob/ztd/cli → gitlab.com/frob/ztd; rewrote every import; updated .goreleaser.yaml (dropped gomod.dir/builds.dir, fixed the cmd.Version ldflag), Taskfile.yml (dropped dir: cli, fixed the go:docs path), the ./ztd shim (bin/ztd + root .go staleness find), .gitignore (/bin/), and the Dockerfile comment. Suite green at the new root; ./ztd version/status, task go:docs (no diff), and gofmt all clean.
  • Verify + review — task go:build/vet/test/fmt green; real ./ztd version + ./ztd status (→ vz on macOS) work; README/code/config carry no old module path; goreleaser check validates; go:docs no diff. Independent review: all categories clean EXCEPT a half-staged index I’d created (git mv staged the renames; the sed import-rewrites were unstaged → a commit would’ve been non-building). Fixed with git add -A (staged .go files with the old path: 24 → 0; fresh build green) + the two cosmetic comment stragglers it flagged (ztd/Taskfile “cli/” mentions).
  • Done when: the module is rooted at the repo, the full suite is green, and ./ztd runs unchanged. ✅
  • Update docs: CLAUDE.md Layout (root Go module) + the vz-gotcha path (cmd/vz.go); Dockerfile comment. (Module-path change is internal — no user-facing behavior change; the ./ztd interface is identical.)

Phase B — Embed assets; split assets dir vs project dir

• Phase B complete
  • Red/Green — assets.go at the repo ROOT (package main) //go:embeds scripts lima cloud-init + the Terraform root + compose.yaml docker Taskfile.yml; main registers it via cmd.SetAssets. cmd/assets.go resolves the assets dir ($ZTD_ASSETS_DIR → extract embedded to <UserCacheDir>/ztd/<version>/ idempotently, “dev” re-extracts → "" in tests so assetPath stays relative and the dispatch tests are unchanged); hostLane runs bash assetPath(script). The project dir is split out: main exports ZTD_PROJECT_DIR + ZTD_BIN, root prefers $ZTD_PROJECT_DIR, vz.sh splits ASSETS (templates) vs PROJECT (.ztd/), and the agent scripts call "${ZTD_BIN:-./ztd}". Dev shim exports ZTD_ASSETS_DIR=$PWD for live scripts. Unit tests: cmd/assets_test.go (resolveAssetsDir/extractAssets).
    • Deviation from the sketch: assets stay at the repo root (embedded from the root main package + threaded via SetAssets) rather than moved under internal/assets/ — go:embed must be co-located with a package, and the root package is main; this keeps the terraform/scripts dev workflow in place.
    • Scoped: the containerLane (kvm/proxmox/ec2) still runs docker compose in cwd=project (works from the source repo; from an arbitrary dir it needs the assets tf mounted separately from the project) — that landed in Phase D1, not “Phase C/D”: Phase C moved only the state. Phase B proves the toolbox-free vz path.
  • Verify + review — proven: the built binary run from an empty /tmp dir (no source, env -u ZTD_ASSETS_DIR) extracted its assets and booted a real vz guest; .ztd/keys + the rendered lima config landed in the PROJECT dir, scripts/templates came from the extracted cache; status Running; clean down. Full Go suite + gofmt/vet green; the dev-shim flow still works. Independent review: all categories clean except one HIGH — captureVZEndpoint (status --watch) hardcoded bash scripts/vz.sh instead of assetPath, so --watch on vz would’ve failed from a non-source dir. Fixed (status.go → assetPath(vzScript)). Also hardened the two flagged extractAssets items: keyed cache reuse on a build id (the binary’s mtime) so repeated + recursive invocations reuse instead of re-extracting (was “dev always re-extracts”, which could yank files from a running script), and a test init clears ZTD_ASSETS_DIR so the dispatch tests can’t be perturbed by a shim-sourced shell. Left (noted): echoed ./ztd next-step hints and the container-lane “no compose.yaml” native error are cosmetic / Phase C/D.
  • Done when: a binary run from a project dir with no ztd source present boots a vz guest end-to-end. ✅
  • Update docs: CLAUDE.md — the embed/assets-dir/project-dir/ZTD_BIN mechanism + the containerLane caveat. (User-facing “install & run anywhere” how-to lands with the release in Phase E.)

Phase C — ztd.toml as the sole config

• Phase C complete (macOS-provable scope; kvm/proxmox/ec2 boot is Linux-gated — see Phase C-verify)
  • Red/Green — config.Resolver (reusable env>toml>default) + ConfigPath (renamed .ztd/ztd.toml, legacy .ztd/.ztd.toml fallback); orchestration.BackendEnv emits ZTD_* (vz sizing/user/image/repo_target) + the full TF_VAR_* set from ztd.toml+defaults, merged into dispatchEnv; compose.yaml forwards the whole TF_VAR_* namespace. Ported all host scripts OFF the tfvar() helper to ZTD_*/ TF_VAR_* env (helper removed). vz.sh reads ZTD_MEMORY_MB/ZTD_DISK_GB + honors ZTD_REPO_TARGET. New ztd init scaffolds .ztd/ztd.toml (the old terraform-init verb removed; Taskfile up/plan/validate auto-run terraform init). Deleted terraform.tfvars + .ztd/.ztd.toml.example. Config Config struct
    • its tests unchanged (topology read via the Resolver, not typed fields).
    • Deferred to Phase C-verify (per the option-2 decision): the Terraform state → .ztd/state/ relocation ships with the Linux boot verification (it needs a real apply to confirm), not this turn.
  • Verify + review — proven on macOS: ztd init scaffolds + is idempotent; ztd up (vz) from a project whose ONLY config is a custom .ztd/ztd.toml (vcpus=1 memory_mb=2048 username=agent) rendered cpus: 1, memory: "2048MiB", user agent and booted — config→env→vz end-to-end. TARGET=proxmox ./ztd validate → “configuration is valid” with NO terraform.tfvars. Full Go suite + vet + gofmt green. kvm/proxmox/ec2 boot is unverifiable on macOS → Phase C-verify (Linux). Independent review — one HIGH fixed: the deleted terraform.tfvars held this repo’s REAL topology (proxmox mcp/ztd-ssh/insecure, the AWS ingress IP) which BackendEnv’s defaults did NOT reproduce (→ broken proxmox boot / world-open AWS SG); migrated it into a committed .ztd/ztd.toml. MEDIUM fixed: [vm] arch was advertised but unread — BackendEnv now resolves vm.arch (tested). LOWs fixed: stale terraform.tfvars hints in check-requirements.sh → ztd.toml; a config test for the primary .ztd/ztd.toml name. Left (LOW, tracked): the dead scripts/load-config.sh + its ztd test:config smoke check still key off the old name — remove in a cleanup pass.
  • Done when: the whole system runs from .ztd/ztd.toml + defaults; no external dependency-config file exists or is read. (vz ✅; kvm/proxmox/ec2 ✅ — see Phase C-verify below.)
  • Update docs: a full ztd.toml reference (every key + default); retire the tfvars references in the proxmox/ec2 how-tos. (Landed with Phase C-verify’s doc pass below — docs/content/reference/configuration.md rewritten as the ztd.toml reference; proxmox-backend.md/ec2-backend.md/ README.md/variables.tf comments retired their terraform.tfvars mentions.)

Phase C-verify — boot the Terraform backends from ztd.toml (Linux)

Confirm on a Linux host what macOS can’t: that the TF_VAR_*-from-config flow actually provisions kvm/proxmox/ec2 with no terraform.tfvars.

This repo already carries a migrated .ztd/ztd.toml (proxmox mcp/ztd-ssh/ insecure, the AWS ingress IP — the values from the retired terraform.tfvars), so the dogfood config is ready; the toolbox mounts the source repo at /work, so a run from this checkout is the test (from-an-arbitrary-dir Terraform is Phase B’s deferred containerLane rework, tracked separately — not needed here).

• Phase C-verify complete
  • ./ztd up (kvm) boots green from the committed .ztd/ztd.toml (+ .ztd/secrets/*.env), with no terraform.tfvars; ./ztd down clean. Confirm the guest got the config’s sizing (virsh dominfo) — proves TF_VAR_vcpus/etc. flowed. Spot-check a non-default override (e.g. [vm] vcpus = 4) actually changes the VM. — Booted ztd-ztd-dev; virsh dominfo showed CPU(s)=2, Max memory=4194304 KiB, exactly matching .ztd/ztd.toml’s vcpus=2/memory_mb=4096. ZTD_VCPUS=4 ./ztd up (env-override path) rebuilt with CPU(s)=4, confirming env > toml precedence. Guest Docker (docker run hello-world) and claude --version both worked over SSH. ./ztd down destroyed cleanly both times.
  • TARGET=proxmox ./ztd up + TARGET=ec2 ./ztd up boot green — the real test of the migrated topology (proxmox_node=mcp reaches the node; the AWS SG uses the [aws] ssh_ingress_cidr, not 0.0.0.0/0). Confirm the SG ingress with aws ec2 describe-security-groups. — Proxmox: booted VM 132124044 on node mcp at 192.168.1.133 (2 vcpu/3.8Gi RAM matching config); guest Docker + claude --version worked over SSH; ./ztd down destroyed cleanly. Needed an ssh-agent with the operator key loaded, exported as SSH_AUTH_SOCK (not ZTD_SSH_AUTH_SOCK — that’s compose’s mount target; orchestration/hostenv.go reads SSH_AUTH_SOCK) — this Linux box had no agent running by default, unlike the prior macOS session. EC2: booted at 18.237.87.121; no aws CLI on this host, so the SG ingress was confirmed by reading aws_security_group.ztd’s ingress[0].cidr_blocks straight out of terraform.tfstate — ["203.0.113.10/32"], matching .ztd/ztd.toml’s [aws] ssh_ingress_cidr, not 0.0.0.0/0. Guest Docker + Claude Code confirmed over SSH; ./ztd down destroyed cleanly.
  • Relocate Terraform state → <project>/.ztd/state/terraform.tfstate (currently lands at the repo root as terraform.tfstate). Mechanism: add a backend "local" { path = ".ztd/state/terraform.tfstate" } to main.tf’s terraform {} block (path is relative to /work = the mounted project), or set TF_DATA_DIR + -state in the Taskfile. Confirm a full up/down cycle and that the repo root is clean. Add .ztd/state/ to .gitignore (already listed). — Added the backend "local" block to main.tf. Verified with a real kvm up/down: .ztd/state/terraform.tfstate was created (root-owned, written by the toolbox), validate/up needed no interactive migration since the pre-existing root-level state was empty (0 resources). Deleted the now-stale root terraform.tfstate/terraform.tfstate.backup; repo root is clean (.ztd/state/ was already gitignored).
  • Review: every TF_VAR_* BackendEnv/compose supplies matches variables.tf; no default silently changed the VM shape vs the old terraform.tfvars. — Diffed orchestration/backendenv.go’s TF_VAR_* map against every variable in variables.tf: all topology vars (name/arch/vcpus/ memory_mb/disk_gb/username/repo_dirname/repo_target/repo_source/ debian_image_url/proxmox_*/aws_*) are supplied by BackendEnv from ztd.toml + matching defaults. The 3 vars BackendEnv does NOT set (target, ssh_public_key_path, libvirt_uri) are intentionally out of scope — target flows via TARGET/dispatch elsewhere, and the other two keep their variables.tf defaults (ephemeral-key path, qemu:///system), which were never in the old terraform.tfvars either. No drift found.
  • Update docs: proxmox/ec2 how-tos — topology now in ztd.toml, secrets in .ztd/secrets/; note state lives in .ztd/state/. — docs/content/reference/configuration.md rewritten as the full ztd.toml reference (every [vm]/[agent]/[proxmox]/[aws] key, env override, default); how-to/proxmox-backend.md + how-to/ec2-backend.md + README.md retired their terraform.tfvars mentions in favor of .ztd/ztd.toml; variables.tf comments updated too. Left (LOW, tracked): the dead scripts/load-config.sh + ztd test:config still key off the legacy .ztd/.ztd.toml name — a pre-existing cleanup item noted at the end of Phase C, unchanged by this pass.

Phase D — Publish the toolbox image (Terraform backends)

• Phase D complete

  • Red — an acceptance assertion that kvm/proxmox/ec2 provision using a pulled ztd-toolbox:<version> image (no local build:), failing today (compose builds from ./docker). — Two layers. Go: orchestration/toolbox_test.go — TestComposeConsumesToolboxEnv reads the real compose.yaml and fails on the hard-coded image: ztd-toolbox:local / absent pull_policy, plus TestToolboxImageIsVersionPinned / …PullPolicyFollowsVersion / …EnvOverrides for the derivation. Shell: assert_toolbox_image in scripts/smoke-lib.sh, called from the three container-lane suites’ step 0 — it resolves the image through real docker compose config (so an interpolation typo can’t pass) and asserts the image/policy pairing matches the build kind. Confirmed red: the Go suite failed to build against the missing orchestration.Toolbox* seam, and every wantFullEnv-based dispatch test went red once the expectation included ToolboxEnv.

  • Green — CI builds + pushes ztd-toolbox to a registry on tag; the embedded compose.yaml references the pinned published image (version == binary version), with build: kept only as a dev fallback. — orchestration/toolbox.go derives the image from cmd.Version (release → registry.gitlab.com/frob/ztd/ztd-toolbox:<version>; dev/snapshot → ztd-toolbox:local) with pull policy missing for BOTH — compose resolves it per image (registry tag → pulled; ztd-toolbox:local, in no registry → falls through to build:). dispatchEnv injects ZTD_TOOLBOX_IMAGE/ZTD_TOOLBOX_PULL_POLICY into every verb’s env, and compose.yaml consumes them with the dev values as fallbacks — build: kept. ztd config gained toolbox_image/toolbox_pull_policy so the resolution is inspectable (and so the shell assertion has a non-circular source of truth). New .gitlab-ci.yml (the repo had NO CI): test:go on every push (gofmt/vet/ test inside a freshly built toolbox, so a broken Dockerfile can’t reach a tag) and toolbox:publish on tags only — multi-arch buildx push of $CI_REGISTRY_IMAGE/ztd-toolbox:${CI_COMMIT_TAG#v} + :latest, then an imagetools inspect so a missing tag fails loudly.

  • Verify + review — a Terraform-backend run pulls the image and provisions; the image tag matches the binary version. Review: air-gapped/build fallback still works for contributors; registry auth for private images. — Go suite + vet green. Both paths exercised end-to-end on the host: the dev binary reports ztd-toolbox:local/build and assert_toolbox_image passes 2/2; a binary rebuilt with -X …cmd.Version=0.4.0 reports registry.gitlab.com/frob/ztd/ztd-toolbox:0.4.0/missing, docker compose config --images resolves to exactly that ref, and the assertion correctly fails only its “image is present locally” leg — nothing is published at 0.4.0 yet, which is the assertion doing its job (it goes green on the first tagged CI publish). Air-gapped/contributor fallback: build: retained and ZTD_TOOLBOX_PULL_POLICY=build forces it on a release. Registry auth is deliberately NOT ztd’s job — docker login first (documented). REGRESSION FOUND AND FIXED AFTER THIS BOX WAS FIRST TICKED. The policy was initially build for dev. That rebuilds the toolbox on every container-lane invocation, and Docker Compose writes build progress to stdout — so IP="$(ztd ip)", which every host-side script uses to locate the guest, captured BuildKit output instead of an address, silently breaking ztd auth, mount, fetch, run and audit. Caught by ./ztd test (auth failed); the original Phase-D verification missed it because it only checked that compose resolved the right image name, never that a verb’s stdout stayed clean. Fixed to missing for both (semantics confirmed empirically with throwaway compose files: a registry-resolvable tag is PULLED under missing, an unresolvable one falls through to build:), guest-endpoint.sh hardened to take the last non-empty line + shape-check it, and assert_toolbox_image now pins pull_policy = missing so it cannot regress. Re-verified on real infrastructure afterwards: ./ztd test (kvm) 32/32, TARGET=proxmox ./ztd test 35/35, TARGET=ec2 ./ztd test 35/35 — the two remote suites resolve the guest through ./ztd ip for mount/fetch/run/audit, so they are the broad confirmation the fix is correct. Since resolved: the CI file has now been run by real GitLab pipelines and the image is published — ztd-toolbox:0.4.2 (and :latest) resolve as a two-platform manifest list, confirmed against the registry.

  • Done when: an installed binary runs kvm/remote backends with no local Docker build, from the pinned image. — Demonstrated end to end on 0.4.2. CI published registry.gitlab.com/frob/ztd/ztd-toolbox:0.4.2 (manifest list = exactly linux/amd64 + linux/arm64, no attestation entries; :latest at the same digest). A binary stamped -X cmd.Version=0.4.2, run from outside the source tree, resolved that ref, pulled it after the local copy was deleted (Image …:0.4.2 Pulling), ran terraform validate inside it (Success!), provisioned a real guest (up exit 0, IP 192.168.122.168, hostname = ztd-ztd-dev over SSH), and tore it down clean. ztd-toolbox:local was never touched — no local build happened at any point. — Registry visibility RESOLVED: the published image is anonymously pullable (docker manifest inspect succeeds with no docker login), so a released binary works for someone who has never authenticated to GitLab. — Caveat at the time, RESOLVED by Phase D1: the container lane then ran docker compose in the PROJECT dir and mounted ./:/work, so the Terraform backends still needed the source repo in CWD. Phase D1 split that into /work (assets) + /project.

  • Publish the first image — done on tag 0.4.2. Note 0.4.1 FAILED to publish: buildx attaches provenance attestations as unknown/unknown manifest entries and the GitLab registry rejects them with blob unknown to registry after the layers upload. Fixed with --provenance=false --sbom=false.

  • Run a Terraform backend from a released binary — see “Done when”.

  • Update docs: how-to — image source + version pinning; contributor build. — New docs/content/how-to/toolbox-image.md (the rule, ztd config output, both env overrides, private registries, building it yourself) + how-to index; CLAUDE.md gained a toolbox-pinning gotcha including the tag-must-match-CI invariant.

Phase D1 — Relocate the compose root (run from an arbitrary dir)

Phase D made the toolbox image installable. This makes the toolbox invocation installable. Until it lands, brew install ztd && cd anywhere && ztd up is true only for vz; the three Terraform backends still require a source checkout in the working directory, which makes the Phase-E packaging hollow.

The problem. containerLane runs docker compose with cwd = the project dir and compose.yaml mounts ./:/work, so the toolbox receives the directory you invoked from and the Terraform root has to be inside it. Note this is NOT the state relocation deferred in Phase B — Phase C already moved state to <project>/.ztd/state/. What remains is the config root and the compose file itself. (Phase B’s note and CLAUDE.md both call this “Phase C/D — state relocation”; both are stale and are corrected by this phase.)

The shape of the fix. Split the single /work mount in two, mirroring the assets-vs-project split the binary already makes everywhere else:

Four consequences, each of which is a real edit rather than a detail:

• docker compose -f <assets>/compose.yaml — the compose file itself comes from the assets tree, since an arbitrary project dir has none.

• An explicit -p <project> is REQUIRED, not cosmetic. With -f pointing at the cache, compose derives the project name from that dir’s basename — which is the version (0.4.2). Dots are illegal in a compose project name, so without -p every released binary fails immediately.

• TF_DATA_DIR=/project/.ztd/terraform so terraform init writes providers into the project, not the read-mostly asset cache.

• Taskfile paths that are project state (.ztd/keys/id_ed25519) must move to /project/...; paths that are config stay relative to /work.

• Phase D1 complete

  • Red — an acceptance assertion that a container-lane backend provisions from a project dir containing only .ztd/. — cmd/compose_root_test.go: TestContainerLaneUsesAssetsComposeFile, TestContainerLaneSetsProjectName, TestComposeSplitsAssetsFromProject, TestDispatchEnvCarriesBothDirs. All four confirmed red. The behavioural red was the real one: booting from an empty /tmp dir with an installed-style binary.
  • Green — the /work (assets) + /project split, -f/-p on the lane, TF_DATA_DIR=/project/.ztd/terraform, Taskfile key paths under /project, .terraform.lock.hcl embedded, and ZTD_ASSETS_DIR/ZTD_PROJECT_DIR set EXPLICITLY by dispatchEnv rather than relying on compose’s ${VAR:-.} fallback (which resolves to the compose file’s own dir — correct today only by coincidence). Two latent bugs surfaced and were fixed:
    • cmd/status.go hand-rolled its own docker compose … task ip argv instead of using containerLane, so ztd status would have broken from any non-checkout dir. Now routed through the lane.
    • var.ssh_public_key_path defaulted to .ztd/keys/id_ed25519.pub, relative to the terraform root — which is now the ASSET tree. main.tf guards the read with fileexists(), so it did not error: it injected an EMPTY authorized_key and booted a guest nobody could reach. It passed from a checkout purely because /work happened to contain .ztd/. Default is now absolute under /project. This is exactly the class of bug the bare-dir acceptance existed to catch, and only the bare-dir run caught it.
  • Verify + review — no regression from the source checkout: ./ztd test (kvm) 32/32, TARGET=proxmox ./ztd test 35/35, TARGET=ec2 ./ztd test 35/35 (the remote suites are the strong ones here — mount/fetch/run/audit all resolve the guest through the container lane). Bare-dir acceptance: a binary stamped 0.4.2, run with ZTD_ASSETS_DIR/ZTD_PROJECT_DIR unset from /tmp/ztd-d1-proj containing only .ztd/, booted a guest (192.168.122.147), SSH’d in on the PROJECT’s own ephemeral key, and tore down clean. Review — nothing writes to the asset cache: state, providers (TF_DATA_DIR) and keys all landed under the project, and a find over ~/.cache/ztd/0.4.2 for terraform.tfstate*/id_ed25519* came back empty. Two projects on one host: the lane pins -p ztd, and every container is a run --rm one-shot with a compose-assigned random suffix; with network_mode: host and no named volumes there is no project-scoped resource left to collide over.
  • Done when: kvm/proxmox/ec2 provision from a project dir holding only .ztd/, matching what vz already does. — Demonstrated above.
  • Update docs: — CLAUDE.md’s assets-vs-project gotcha now states the /work vs /project container-lane contract and drops the “needs the source repo in CWD” caveat; Phase B’s stale “lands in Phase C/D” note and Phase D’s Done-when caveat both corrected to point here.

Phase E — Release pipeline + Homebrew (non-red/green)

Implement-and-review pass (packaging/CI, not unit-testable).

• Phase E complete (the clean-Mac brew install is the one item that needs a Mac and a real tag — see “Left to verify” below)
  • Homebrew — a homebrew_casks: block, NOT brews:. Homebrew deprecated formulae that install pre-built binaries, goreleaser followed, and goreleaser check FAILS on brews: as of v2.17 (confirmed empirically before writing the config). A cask is macOS-only, which matches the plan exactly: brew on macOS, tar.gz on Linux. Declares lima as a dependency and strips the Gatekeeper quarantine attribute in a postflight (unsigned binaries). Docker is left a documented prerequisite — Desktop/Colima/other is the user’s choice, not something a cask should decide. nfpm .deb/.rpm skipped (the plan marked it optional); the tarball + checksums cover Linux.
  • Tag → release pipeline — release:cli in a new release stage, so it runs AFTER toolbox:publish. That ordering is load-bearing, not tidiness: a released binary derives its toolbox image tag from its own version, so binaries published before the image exists hand anyone installing in the gap a ztd up that cannot pull. scripts/ci/release.sh asserts the stamped version equals ${CI_COMMIT_TAG#v} — the exact string publish.sh pushed. The job image IS goreleaser (no dind), which sidesteps the bind-mount trap ci/test.sh documents rather than working around it; the shared script uses a native goreleaser when one is on PATH and docker run locally.
  • Verify — task release:ci green: four archives (linux/darwin × amd64/arm64), checksums.txt, and a generated cask. The rehearsal asserts the things a bare --snapshot will not: the full arch matrix (goreleaser exits 0 on a subset just as happily), a runnable binary whose version ldflag is actually stamped (an unstamped dev build resolves its toolbox image to ztd-toolbox:local, which exists in no registry), ztd init succeeding from a bare dir with ZTD_ASSETS_DIR/ZTD_PROJECT_DIR unset — the embedded assets survived the release build — and the cask’s dependency, quarantine hook and download URL.
    • One real bug caught by that URL assertion: goreleaser picks its forge from which *_TOKEN is in the environment, NOT from the git remote or the release: block. With none set it assumes GitHub and wrote github.com/frob/ztd download URLs into the cask — a repo that does not exist — while the build, archives and checksums all stayed green. The rehearsal now exports a dummy GITLAB_TOKEN (--skip=publish, so nothing is published) to reproduce CI faithfully.
    • The real tag has since happened. 0.4.3 published all four archives plus checksums.txt to the GitLab release, and pushed Casks/ztd.rb to frob/homebrew-ztd — both confirmed against the API. Two defects surfaced only once a tag ran for real, and are fixed: the release body was empty (goreleaser generates none when changelog.disable is set, and does not treat that as an error — the CHANGELOG section for the tag is now extracted and passed via --release-notes, with a missing section a hard failure), and tags 0.4.0–0.4.2 predated the release:cli job so they published an image but no CLI.
    • Left to verify (needs a clean Mac): brew tap + brew install --cask ztd, then ztd up on vz from a project whose only config is .ztd/ztd.toml. Everything up to and including the tap push is now proven.
  • Update docs: new how-to/install.md (brew, the Linux download with checksum verification, from source, the first-run flow, and where ztd puts things — <project>/.ztd/ vs the disposable version-scoped asset cache); the Quickstart and README rewritten to start from an installed binary rather than a cloned repo; CONTRIBUTING gained a “Cutting a release” section and the release:ci rehearsal.

Specification

§1 — Assets & the cache dir

Embed scripts/ lima/ cloud-init/ tf/ compose.yaml docker/ Taskfile.yml via //go:embed. Extract once per version to <cache>/ztd/<version>/ (macOS ~/Library/Caches/ztd, else $XDG_CACHE_HOME/ztd → ~/.cache/ztd). The extracted tree is read-only program payload; never the project. Re-extract only when the version dir is absent (idempotent, cheap).

§2 — Project dir vs assets dir

ZTD_PROJECT_DIR = the dir the user invoked from (holds .ztd/, the tree to mount). The binary exports it; host scripts read .ztd/… and repo-source from it, and templates from the assets dir. This retires the current CWD==repo assumption (vz.sh cd $(dirname)/..; agent scripts’ $PWD/.ztd).

§3 — Config schema (.ztd/ztd.toml)

Superset of today’s [vm]/[agent]: backend, name, vcpus, memory, disk, arch, username, repo-sync; [proxmox] node/datastore/bridge/…; [aws] region/type/…; [agent] skills/commands/max_time/max_turns. Every key has a default; the binary resolves env > ztd.toml > default (unchanged precedence) and emits the full TF_VAR_* set. Secrets are NOT here — .ztd/secrets/*.env as today.

§4 — Generated dependency config

From the resolved config the binary produces, in the scratch/cache dir only: TF_VAR_* env (no terraform.tfvars); the rendered lima YAML; the rendered cloud-init; the compose env. Terraform state lives in <project>/.ztd/state/. Nothing here is a user-editable file.

§5 — ztd init

Scaffold <project>/.ztd/ztd.toml from an embedded template with commented defaults, and .ztd/secrets/*.env.example stubs for the remote backends. Idempotent (won’t clobber an existing ztd.toml).

§6 — Release

goreleaser at the repo root (module hoisted): cross-platform binaries + tar.gz + checksums + a Homebrew formula pushed to a tap, plus the tagged toolbox image. Version flows via the existing cmd.Version ldflag and stamps the asset cache dir + the image tag.

Taskfile

Two Taskfiles, unchanged in spirit: the host dev Taskfile (go:*, build:docs) points at the root module after Phase A; the toolbox Taskfile (up/down/… run inside the container) becomes an embedded asset (extracted into the toolbox’s /work). No new user-facing tasks — the Go CLI is the interface. Everything stays containerized (containerized-tooling) and verb:subject (taskfile-conventions).

Definition of done

• brew install ztd (macOS) / a downloaded binary (Linux) yields a working ztd with no ztd source checkout present. — The Linux half is proven end-to-end (task test:standalone, and task release:ci runs ztd init from a bare dir using the archived binary). The macOS half is proven through the tap push — 0.4.3 put the archives on the release and Casks/ztd.rb in the tap — leaving only the brew install on a clean Mac.
• From an arbitrary project dir: ztd init → edit/accept .ztd/ztd.toml → ztd up boots a guest (vz on macOS, libvirt on Linux) using only .ztd/ztd.toml
  • defaults. — Phase D1 verify (a 0.4.2-stamped binary booting from /tmp/ztd-d1-proj) and task test:standalone.
• No external dependency-config file exists or is read — no terraform.tfvars, no user-facing lima/cloud-init/compose files; all generated inside the binary. — Phase C.
• Secrets remain in .ztd/secrets/; ztd.toml is safe to commit. — Phase C; ztd check’s git-hygiene section now enforces the second half.
• All backends (vz/kvm/proxmox/ec2) work from the installed binary; Terraform backends pull the pinned toolbox image. — kvm/proxmox/ec2 proven from an installed-style binary (Phase D1 + test:standalone, with assert_toolbox_image guarding the pin). vz is proven from the dev wrapper only — the macOS backend has not been re-run against an installed binary; it shares the same asset/project resolution, but that is inference, not a run.
• Go suite green; docs build green; every code phase (A–D) landed via red/green/verify + independent review, Phase E the implement-and-review pass.