fix(release): refuse crate packaging that would reuse same-version Cargo state

publish-crates.sh now stops before packaging, and before the publish-mode
tag/asset guards, when Cargo would verify release crates against state left
by an earlier package run at the same version:

- a <crate>-<version> extraction of a release crate under any
  $CARGO_HOME/registry/src/* other than index.crates.io-* (Cargo skips
  re-extraction once .cargo-ok exists), or
- a rustc dep-info file <build-dir>/*/deps/<lib>-*.d that names such an
  extraction (tmp-registry units track no sources, so Cargo keeps them
  while version, deps, features and profile match).

The refusal lists every extraction and, per extraction, how many compiled
units in the build dir were built from it. It deletes nothing (deleting
shared caches needs founder authority). It prints the isolated rerun that
produced the 0.10.2 receipt at 14683c9983: a new, empty CARGO_HOME and
CODEWHALE_CACHE_ROOT under <cache root>/release-verify-<sha>, plus
CODEWHALE_BUILD_LOCK_FILE so the rerun keeps the machine build lock (the
2026-10-16 run's lock lived inside its fresh root and did not serialize
with other builds).

validate-crate-publish-order.py already loads cargo metadata for this
script; it now also emits build_directory (falling back to
target_directory), so the guard reuses that call instead of running a
second cargo metadata. A missing build dir fails loud.

Why: the dry runs at 9f25ef094c and db34bc8949 are not valid 0.10.2
receipts. One failed with E0432 because the codewhale-core extraction from
0b2f5022f4 still had `use anyhow::Result`. After that extraction was moved
aside, the next run passed while linking codewhale-state and
codewhale-config rlibs built 2026-10-15 23:45 from older sources.

publish-crates.test.py: each test now sets its own CARGO_HOME. The change
made the suite wrong without it. With one CARGO_HOME shared across the six
tests (CI's ~/.cargo, or any dev machine), the first verifying test leaves
registry/src/-<hash>/codewhale-preflight-base-0.0.0, and the guard then
refuses the next four tests (run with a shared scratch CARGO_HOME: FAILED
failures=4).

docs/RELEASE_RUNBOOK.md documents the refusal and the isolated rerun. It
also covers the consequence for publish and partial-publish reruns (each
run needs a new root and cargo login for its CARGO_HOME), the CNB work/v*
caller (.cnb.yml runs this script in a fresh container) and the known
limits. It fixes the interrupted-dry-run paragraph too: the half-packed
package/ lives in Cargo's build directory, which is target/package only
with plain Cargo.

Verification (this Mac, HEAD 8f54845f78 plus this diff):
- Real state, through the strict wrapper chain (dev-cargo.sh, build lock),
  with a scratch shim that exits 99 on cargo publish/package/build/check:
  exit 1 in 3.8 s. It listed 28 extractions in
  ~/.cargo/registry/src/-bef5e33d39b5c38c and 145 compiled units across 28
  crates in /Volumes/VIXinSSD/cw-build-cache/build/67/a5b75c886c6102. The
  shim was never reached. The extraction listing, the deps entry count
  (181346) and the codewhale-state eb1a8b21e14a9d0b rlib were unchanged.
- Empty CARGO_HOME with the shared build dir: refused on the 145 units
  alone (0 extractions), which is the moved-aside case.
- Empty isolated CARGO_HOME and CODEWHALE_CACHE_ROOT (scratch, offline):
  the guard passed and the script reached "Verifying all 29 release package
  tarballs"; the shim stopped cargo publish (exit 99). Nothing was built;
  the root holds only an empty build/.
- python3 scripts/release/publish-crates.test.py (under build-lock.py):
  "Ran 6 tests ... OK"; ~/.cargo/registry/src unchanged.
- bash scripts/release/validate-crate-publish-order.test.sh: passed.
- node .github/scripts/release-workflows.test.js: OK.
- bash -n passes; shellcheck reports only the 3 findings HEAD already has.
- cargo fmt --all -- --check: exit 0. No Rust changed, so there is no
  clippy or nextest target.
- npm test: root 101/101, runtime-sdk 19/19, extension-host 420 (413 pass,
  7 skipped), web node 7/7. npm run check:web: exit 0, vitest 772/772
  (74 files).

Not verified: a full dry run with the guard (deliberately not run); the
changed script and test under GNU grep in CI (ubuntu) or CNB; publish mode
end to end; cargo login into a fresh CARGO_HOME.

Not a CHANGELOG item (maintainer release tooling). Not mirrored to GitHub.

Refs CW-7445 (the 0.10.2 release log in CW-7384 recorded the finding)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NBKbhx19BRbZtvWK8rVKjz

docs/RELEASE_RUNBOOK.md

@@ -133,8 +133,51 @@ source edits, but the final release receipt must describe the frozen source.
 It stops without uploading. CI/RC does not verify Cargo publication tarballs
 and cannot replace this gate. Publish mode necessarily verifies them again.
 
+Cargo treats every crate it serves from that temporary registry as immutable by
+name and version, so a later run at the same version can verify an earlier
+run's code. It skips extracting a crate under `$CARGO_HOME/registry/src/` once
+`.cargo-ok` exists, and it keeps compiled units of those crates in the build
+directory because they track no source files. At 0.10.2 one dry run failed with
+`E0432 unresolved import anyhow` in codewhale-core from an older extraction;
+after that extraction was moved aside, the next run passed while linking
+codewhale-state and codewhale-config built from older sources. Before
+packaging, both modes therefore refuse to start when they find a release
+crate's `<crate>-<version>` directory under any `$CARGO_HOME/registry/src/*`
+other than crates.io's `index.crates.io-*`, or a dep-info file under the build
+directory's `*/deps/` that names such a directory. The refusal lists what it
+found and deletes nothing; removing shared cache directories still needs
+explicit cleanup approval. It prints, with the paths filled in, the rerun that
+produced the 0.10.2 receipt at 14683c9983 on 2026-10-16: a new, empty
+`CARGO_HOME` and `CODEWHALE_CACHE_ROOT`.
+
+```bash
+lock="${CODEWHALE_CACHE_ROOT}/build.lock"
+root="${CODEWHALE_CACHE_ROOT}/release-verify-$(git rev-parse --short=10 HEAD)"
+mkdir "${root}"
+CODEWHALE_BUILD_LOCK_FILE="${lock}" CARGO_HOME="${root}/cargo-home" \
+  CODEWHALE_CACHE_ROOT="${root}" ./scripts/release/publish-crates.sh dry-run
+```
+
+That run was a cold build of about 19 GB and 962 compile steps, so check
+`df -h` first. It kept its lock file inside the new root and did not serialize
+with other builds on the machine; `CODEWHALE_BUILD_LOCK_FILE` keeps the machine
+lock. `CODEWHALE_CACHE_ROOT` moves the build directory only when `cargo` runs
+through `scripts/dev-cargo.sh`; with plain Cargo, set
+`CARGO_TARGET_DIR="${root}/target"` instead. Every run leaves state that the
+next run at the same version refuses, so publish mode needs another new root,
+with `cargo login` run for that `CARGO_HOME` (the token lives there). The CNB
+`work/v*` release preflight in `.cnb.yml` calls this script in a fresh
+container, where the check finds nothing. Known limits: the check also refuses
+compiled units recorded against a different `CARGO_HOME`, which Cargo would
+rebuild rather than reuse, and it does not scan `<target-triple>/`
+subdirectories of the build directory, which host-only package verification
+does not use.
+
 If a dry-run is interrupted (Ctrl-C, or a shell that gets killed), inspect the
-half-packed `target/package` before retrying and coordinate with active builds.
+half-packed `package/` directory in Cargo's build directory before retrying and
+coordinate with active builds. With plain Cargo that is `target/package`;
+`scripts/dev-cargo.sh` moves it under `${CODEWHALE_CACHE_ROOT}/build/`, and
+`cargo metadata --format-version 1 --no-deps` reports it as `build_directory`.
 Cargo can resume against it and every unpacked tarball may then fail with
 `error: Current directory is invalid:
 No such file or directory (os error 2)` — which looks like a compiler or
@@ -351,7 +394,9 @@ and fails branch-only release sources before assets are published.
    dependency order lives in `scripts/release/crates.sh`; do not maintain a
    second handwritten order in this runbook. The helper waits for each new
    version to appear on crates.io before moving to dependents and safely skips
-   versions that are already public on a rerun.
+   versions that are already public on a rerun. Run it from a new, empty
+   `CARGO_HOME` and cache root, as the package preflight above describes; the
+   dry run's state makes it refuse otherwise.
    If shared `main` advances or the checkout becomes dirty, stop and coordinate
    a new candidate or separately authorized recovery with Hunter. Do not switch
    branches, create another checkout, force-fetch a conflicting tag or bypass
@@ -576,7 +621,9 @@ for that version — bump to the next patch instead.
   - keep [docs/INSTALL.md](INSTALL.md#roll-back-to-a-previous-release) in sync
     with these commands
 - Crates publish partially:
-  - rerun `./scripts/release/publish-crates.sh publish`
+  - rerun `./scripts/release/publish-crates.sh publish` from another new,
+    empty `CARGO_HOME` and cache root (see the package preflight); the failed
+    run's own state makes it refuse otherwise
   - already-published crate versions will be skipped
 - GitHub assets missing or checksum manifest incomplete:
   - fix `.github/workflows/release.yml`, but do not rerun it over an existing

scripts/release/publish-crates.sh

@@ -27,6 +27,7 @@ packages=("${release_crates[@]}")
 crates_user_agent="CodeWhale release publish check (https://github.com/codewhale-hq/CodeWhale)"
 
 workspace_version=""
+build_dir=""
 
 metadata_inventory="$(
   python3 "${script_dir}/validate-crate-publish-order.py" "${packages[@]}"
@@ -36,6 +37,9 @@ while IFS=$'\t' read -r kind name value; do
     version)
       workspace_version="${name}"
       ;;
+    build-dir)
+      build_dir="${name}"
+      ;;
   esac
 done <<<"${metadata_inventory}"
 
@@ -43,9 +47,57 @@ if [[ -z "${workspace_version}" ]]; then
   echo "Could not determine workspace version." >&2
   exit 1
 fi
+if [[ -z "${build_dir}" ]]; then
+  echo "Could not determine Cargo's build directory." >&2
+  exit 1
+fi
 
 echo "Crate publication order OK: ${#packages[@]} workspace crates."
 
+cargo_home="${CARGO_HOME:-${HOME}/.cargo}"
+reused=()
+for package in "${packages[@]}"; do
+  for extracted in "${cargo_home}"/registry/src/*/"${package}-${workspace_version}"; do
+    if [[ -e "${extracted}" && "${extracted}" != "${cargo_home}"/registry/src/index.crates.io-*/* ]]; then
+      reused+=("extracted source ${extracted}")
+    fi
+  done
+done
+deps_dirs=("${build_dir}"/*/deps)
+if [[ -d "${deps_dirs[0]}" ]]; then
+  include_args=()
+  for package in "${packages[@]}"; do
+    include_args+=("--include=${package//-/_}-*.d")
+  done
+  crate_pattern="$(IFS='|'; echo "${packages[*]}")"
+  version_pattern="${workspace_version//./\\.}"
+  version_pattern="${version_pattern//+/\\+}"
+  units="$(
+    grep -rHoE "${include_args[@]}" -- "[^ ]*/registry/src/[^/ ]+/(${crate_pattern})-${version_pattern}/" "${deps_dirs[@]}" ||
+      [[ $? -eq 1 ]]
+  )" || {
+    echo "Could not scan ${build_dir} for compiled release crates." >&2
+    exit 1
+  }
+  while read -r count extracted; do
+    reused+=("${count} compiled unit(s) in ${build_dir} built from ${extracted%/}")
+  done < <(grep -v -e '^$' -e '/registry/src/index\.crates\.io-' <<<"${units}" | sort -u | cut -d: -f2- | sort | uniq -c)
+fi
+if (( ${#reused[@]} > 0 )); then
+  cache_root="${CODEWHALE_CACHE_ROOT:-${XDG_CACHE_HOME:-${HOME}/.cache}/codewhale}"
+  verify_root="${cache_root}/release-verify-$(git rev-parse --short=10 HEAD 2>/dev/null || echo candidate)"
+  {
+    echo "Refusing to package release crates: Cargo would reuse ${workspace_version} state from an earlier package run instead of the current sources:"
+    printf '  %s\n' "${reused[@]}"
+    echo "Cargo treats crates served from its temporary package registry as immutable by name and version. It skips extraction once .cargo-ok exists and keeps compiled units that do not track their sources, so this run would verify old code. Nothing was deleted."
+    echo "Rerun from a new, empty Cargo home and cache root:"
+    echo "  mkdir \"${verify_root}\""
+    echo "  CODEWHALE_BUILD_LOCK_FILE=\"${CODEWHALE_BUILD_LOCK_FILE:-${cache_root}/build.lock}\" CARGO_HOME=\"${verify_root}/cargo-home\" CODEWHALE_CACHE_ROOT=\"${verify_root}\" $0 ${mode}"
+    echo "If cargo is not scripts/dev-cargo.sh, set CARGO_TARGET_DIR=\"${verify_root}/target\" in place of CODEWHALE_CACHE_ROOT. Publish mode also needs cargo login with that CARGO_HOME. See docs/RELEASE_RUNBOOK.md."
+  } >&2
+  exit 1
+fi
+
 if [[ "${mode}" == "publish" ]]; then
   "${script_dir}/require-release-tag-checkout.sh"
   "${script_dir}/verify-release-assets.sh"

scripts/release/publish-crates.test.py

@@ -82,6 +82,7 @@ class PublishPreflightTests(unittest.TestCase):
             "TEST_UPLOADS": str(self.uploads),
             "PATH": str(bin_dir) + os.pathsep + os.environ["PATH"],
             "CARGO_NET_OFFLINE": "true",
+            "CARGO_HOME": str(self.root / "cargo-home"),
             "CARGO_TARGET_DIR": str(self.root / "target"),
         }
         self.run_command(["cargo", "generate-lockfile", "--offline"], success=True)

scripts/release/validate-crate-publish-order.py

@@ -177,13 +177,17 @@ def validate_order(
 def main() -> int:
     args = parse_args()
     try:
-        packages = workspace_packages(load_metadata(args.metadata_file))
+        metadata = load_metadata(args.metadata_file)
+        packages = workspace_packages(metadata)
         version, dependency_flags = validate_order(packages, args.crates)
     except ValidationError as error:
         print(error, file=sys.stderr)
         return 1
 
     print(f"version\t{version}\t")
+    build_directory = metadata.get("build_directory") or metadata.get("target_directory")
+    if isinstance(build_directory, str) and build_directory:
+        print(f"build-dir\t{build_directory}\t")
     for name in sorted(dependency_flags):
         print(f"crate\t{name}\t{1 if dependency_flags[name] else 0}")
     return 0