CI: retry pip installs on haiku, the VM corrupts data

This VM hands back data that does not match what was written to it: rustc
crashed on crate metadata it had written itself ("assertion failed:
bytes[len] == STR_SENTINEL"), and pip found sha256 mismatches reading
wheels back from its own cache - twice on cython (1.3MB), once on
virtualenv (5.5MB), with a different wrong hash every time and none of
them matching any published artifact. Only the big files were hit.

Retry the installs with a purged pip cache, so a damaged file does not
fail the job - and does not get stored in the actions cache, from where
it would fail every following run until someone deletes the cache.

Co-Authored-By: Claude Opus 5 <[email protected]>

.github/workflows/ci.yml

@@ -661,18 +661,30 @@ jobs:
               export BORG_LIBLZ4_PREFIX=/system/develop
               export BORG_OPENSSL_PREFIX=/system/develop
 
-              # rustc 1.94.1 crashes while compiling the blake3 crate ("assertion failed:
-              # bytes[len] == STR_SENTINEL"), i.e. it read back metadata that does not
-              # match what it wrote. Build outside /boot/system/cache/tmp (haiku's /tmp)
-              # and run one rustc at a time, in case that corruption comes from disk or
-              # memory pressure.
+              # this VM corrupts data every now and then: rustc crashed on crate metadata it
+              # had written itself ("assertion failed: bytes[len] == STR_SENTINEL") and pip
+              # found sha256 mismatches reading big wheels back from its own cache. Build
+              # outside /boot/system/cache/tmp (haiku's /tmp) and run one rustc at a time,
+              # in case that corruption comes from disk or memory pressure.
               export TMPDIR=/boot/home/borg-tmp
               mkdir -p "$TMPDIR"
               export CARGO_TARGET_DIR="$TMPDIR/cargo-target"
               export CARGO_BUILD_JOBS=1
 
-              pip install -r requirements.d/development.lock.txt
-              pip install -e .
+              # retry a pip install that hit a corrupted file, dropping the cached (possibly
+              # corrupted) files first, so one of them does not fail the whole job - and does
+              # not end up in the actions cache, failing every following run, too.
+              pip_install() {
+                for attempt in 1 2 3; do
+                  pip install "$@" && return 0
+                  echo "*** pip install failed (attempt $attempt), purging the pip cache and retrying ***"
+                  pip cache purge || true
+                done
+                return 1
+              }
+
+              pip_install -r requirements.d/development.lock.txt
+              pip_install -e .
 
               # troubles with either tox or pytest xdist, so we run pytest manually:
               pytest -v -n auto -rs --cov=borg --cov-config=pyproject.toml --cov-report=xml --junitxml=test-results.xml --benchmark-skip -k "not remote and not socket"