mirror of
https://github.com/Droid-Deck/DroidDeck-Components.git
synced 2026-11-07 10:46:08 -10:00
Host DroidDeck's package catalogs and releases here, mirrored, fetched and built by CI (#1)
The four catalogs DroidDeck reads (components.json, desktop.json, linuxfs.json, steam-seed.json) move here from The412Banner/winlator-contents with every link rewritten to this repository, and every file they point at now lives on a release here, produced by a workflow: - mirror-release.yml brings a release in: each asset is checked against what the catalog promises for it (md5/size from the Bottles manifests, sha256 for the packages, the files archive's sha256 inside each recording), uploaded under the same tag, its sha256 confirmed against what GitHub stored, and the run named in MIRROR.md beside a SHA256SUMS. In upstream mode the vendor installers a pinned Bottles manifest (2026-06-23) describes with the same md5 are fetched from the vendor's own link, 55 of the 131 Windows packages; the rest are copies. Releases: system-libraries-v1, component-snapshots-v1, steamdeck-desktop-r1, linuxfs-r9, steam-proton-arm64-25502785. - build-dll-bundles.yml rebuilds the 39 hand-made DLL bundles from the vendors' packages (tools/bundles: recipes from the Bottles manifests, opened with cabextract, 7-Zip, zipfile and DroidDeck's own installer unpacker), each refused unless file-for-file identical to the bundle it replaces; 38 built, PowerShell kept as the verified copy (origin unknown; the app leaves it out), the VC++ 2022 twin built from the 2026 release channel since Microsoft no longer serves the version the hand-made one held. BUILT.md names the run and every source. - The recorder (component-snapshots.yml + tools/snapshot) and the desktop, emulator, AppImage and linuxfs builders move here, so future releases are produced here. Two catalog errors inherited from the Bottles manifests are corrected on the way: the VC++ 2012 32-bit installer entry was the 2010 installer, and the 2010 32-bit one was the original release while its 64-bit twin was SP1 (tools/mirror/upstream-overrides.json holds the vendor links). Nothing on any release was uploaded by hand.
This commit is contained in:
1 parent
780ab619de
commit
a9ce53ffb3
22 files changed
+8778
No files matched your search
@@ -0,0 +1,55 @@
|
||||
name: Build desktop packages (SteamDeck)
|
||||
|
||||
# The hosted packages the SteamDeck app installs into its Linux runtime on request: the LXQt/labwc
|
||||
# desktop and the tier-1 emulator set, each the closure of a seed list over Arch Linux ARM minus
|
||||
# what the runtime image already carries. Published to one release the app's desktop.json points at.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release:
|
||||
description: 'Release tag to publish to (created if missing)'
|
||||
required: true
|
||||
default: 'steamdeck-desktop-r1'
|
||||
packages:
|
||||
description: 'Space-separated package names (seed files under desktop/seeds/)'
|
||||
required: true
|
||||
default: 'desktop emulators'
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 90
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Build
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sudo apt-get update -qq && sudo apt-get install -y -qq zstd >/dev/null
|
||||
mkdir -p out
|
||||
for name in ${{ inputs.packages }}; do
|
||||
desktop/build-pkg.sh "$name" "desktop/seeds/$name.txt" desktop/base-packages.txt out
|
||||
done
|
||||
ls -l out
|
||||
|
||||
- name: Publish
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
TAG='${{ inputs.release }}'
|
||||
if ! gh release view "$TAG" >/dev/null 2>&1; then
|
||||
gh release create "$TAG" --prerelease --title "SteamDeck desktop packages ($TAG)" \
|
||||
--notes "Hosted packages for the SteamDeck app's Linux runtime. Each extracts over the r9 rootfs; see desktop.json for the catalog."
|
||||
fi
|
||||
gh release upload "$TAG" out/* --clobber
|
||||
echo "published:"; for f in out/*.tar.zst; do n=$(basename "$f" .tar.zst); echo " $n $(stat -c %s "$f") $(cat out/$n.sha256)"; done
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: desktop-packages
|
||||
path: out/*.packages.txt
|
||||
@@ -0,0 +1,79 @@
|
||||
name: Build the DLL bundles
|
||||
|
||||
# Rebuilds the DLL bundles on system-libraries-v1 from the vendors' own packages
|
||||
# (tools/bundles/recipes.json), checks each against the hand-made bundle it replaces file for
|
||||
# file (tools/bundles/expect.json), packs it reproducibly and publishes it, with BUILT.md naming
|
||||
# this run and every source package's origin and md5. A bundle that differs is refused.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
bundles:
|
||||
description: 'Space-separated bundle file names, or all'
|
||||
required: true
|
||||
default: 'all'
|
||||
release:
|
||||
description: 'Release the bundles are published to'
|
||||
required: true
|
||||
default: 'system-libraries-v1'
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
# One writer at a time on the release (the mirror workflow shares this group).
|
||||
concurrency:
|
||||
group: system-libraries-v1
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 180
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Tools (cabextract, 7-Zip, msitools for DroidDeck's unpacker)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y -qq --no-install-recommends cabextract p7zip-full msitools >/dev/null
|
||||
# DroidDeck's own installer unpacker, as the phone runs it (pinned to the commit that merged it).
|
||||
curl -fsSL -o droiddeck-msi-install "https://raw.githubusercontent.com/Droid-Deck/DroidDeck/9b90386e73b83ac5b43a64831b150c7852c18e38/tools/linuxfs/overlay/usr/local/bin/droiddeck-msi-install" \
|
||||
|| curl -fsSL -o droiddeck-msi-install "https://raw.githubusercontent.com/Droid-Deck/DroidDeck/main/tools/linuxfs/overlay/usr/local/bin/droiddeck-msi-install"
|
||||
python3 -m py_compile droiddeck-msi-install
|
||||
echo "DROIDDECK_MSIINFO=$(command -v msiinfo)" >> "$GITHUB_ENV"
|
||||
echo "DROIDDECK_CABEXTRACT=$(command -v cabextract)" >> "$GITHUB_ENV"
|
||||
echo "DROIDDECK_7Z=$(command -v 7z)" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Build
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
BUNDLES: ${{ inputs.bundles || 'all' }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
only=""
|
||||
[ "$BUNDLES" = all ] || only="--only $BUNDLES"
|
||||
python3 tools/bundles/build.py --recipes tools/bundles/recipes.json --expect tools/bundles/expect.json \
|
||||
--out out --cache cache --unpacker ./droiddeck-msi-install $only 2>&1 | tee build.log
|
||||
{ echo "## DLL bundles"; echo; tail -1 build.log; echo; grep -E '^ (built|skipped|REFUSED)' build.log | sed 's/^ /- /'; } >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Publish
|
||||
if: success()
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
TAG: ${{ inputs.release || 'system-libraries-v1' }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sed -i "1a\\\\nBuilt by $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID.\n" out/BUILT.md
|
||||
gh release upload "$TAG" out/* --clobber
|
||||
# SHA256SUMS from what GitHub now holds, so it covers the rebuilt files.
|
||||
gh api "repos/$GITHUB_REPOSITORY/releases/tags/$TAG" --paginate --jq '.assets[] | select(.name!="SHA256SUMS" and .name!="MIRROR.md" and .name!="BUILT.md") | "\(.digest|sub("^sha256:";"")) \(.name)"' | sort -k2 > SHA256SUMS
|
||||
gh release upload "$TAG" SHA256SUMS --clobber
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: dll-bundles
|
||||
path: |
|
||||
out/
|
||||
build.log
|
||||
@@ -0,0 +1,117 @@
|
||||
name: Build emulator from source (SteamDeck, tier 3)
|
||||
|
||||
# The emulators that ship no ARM64 Linux binary — PCSX2, Dolphin, Cemu — built on GitHub's ARM
|
||||
# runner and published as AppImages to the desktop release. Each is a large build; one at a time.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
target:
|
||||
description: 'dolphin | pcsx2 | cemu'
|
||||
required: true
|
||||
default: 'dolphin'
|
||||
release:
|
||||
description: 'Release tag to publish to (none = keep the AppImage as a run artifact only)'
|
||||
required: true
|
||||
default: 'steamdeck-desktop-r1'
|
||||
cemu_ref:
|
||||
description: 'Cemu commit or branch to build (cemu only; empty = main)'
|
||||
required: false
|
||||
default: ''
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-24.04-arm
|
||||
timeout-minutes: 340
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
sparse-checkout: desktop
|
||||
- name: Toolchain
|
||||
run: |
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y --no-install-recommends git cmake ninja-build build-essential \
|
||||
pkg-config python3 curl file desktop-file-utils libfuse2 \
|
||||
qt6-base-dev qt6-base-private-dev qt6-svg-dev qt6-tools-dev \
|
||||
libgl1-mesa-dev libvulkan-dev libsdl2-dev libx11-dev libxext-dev libwayland-dev \
|
||||
libudev-dev libevdev-dev libasound2-dev libpulse-dev libcurl4-openssl-dev \
|
||||
zlib1g-dev libzstd-dev liblz4-dev libbz2-dev libpng-dev libhidapi-dev \
|
||||
libbluetooth-dev libusb-1.0-0-dev libsystemd-dev gettext
|
||||
|
||||
- name: Fetch linuxdeploy (AppImage packager, aarch64)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
for tool in linuxdeploy linuxdeploy-plugin-qt; do
|
||||
curl -fsSL -o "/usr/local/bin/$tool" \
|
||||
"https://github.com/linuxdeploy/$tool/releases/download/continuous/$tool-aarch64.AppImage"
|
||||
chmod +x "/usr/local/bin/$tool"
|
||||
done
|
||||
|
||||
- name: Build ${{ inputs.target }}
|
||||
id: build
|
||||
run: |
|
||||
set -euo pipefail
|
||||
T='${{ inputs.target }}'
|
||||
case "$T" in
|
||||
dolphin)
|
||||
git clone --depth 1 https://github.com/dolphin-emu/dolphin.git src
|
||||
git -C src submodule update --init --recursive --depth 1
|
||||
cmake -S src -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DLINUX_LOCAL_DEV=false \
|
||||
-DDISTRIBUTOR=SteamDeck -DENABLE_TESTS=OFF -DCMAKE_INSTALL_PREFIX=/usr
|
||||
cmake --build build -j"$(nproc)"
|
||||
DESTDIR="$PWD/AppDir" cmake --install build
|
||||
# The install already placed the entry and icon; point at those copies.
|
||||
APPRUN_BIN=dolphin-emu; DESKTOP=AppDir/usr/share/applications/dolphin-emu.desktop
|
||||
ICON=AppDir/usr/share/icons/hicolor/256x256/apps/dolphin-emu.png
|
||||
;;
|
||||
pcsx2|cemu)
|
||||
# Newer libraries than the runner image has (Qt 6.10 for PCSX2, wxWidgets 3.3 for
|
||||
# Cemu): built in an Arch Linux ARM container, the rootfs's own distribution.
|
||||
docker run --rm -v "$PWD:/work" -w /work -e CEMU_REF='${{ inputs.cemu_ref }}' menci/archlinuxarm:base-devel \
|
||||
bash desktop/build-emulator-arch.sh "$T"
|
||||
sudo chown -R "$(id -u):$(id -g)" .
|
||||
echo "packaged=yes" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
;;
|
||||
*) echo "unknown target"; exit 2 ;;
|
||||
esac
|
||||
# The desktop entry and icon appimagetool insists on, placed where linuxdeploy looks for
|
||||
# them. The icon takes the name the entry's Icon= line asks for, or the two never meet.
|
||||
ICONNAME=$(sed -n 's/^Icon=//p' "$DESKTOP" | head -1); ICONNAME=${ICONNAME:-$APPRUN_BIN}
|
||||
D="AppDir/usr/share/applications/$(basename "$DESKTOP")"; I="AppDir/usr/share/icons/hicolor/256x256/apps/$ICONNAME.png"
|
||||
[ "$DESKTOP" -ef "$D" ] || install -Dm644 "$DESKTOP" "$D"
|
||||
[ "$ICON" -ef "$I" ] || install -Dm644 "$ICON" "$I"
|
||||
# -print -quit, not a pipe into head: under pipefail a closed pipe would end the step.
|
||||
BIN=$(find AppDir/usr -maxdepth 3 -type f -name "$APPRUN_BIN" -print -quit)
|
||||
[ -n "$BIN" ] || { echo "no $APPRUN_BIN in AppDir"; find AppDir -maxdepth 3 -type d; exit 3; }
|
||||
echo "apprun=$BIN" >> "$GITHUB_OUTPUT"
|
||||
echo "desktop=AppDir/usr/share/applications/$(basename "$DESKTOP")" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Package AppImage
|
||||
if: steps.build.outputs.packaged != 'yes'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
export OUTPUT='${{ inputs.target }}.AppImage' QMAKE=/usr/lib/qt6/bin/qmake6 EXTRA_QT_PLUGINS=svg
|
||||
linuxdeploy --appdir AppDir -e '${{ steps.build.outputs.apprun }}' \
|
||||
-d '${{ steps.build.outputs.desktop }}' --plugin qt --output appimage
|
||||
ls -l *.AppImage
|
||||
sha256sum '${{ inputs.target }}.AppImage' | awk '{print $1}' > '${{ inputs.target }}.sha256'
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: inputs.release == 'none'
|
||||
with:
|
||||
name: ${{ inputs.target }}-appimage
|
||||
path: |
|
||||
${{ inputs.target }}.AppImage
|
||||
${{ inputs.target }}.sha256
|
||||
- name: Publish
|
||||
if: inputs.release != 'none'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
gh release upload '${{ inputs.release }}' '${{ inputs.target }}.AppImage' '${{ inputs.target }}.sha256' --clobber
|
||||
echo "published ${{ inputs.target }} $(stat -c %s '${{ inputs.target }}.AppImage') $(cat '${{ inputs.target }}.sha256')"
|
||||
@@ -0,0 +1,65 @@
|
||||
name: Build linuxfs (DroidDeck base runtime)
|
||||
|
||||
# The trimmed base runtime (linuxfs/build-base.sh) as a run artifact, and optionally attached to
|
||||
# a release tag of this repo. The catalog (linuxfs.json on main) is never touched here.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
label:
|
||||
description: "Label for the artifact and tar name (e.g. r10)"
|
||||
required: true
|
||||
default: "r10"
|
||||
turnip_url:
|
||||
description: "URL of the -Linux Turnip zip to ship as the default driver"
|
||||
required: true
|
||||
default: ""
|
||||
release:
|
||||
description: "Release tag to attach linuxfs.tar.zst to (empty = artifact only)"
|
||||
required: false
|
||||
default: ""
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
linuxfs:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 90
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install build dependencies
|
||||
run: |
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y --no-install-recommends curl tar zstd binutils xz-utils unzip python3 proot qemu-user-static
|
||||
- name: Build the rootfs
|
||||
env:
|
||||
TURNIP_URL: ${{ inputs.turnip_url }}
|
||||
run: |
|
||||
mkdir -p out
|
||||
linuxfs/build-base.sh "${RUNNER_TEMP}/linuxfs-work" "${GITHUB_WORKSPACE}/out/linuxfs.tar.zst"
|
||||
- name: Report
|
||||
run: |
|
||||
set -e
|
||||
ls -l out/linuxfs.tar.zst
|
||||
sha256sum out/linuxfs.tar.zst | cut -d' ' -f1 > out/linuxfs.sha256
|
||||
{
|
||||
echo "### linuxfs ${{ inputs.label }}"
|
||||
echo ""
|
||||
echo "| | |"
|
||||
echo "|---|---|"
|
||||
echo "| size | $(stat -c %s out/linuxfs.tar.zst) bytes |"
|
||||
echo "| sha256 | \`$(cat out/linuxfs.sha256)\` |"
|
||||
echo "| turnip | ${{ inputs.turnip_url }} |"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: linuxfs-${{ inputs.label }}
|
||||
path: |
|
||||
out/linuxfs.tar.zst
|
||||
out/linuxfs.sha256
|
||||
out/linuxfs-packages.txt
|
||||
- name: Attach to release
|
||||
if: inputs.release != ''
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: gh release upload '${{ inputs.release }}' out/linuxfs.tar.zst out/linuxfs.sha256 out/linuxfs-packages.txt --clobber
|
||||
@@ -0,0 +1,111 @@
|
||||
name: Record component snapshots
|
||||
|
||||
# Runs a Windows component's installer once, unattended, in a fresh Wine prefix on an x86 runner
|
||||
# and records what it left behind: the registry values it wrote (the point: self-registering
|
||||
# DLLs decide their own entries at run time) and the files it placed, each mapped back to where
|
||||
# the same bytes sit inside the vendor's installer (innoextract / 7-Zip), so a device can pull them
|
||||
# from that installer rather than from a copy hosted here. Recipes: tools/snapshot/recipes.json.
|
||||
# Output per component: <name>.snapshot.json + <name>.summary.md on one release.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
components:
|
||||
description: 'Space-separated component names (keys of tools/snapshot/recipes.json), or all'
|
||||
required: true
|
||||
default: 'all'
|
||||
release:
|
||||
description: 'Release tag to publish to (created if missing)'
|
||||
required: true
|
||||
default: 'component-snapshots-v1'
|
||||
installers_release:
|
||||
description: 'Release the installers are downloaded from'
|
||||
required: true
|
||||
default: 'system-libraries-v1'
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
|
||||
jobs:
|
||||
record:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 300
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install Wine and the extractors
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sudo dpkg --add-architecture i386
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y -qq --no-install-recommends wine wine64 wine32:i386 xvfb p7zip-full \
|
||||
cmake build-essential libboost-iostreams-dev libboost-filesystem-dev libboost-date-time-dev \
|
||||
libboost-system-dev libboost-program-options-dev liblzma-dev >/dev/null
|
||||
wine --version
|
||||
# innoextract from its main branch: the 1.9 release does not read Inno Setup 6.4 (K-Lite).
|
||||
git clone -q --depth 1 https://github.com/dscharrer/innoextract.git /tmp/innoextract
|
||||
cmake -S /tmp/innoextract -B /tmp/innoextract/build -DCMAKE_BUILD_TYPE=Release -DUSE_LZMA=ON >/dev/null
|
||||
cmake --build /tmp/innoextract/build -j"$(nproc)" >/dev/null
|
||||
sudo install -m755 /tmp/innoextract/build/innoextract /usr/local/bin/innoextract
|
||||
innoextract --version | head -1
|
||||
|
||||
- name: Record
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
mkdir -p installers out
|
||||
names='${{ inputs.components }}'
|
||||
if [ "$names" = all ]; then
|
||||
names=$(python3 -c "import json; print(' '.join(k for k in json.load(open('tools/snapshot/recipes.json')) if k != '_'))")
|
||||
fi
|
||||
# Every asset the chosen recipes and their prerequisites name, downloaded once.
|
||||
python3 - $names > assets.txt <<'PY'
|
||||
import json, sys
|
||||
r = json.load(open("tools/snapshot/recipes.json"))
|
||||
wanted = set()
|
||||
def add(name):
|
||||
rec = r[name]
|
||||
for item in rec.get("installers") or [{"asset": rec.get("installer")}]:
|
||||
wanted.add(item["asset"])
|
||||
for dep in rec.get("after", []):
|
||||
add(dep)
|
||||
for name in sys.argv[1:]:
|
||||
add(name)
|
||||
print("\n".join(sorted(wanted)))
|
||||
PY
|
||||
while read -r asset; do
|
||||
[ -n "$asset" ] && gh release download "${{ inputs.installers_release || 'system-libraries-v1' }}" -p "$asset" -D installers --clobber
|
||||
done < assets.txt
|
||||
ls -l installers
|
||||
failed=""
|
||||
for name in $names; do
|
||||
echo "::group::$name"
|
||||
if ! python3 tools/snapshot/record.py --component "$name" --installers-dir installers --out out; then
|
||||
echo "::warning::$name: the recording failed"
|
||||
failed="$failed $name"
|
||||
fi
|
||||
echo "::endgroup::"
|
||||
done
|
||||
ls -l out
|
||||
{ echo "# Recordings"; echo; for f in out/*.summary.md; do sed -n '1,14p' "$f"; echo; done; [ -n "$failed" ] && echo "**Failed:**$failed"; } >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "FAILED=$failed" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Publish
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tag='${{ inputs.release || 'component-snapshots-v1' }}'
|
||||
if ! gh release view "$tag" >/dev/null 2>&1; then
|
||||
gh release create "$tag" --title "Component snapshots" --notes "What each Windows component's installer leaves behind in a Wine prefix, recorded by tools/snapshot/record.py: the registry values it writes and the files it places, mapped back into the vendor's own installer. Read by the apps' Windows components page; one <name>.snapshot.json and <name>.summary.md per component." --latest=false
|
||||
fi
|
||||
gh release upload "$tag" out/*.snapshot.json out/*.summary.md $(ls out/*.files.tar.xz 2>/dev/null) --clobber
|
||||
[ -z "${FAILED:-}" ] || { echo "::error::recordings failed:$FAILED"; exit 1; }
|
||||
|
||||
- name: Keep the recordings as an artifact too
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: snapshots
|
||||
path: out/
|
||||
@@ -0,0 +1,42 @@
|
||||
name: Mirror ARM64 AppImages (SteamDeck)
|
||||
|
||||
# Tier-2 emulators: upstream's own aarch64 Linux builds, copied to our release so the catalog can
|
||||
# point at a URL that does not move and a checksum that was taken here. Nothing is modified.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release:
|
||||
description: 'Release tag to publish to'
|
||||
required: true
|
||||
default: 'steamdeck-desktop-r1'
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
mirror:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- name: Fetch and publish
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
GH_REPO: ${{ github.repository }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p out && cd out
|
||||
# id upstream url. Only what is being added: every file listed is re-fetched and replaces
|
||||
# the release copy, and a moving upstream link (DuckStation's "latest") would no longer
|
||||
# match its checksum in desktop.json. Mirrored before: rpcs3 (build 9e86f165), duckstation
|
||||
# (latest @ 2026-09-20), melonds (1.1), armsx2 (nightly-20260923, 4K pages).
|
||||
while read -r id url; do
|
||||
[ -n "$id" ] || continue
|
||||
ext=${url##*.}
|
||||
curl -fsSL --retry 5 --retry-delay 10 -o "$id.$ext" "$url"
|
||||
sha256sum "$id.$ext" | awk '{print $1}' > "$id.sha256"
|
||||
echo " $id $(stat -c %s "$id.$ext") $(cat "$id.sha256") $id.$ext"
|
||||
done <<'LIST'
|
||||
armsx2-16k https://github.com/ARMSX2/ARMSX2/releases/download/nightly-20260923/ARMSX2-nightly-20260923-92aa2174da-Linux-arm64-16K-pages.AppImage
|
||||
LIST
|
||||
gh release upload '${{ inputs.release }}' ./* --clobber
|
||||
@@ -0,0 +1,185 @@
|
||||
name: Mirror a release
|
||||
|
||||
# Brings a release's assets into this repo by CI: every asset is downloaded from the source
|
||||
# release, checked against what the catalog that points at it promises (md5/size from the Bottles
|
||||
# manifests in components.json, sha256/size in desktop.json, linuxfs.json and steam-seed.json,
|
||||
# the files-archive sha256 inside each recording), uploaded under the same tag here, and its
|
||||
# sha256 confirmed against what GitHub stored. A SHA256SUMS and a MIRROR.md naming this run go
|
||||
# with it. Nothing is uploaded by hand.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
tag:
|
||||
description: 'Release tag (same on the source and here)'
|
||||
required: true
|
||||
source_repo:
|
||||
description: 'Repository the release is mirrored from'
|
||||
required: true
|
||||
default: 'The412Banner/winlator-contents'
|
||||
catalogs:
|
||||
description: 'Space-separated catalog files (or "snapshots") that point at this release, for verification'
|
||||
required: true
|
||||
default: 'components.json'
|
||||
title:
|
||||
description: 'Release title (used when the release is created here)'
|
||||
required: false
|
||||
default: ''
|
||||
upstream:
|
||||
description: 'Fetch vendor files from their original links (Bottles manifests, pinned) rather than the source release'
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
concurrency:
|
||||
group: system-libraries-v1
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
# The Bottles dependency manifests the catalog's checksums came from, at the commit current when
|
||||
# the catalog was made (2026-06-23): their links are where the vendor files are fetched from.
|
||||
BOTTLES_SHA: 8dd62af0e53f9b46e6757193a98c146384b0a0d8
|
||||
|
||||
jobs:
|
||||
mirror:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 300
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Mirror
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
SOURCE: ${{ inputs.source_repo || 'The412Banner/winlator-contents' }}
|
||||
ONE_TAG: ${{ inputs.tag }}
|
||||
ONE_CATALOGS: ${{ inputs.catalogs }}
|
||||
ONE_TITLE: ${{ inputs.title }}
|
||||
UPSTREAM: ${{ inputs.upstream }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The pinned Bottles manifests, for the vendors' own links.
|
||||
bottles=$(mktemp -d)
|
||||
curl -fsSL "https://github.com/bottlesdevs/dependencies/archive/$BOTTLES_SHA.tar.gz" | tar -xz --strip-components=1 -C "$bottles"
|
||||
echo "Bottles manifests at $BOTTLES_SHA: $(find "$bottles" -name '*.yml' | wc -l)"
|
||||
# tag|catalogs|title, one per line; a dispatch mirrors just its own tag.
|
||||
if [ -n "$ONE_TAG" ]; then
|
||||
sets="$ONE_TAG|$ONE_CATALOGS|$ONE_TITLE"
|
||||
else
|
||||
sets=$(cat <<'LIST'
|
||||
system-libraries-v1|components.json|Windows components (system libraries)
|
||||
component-snapshots-v1|snapshots|Component snapshots
|
||||
steamdeck-desktop-r1|desktop.json|Desktop and emulator packages
|
||||
linuxfs-r9|linuxfs.json|Linux runtime r9
|
||||
steam-proton-arm64-25502785|steam-seed.json|Steam seed: Proton Experimental (ARM64) 25502785
|
||||
LIST
|
||||
)
|
||||
fi
|
||||
failed=""
|
||||
echo "$sets" | while IFS='|' read -r tag catalogs title; do
|
||||
tag=$(echo "$tag" | xargs); catalogs=$(echo "$catalogs" | xargs); title=$(echo "$title" | xargs)
|
||||
[ -n "$tag" ] || continue
|
||||
echo "::group::$tag from $SOURCE"
|
||||
work=$(mktemp -d)
|
||||
assets=$(gh release view "$tag" -R "$SOURCE" --json assets -q '.assets[].name' | grep -vE '^(SHA256SUMS|MIRROR.md|BUILT.md)$' || true)
|
||||
count=$(echo "$assets" | grep -c . || true)
|
||||
echo "$count assets on $SOURCE/$tag"
|
||||
# GitHub's stored sha256 of every asset on both sides: an asset already here with the
|
||||
# same digest as the source is not moved again, so a re-run only does what is missing.
|
||||
gh api "repos/$SOURCE/releases/tags/$tag" --paginate --jq '.assets[] | "\(.name) \(.digest // "")"' > "$work/source.digests"
|
||||
gh api "repos/$GITHUB_REPOSITORY/releases/tags/$tag" --paginate --jq '.assets[] | "\(.name) \(.digest // "")"' > "$work/here.digests" 2>/dev/null || : > "$work/here.digests"
|
||||
# Expectations: the catalogs in this checkout, or the recordings themselves.
|
||||
if [ "$catalogs" = snapshots ]; then
|
||||
mkdir -p "$work/snap"
|
||||
echo "$assets" | grep '\.snapshot\.json$' | while read -r name; do
|
||||
gh release download "$tag" -R "$SOURCE" -p "$name" -D "$work/snap" --clobber
|
||||
done
|
||||
python3 tools/mirror/expectations.py "$tag" "$work/snap" > "$work/expect.json"
|
||||
else
|
||||
python3 tools/mirror/expectations.py "$tag" $catalogs > "$work/expect.json"
|
||||
fi
|
||||
# Vendor links for the assets a Bottles manifest describes with the same checksum.
|
||||
echo '{}' > "$work/upstream.json"
|
||||
if [ "$UPSTREAM" = true ] && [ "$catalogs" != snapshots ] && echo "$catalogs" | grep -q components.json; then
|
||||
python3 tools/mirror/upstream.py components.json "$bottles" "$tag" > "$work/upstream.json"
|
||||
echo "$(python3 -c "import json;print(len(json.load(open('$work/upstream.json'))))") assets have a vendor link"
|
||||
fi
|
||||
if ! gh release view "$tag" >/dev/null 2>&1; then
|
||||
gh release create "$tag" --title "${title:-$tag}" --latest=false \
|
||||
--notes "Mirrored by CI from $SOURCE ($tag): $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID. Each asset was checked against the catalog that points at it (see MIRROR.md) and its sha256 confirmed after upload."
|
||||
fi
|
||||
: > "$work/SHA256SUMS"
|
||||
{
|
||||
echo "# Mirror of $SOURCE/$tag"
|
||||
echo
|
||||
echo "Run: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
|
||||
echo "Verified against: $catalogs"
|
||||
echo
|
||||
echo "| asset | size | catalog check | origin |"
|
||||
echo "|---|---|---|---|"
|
||||
} > "$work/MIRROR.md"
|
||||
: > "$work/failed"
|
||||
echo "$assets" | while read -r name; do
|
||||
[ -n "$name" ] || continue
|
||||
case "$name" in SHA256SUMS|MIRROR.md|BUILT.md) continue;; esac
|
||||
src_digest=$(awk -v n="$name" '$1==n{print $2}' "$work/source.digests")
|
||||
here_digest=$(awk -v n="$name" '$1==n{print $2}' "$work/here.digests")
|
||||
vendor=$(python3 -c "import json,sys;print(json.load(open('$work/upstream.json')).get(sys.argv[1],''))" "$name")
|
||||
origin="copied from $SOURCE"
|
||||
if [ -n "$vendor" ]; then
|
||||
# The vendor's own file; the source release is the fallback if the link has died.
|
||||
if curl -fL --retry 3 --retry-delay 15 --max-time 1500 -o "$work/asset" "$vendor"; then
|
||||
origin="fetched from the vendor: $(echo "$vendor" | cut -d/ -f3)"
|
||||
else
|
||||
echo "::warning::$name: vendor link failed ($vendor); taking the source release's copy"
|
||||
origin="copied from $SOURCE (vendor link failed)"
|
||||
rm -f "$work/asset"
|
||||
fi
|
||||
elif [ -n "$src_digest" ] && [ "$src_digest" = "$here_digest" ]; then
|
||||
echo "$name: already here with the same sha256, skipped"
|
||||
echo "${src_digest#sha256:} $name" >> "$work/SHA256SUMS"
|
||||
echo "| $name | | already mirrored (same sha256) | |" >> "$work/MIRROR.md"
|
||||
continue
|
||||
fi
|
||||
if [ ! -s "$work/asset" ]; then
|
||||
for attempt in 1 2 3; do
|
||||
gh release download "$tag" -R "$SOURCE" -p "$name" -O "$work/asset" --clobber && break
|
||||
[ "$attempt" -lt 3 ] || { echo "::error::could not download $name"; exit 1; }
|
||||
sleep 20
|
||||
done
|
||||
fi
|
||||
if out=$(python3 tools/mirror/verify.py "$work/asset" "$name" "$work/expect.json"); then
|
||||
check=$(echo "$out" | sed -n 2p | sed 's/^ *//')
|
||||
else
|
||||
echo "$out"; echo "::error::$name failed verification and was not uploaded"
|
||||
echo "$name" >> "$work/failed"; rm -f "$work/asset"; continue
|
||||
fi
|
||||
echo "$out" | head -1
|
||||
sha=$(sha256sum "$work/asset" | cut -d' ' -f1)
|
||||
size=$(stat -c %s "$work/asset")
|
||||
cp "$work/asset" "$work/$name"
|
||||
for attempt in 1 2 3; do
|
||||
gh release upload "$tag" "$work/$name" --clobber && break
|
||||
[ "$attempt" -lt 3 ] || { echo "::error::could not upload $name"; exit 1; }
|
||||
sleep 20
|
||||
done
|
||||
stored=$(gh api "repos/$GITHUB_REPOSITORY/releases/tags/$tag" --jq ".assets[] | select(.name==\"$name\") | .digest" | sed 's/^sha256://')
|
||||
if [ -n "$stored" ] && [ "$stored" != "$sha" ]; then
|
||||
echo "::error::$name: GitHub stored sha256 $stored, uploaded $sha"; exit 1
|
||||
fi
|
||||
echo "$sha $name" >> "$work/SHA256SUMS"
|
||||
echo "| $name | $size | $check | $origin |" >> "$work/MIRROR.md"
|
||||
rm -f "$work/$name" "$work/asset"
|
||||
done
|
||||
# SHA256SUMS from what GitHub holds now (covers files other workflows built onto this release).
|
||||
gh api "repos/$GITHUB_REPOSITORY/releases/tags/$tag" --paginate --jq '.assets[] | select(.name!="SHA256SUMS" and .name!="MIRROR.md" and .name!="BUILT.md") | "\(.digest|sub("^sha256:";"")) \(.name)"' | sort -k2 > "$work/SHA256SUMS"
|
||||
gh release upload "$tag" "$work/SHA256SUMS" "$work/MIRROR.md" --clobber
|
||||
here=$(gh release view "$tag" --json assets -q '.assets[].name' | grep -vcE '^(SHA256SUMS|MIRROR.md|BUILT.md)$' || true)
|
||||
nfailed=$(grep -c . "$work/failed" || true)
|
||||
echo "$tag: $here of $count assets mirrored; $nfailed refused by verification"
|
||||
{ echo "## $tag"; echo; echo "$here of $count assets mirrored from \`$SOURCE\`, verified against \`$catalogs\`."; [ "$nfailed" = 0 ] || { echo; echo "Refused:"; sed 's/^/- /' "$work/failed"; }; echo; } >> "$GITHUB_STEP_SUMMARY"
|
||||
[ "$here" = "$count" ] && [ "$nfailed" = 0 ] || { echo "::error::$tag: $here of $count assets, $nfailed refused"; exit 1; }
|
||||
rm -rf "$work"
|
||||
echo "::endgroup::"
|
||||
done
|
||||
@@ -34,3 +34,14 @@ repository pins by name and sha256.
|
||||
|
||||
GPL-3.0, the same as DroidDeck. See [LICENSE](LICENSE). The patch series modifies Wine, which is
|
||||
LGPL-2.1-or-later.
|
||||
|
||||
## Catalogs and mirrored releases
|
||||
|
||||
DroidDeck reads its package catalogs from this repository's `main`: `components.json` (Windows
|
||||
components), `desktop.json` (desktop and emulator packages), `linuxfs.json` (the Linux runtime) and
|
||||
`steam-seed.json` (the Steam seed). Every file they point at lives on a release here, produced by a
|
||||
workflow in `.github/workflows/`: `mirror-release.yml` brings an existing release in with every
|
||||
asset checked against the catalog (and leaves `SHA256SUMS` + `MIRROR.md` naming the run),
|
||||
`component-snapshots.yml` records what a Windows component's installer leaves behind,
|
||||
`build-desktop.yml`, `build-emulator-source.yml`, `mirror-appimages.yml` and `build-linuxfs.yml`
|
||||
build the rest. Nothing on a release is uploaded by hand.
|
||||
+5007
File diff suppressed because it is too large.
Load diff
+115
@@ -0,0 +1,115 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"category": "steamdeck-desktop",
|
||||
"release": "steamdeck-desktop-r1",
|
||||
"note": "Hosted packages the SteamDeck app installs into its Linux runtime on request. A tar package extracts over the r9 rootfs; an appimage lands in /opt/appimages with a .desktop entry and runs extracted (proot has no FUSE). Tier 1 is packaged natively for aarch64 by Arch Linux ARM; tier 2 is upstream's own ARM64 build, mirrored here unmodified with the checksum taken at mirror time; tier 3 is built from source by us.",
|
||||
"packages": [
|
||||
{
|
||||
"id": "desktop",
|
||||
"name": "Desktop (LXQt on labwc)",
|
||||
"tier": 1,
|
||||
"version": "r1",
|
||||
"kind": "tar",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/desktop.tar.zst",
|
||||
"sha256": "c2bb860b3b1688287451e2495be5db313ec07f8c8af501dfac2768a9f417bc86",
|
||||
"size": 426049032,
|
||||
"notes": "panel, file manager, terminal, Firefox, mpv; Steam as a window or Big Picture from an icon"
|
||||
},
|
||||
{
|
||||
"id": "emulators",
|
||||
"name": "Emulators (PPSSPP, RetroArch + cores)",
|
||||
"tier": 1,
|
||||
"version": "r1",
|
||||
"kind": "tar",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/emulators.tar.zst",
|
||||
"sha256": "a1f4806a9663d28f424b4cab1277e0ede80a92e69de5bc063cdc8b45af220164",
|
||||
"size": 727860354,
|
||||
"notes": "PSP; GameCube/Wii (Dolphin), PS1, PS2 (Play!), Dreamcast, DS, N64, GBA, SNES, Genesis, MAME, ScummVM cores; DOSBox, Mednafen"
|
||||
},
|
||||
{
|
||||
"id": "duckstation",
|
||||
"name": "DuckStation (PS1)",
|
||||
"tier": 2,
|
||||
"version": "latest-20260920",
|
||||
"kind": "appimage",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/duckstation.AppImage",
|
||||
"sha256": "d67758c25fe3772558662e39bb508620c997cc6cd3d66ecca835e754923f158a",
|
||||
"size": 85995776,
|
||||
"icon": "duckstation",
|
||||
"category": "Game",
|
||||
"notes": "standalone, upstream ARM64 build"
|
||||
},
|
||||
{
|
||||
"id": "melonds",
|
||||
"name": "melonDS 1.1 (DS)",
|
||||
"tier": 2,
|
||||
"version": "1.1",
|
||||
"kind": "appimage",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/melonds.zip",
|
||||
"sha256": "c537ae018d6dcedfe9da0317a1d5c0163e69201e23abfa163fa2a63e437ace09",
|
||||
"size": 33391635,
|
||||
"icon": "melonDS",
|
||||
"category": "Game",
|
||||
"notes": "standalone, upstream ARM64 build"
|
||||
},
|
||||
{
|
||||
"id": "rpcs3",
|
||||
"name": "RPCS3 (PS3)",
|
||||
"tier": 2,
|
||||
"version": "0.0.42-20024",
|
||||
"kind": "appimage",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/rpcs3.AppImage",
|
||||
"sha256": "bce484ff7071badb209cfb8b061c486294e60ef776121173e129770d76efb8a5",
|
||||
"size": 80330873,
|
||||
"icon": "rpcs3",
|
||||
"category": "Game",
|
||||
"notes": "upstream ARM64 build; expect only light titles on a phone"
|
||||
},
|
||||
{
|
||||
"id": "armsx2",
|
||||
"name": "ARMSX2 (PS2, fast on ARM64)",
|
||||
"tier": 2,
|
||||
"version": "nightly-20260923-92aa2174da",
|
||||
"kind": "appimage",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/armsx2.AppImage",
|
||||
"sha256": "629ebcef049ebe03c0ddefd4e0667dd38e6e242fe9aa275f6cb4d19f6facf64e",
|
||||
"size": 57281032,
|
||||
"icon": "pcsx2",
|
||||
"category": "Game",
|
||||
"notes": "PCSX2 with ARM64 recompilers (github.com/ARMSX2/ARMSX2, GPL-3.0); upstream Linux arm64 4K-pages nightly, mirrored unmodified. Much faster than PCSX2 here, which interprets the PS2's CPUs on ARM64",
|
||||
"pages": {
|
||||
"16384": {
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/armsx2-16k.AppImage",
|
||||
"sha256": "74783f471b95f72132a81e0e573235e8882e57e6653fb94cef04672f68851033",
|
||||
"size": 57264648
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "dolphin",
|
||||
"name": "Dolphin (GameCube / Wii)",
|
||||
"tier": 3,
|
||||
"version": "master-20260920",
|
||||
"kind": "appimage",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/dolphin.AppImage",
|
||||
"sha256": "0df02c52e89e3740858c14397b11a321a3728c818cd2755505642e1cf006da85",
|
||||
"size": 85973512,
|
||||
"icon": "dolphin-emu",
|
||||
"category": "Game",
|
||||
"notes": "built from source for ARM64; no upstream ARM64 build exists"
|
||||
},
|
||||
{
|
||||
"id": "cemu",
|
||||
"name": "Cemu (Wii U)",
|
||||
"tier": 3,
|
||||
"version": "main-20260924",
|
||||
"kind": "appimage",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steamdeck-desktop-r1/cemu.AppImage",
|
||||
"sha256": "9628000a90a91784d223fc57fad64f2662da3028b941d13221a3ff969c3590a4",
|
||||
"size": 64170504,
|
||||
"icon": "info.cemu.Cemu",
|
||||
"category": "Game",
|
||||
"notes": "built from source for ARM64 (Cemu c717fca); no upstream ARM64 build exists. Rumble is optional, so a controller without force feedback no longer stops it at startup."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"category": "linux-runtime",
|
||||
"release": "linuxfs-r9",
|
||||
"name": "Linux runtime (gamescope + native ARM Steam)",
|
||||
"version": "r9",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/linuxfs-r9/linuxfs.tar.zst",
|
||||
"sha256": "d3240c1531531615b0dd06b6ad56e92db9015a8641eb18c3a613835ae45992e7",
|
||||
"size": 790907606,
|
||||
"file_size": 790907606,
|
||||
"note": "Arch Linux ARM aarch64 rootfs with gamescope 3.16, Xwayland, Mesa 26.2 (Turnip with the KGSL backend, Zink), GTK 2, PulseAudio and Bannerlator's session scripts. Installed by the app into files/linuxfs; a gamescope session refuses to start without it. r9 makes a first run reach a game unaided: it creates the ~/.steam/sdkarm64 helper path the arm64 client launches every game through (missing before, so launches died before Proton was invoked), names the ARM64 Proton up front so the client offers and fetches it, and registers that tool in-session when it lands instead of needing a restart. The session also prints a milestone per stage. From r8: the libraries Proton's GStreamer plugins need. From r7: games are told the panel's real refresh rate. From r6: installed Windows titles are set to the ARM64 Proton automatically. From r5: the same proot at opt/android-host, so the runtime installs and runs without root and an app update cannot replace it; the tool is listed as Bannerlator Proton (ARM64) so it is not mistaken for Valve's own ARM64 Protons, which need user namespaces Android does not give. Contains no Valve software - the native arm64 Steam client is fetched from Valve at first use. Built from tools/linuxfs in the Bannerlator repo; ported from WinNative's gamescope runtime (GPL-3.0)."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"category": "steam-seed",
|
||||
"note": "Placed by DroidDeck over its Linux runtime before the Steam client's first start. Not a user-installable package; kept out of desktop.json so it never shows in the package list.",
|
||||
"packages": [
|
||||
{
|
||||
"id": "proton-arm64",
|
||||
"name": "Proton Experimental (ARM64)",
|
||||
"version": "25502785",
|
||||
"kind": "tar",
|
||||
"url": "https://github.com/Droid-Deck/DroidDeck-Components/releases/download/steam-proton-arm64-25502785/proton-experimental-arm64-25502785.tar.zst",
|
||||
"sha256": "d6c480a90d981801be0906a3f3418eb9bd9461c30e411140951cb0daf9d2a189",
|
||||
"size": 398113534,
|
||||
"notes": "Valve's Proton Experimental (ARM64), depot 4427311 manifest 6971993485335684515, with appmanifest_4427310.acf (LastOwner 0). The client adopts it at first sign-in and updates it itself."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Builds the DLL bundles DroidDeck's Windows components lay out, from the vendors' own packages.
|
||||
|
||||
A bundle is a small archive of DLLs in win32/ and win64/ folders that a catalog entry extracts
|
||||
and copies into a prefix. The ones on system-libraries-v1 were made by hand in June 2026; this
|
||||
rebuilds each from the vendor package named in tools/bundles/recipes.json (the Bottles manifest's
|
||||
source, with its md5), by opening the package and picking the same files. The result must match
|
||||
the hand-made bundle file for file (tools/bundles/expect.json, sha256 per file) before it is
|
||||
packed; a difference is reported and the bundle refused, so a rebuilt bundle is never silently
|
||||
different from what users have been installing.
|
||||
|
||||
build.py --recipes recipes.json --expect expect.json --out out/ [--only name ...] [--source-dir dir]
|
||||
|
||||
Sources are downloaded (or taken from --source-dir, where a mirror of the package may already
|
||||
sit), md5-checked, and opened once each: .exe/.cab/.dl_ with cabextract or 7-Zip, .zip with
|
||||
7-Zip, VC++ installers through DroidDeck's own unpacker (droiddeck-msi-install --unpack, which
|
||||
opens burn bundles and cab-SFX wrappers the way the phone does). Archives are packed
|
||||
reproducibly: sorted names, zero timestamps, no owner.
|
||||
"""
|
||||
import argparse
|
||||
import fnmatch
|
||||
import glob
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.request
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def sha256_of(path):
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as source:
|
||||
for block in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def md5_of(path):
|
||||
digest = hashlib.md5()
|
||||
with open(path, "rb") as source:
|
||||
for block in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def run(cmd, **kw):
|
||||
return subprocess.run(cmd, capture_output=True, text=True, errors="replace", **kw)
|
||||
|
||||
|
||||
class Sources:
|
||||
"""Downloads and opens each vendor package once."""
|
||||
|
||||
def __init__(self, cache, source_dir, unpacker):
|
||||
self.cache = Path(cache)
|
||||
self.cache.mkdir(parents=True, exist_ok=True)
|
||||
self.source_dir = Path(source_dir) if source_dir else None
|
||||
self.unpacker = unpacker
|
||||
self.opened = {}
|
||||
self.log = []
|
||||
|
||||
def fetch(self, source):
|
||||
name, url, md5 = source["name"], source["url"], source.get("md5", "").lower()
|
||||
target = self.cache / name
|
||||
if not target.is_file():
|
||||
local = self.source_dir / name if self.source_dir else None
|
||||
if local and local.is_file():
|
||||
shutil.copyfile(local, target)
|
||||
origin = "from the source mirror"
|
||||
else:
|
||||
for attempt in range(1, 5):
|
||||
try:
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "DroidDeck-Components bundle builder"})
|
||||
with urllib.request.urlopen(request, timeout=120) as response, open(target, "wb") as out:
|
||||
shutil.copyfileobj(response, out, 1 << 20)
|
||||
break
|
||||
except Exception as error: # noqa: BLE001
|
||||
if attempt == 4:
|
||||
raise RuntimeError("could not download %s: %s" % (url, error))
|
||||
time.sleep(15 * attempt)
|
||||
origin = "fetched from %s" % url.split("/")[2]
|
||||
actual = md5_of(target)
|
||||
if md5 and actual != md5:
|
||||
target.unlink()
|
||||
raise RuntimeError("%s: md5 %s, the recipe says %s" % (name, actual, md5))
|
||||
self.log.append("%s: %s, md5 %s%s" % (name, origin, actual, "" if md5 else " (unchecked: no recipe md5)"))
|
||||
return target
|
||||
|
||||
def open(self, source):
|
||||
"""The package's contents on disk: a folder, opened once per package."""
|
||||
name = source["name"]
|
||||
if name in self.opened:
|
||||
return self.opened[name]
|
||||
package = self.fetch(source)
|
||||
folder = self.cache / (name + ".opened")
|
||||
if not folder.is_dir():
|
||||
folder.mkdir()
|
||||
how = source.get("open", "auto")
|
||||
if how == "installer":
|
||||
result = run(["python3", self.unpacker, "--unpack", str(package), str(folder)])
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError("%s: unpacker failed: %s" % (name, result.stderr[-400:]))
|
||||
elif how == "install":
|
||||
# DroidDeck's full layout of the installer (drive_c with real file names).
|
||||
result = run(["python3", self.unpacker, str(package), str(folder)])
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError("%s: install layout failed: %s" % (name, result.stderr[-400:]))
|
||||
elif package.suffix.lower() == ".zip" or how == "zip":
|
||||
with zipfile.ZipFile(package) as archive:
|
||||
archive.extractall(folder)
|
||||
elif how in ("auto", "cab") and run(["cabextract", "-q", "-d", str(folder), str(package)]).returncode == 0 \
|
||||
and any(folder.rglob("*")):
|
||||
pass
|
||||
else:
|
||||
result = run(["7z", "x", "-y", "-bso0", "-bsp0", "-o" + str(folder), str(package)])
|
||||
if result.returncode != 0 or not any(folder.rglob("*")):
|
||||
raise RuntimeError("%s: neither cabextract nor 7-Zip opened it" % name)
|
||||
# Cabinets inside (DirectX's per-feature cabs, SP1's nested cab, .dl_ files): opened
|
||||
# in place so a recipe can name a file by its path inside.
|
||||
self.open_nested(folder, 0)
|
||||
self.opened[name] = folder
|
||||
return folder
|
||||
|
||||
def open_nested(self, folder, depth):
|
||||
if depth > 2:
|
||||
return
|
||||
for path in sorted(p for p in folder.rglob("*") if p.is_file()):
|
||||
if path.name.endswith(".opened") or path.suffix.lower() not in (".cab", ".exe", ".msi", ".dl_", ".ex_", ".zip", ".cpl"):
|
||||
continue
|
||||
if path.suffix.lower() in (".exe", ".cpl") and run(["cabextract", "-l", str(path)]).returncode != 0:
|
||||
continue # a plain program, not a cabinet wrapper
|
||||
out = path.with_name(path.name + ".opened")
|
||||
if out.exists():
|
||||
continue
|
||||
out.mkdir()
|
||||
ok = run(["cabextract", "-q", "-d", str(out), str(path)]).returncode == 0 and any(out.rglob("*"))
|
||||
if not ok:
|
||||
shutil.rmtree(out, ignore_errors=True)
|
||||
out.mkdir()
|
||||
ok = run(["7z", "x", "-y", "-bso0", "-bsp0", "-o" + str(out), str(path)]).returncode == 0 and any(out.rglob("*"))
|
||||
if ok:
|
||||
self.open_nested(out, depth + 1)
|
||||
else:
|
||||
shutil.rmtree(out, ignore_errors=True)
|
||||
|
||||
|
||||
def find(folder, patterns, expected=None, log=None):
|
||||
"""The file under folder matching one of the globs (tried in order; relative, / separated,
|
||||
case-insensitive; nested '.opened' folders count as path parts). When a package holds several
|
||||
copies (Windows 7 SP1 carries more than one gdiplus.dll), the one with the expected sha256 is
|
||||
taken and its path logged, so the recipe can be pinned to it."""
|
||||
if isinstance(patterns, str):
|
||||
patterns = [patterns]
|
||||
for pattern in patterns:
|
||||
pattern = pattern.lower()
|
||||
hits = []
|
||||
for path in folder.rglob("*"):
|
||||
if path.is_file():
|
||||
rel = path.relative_to(folder).as_posix().lower()
|
||||
if fnmatch.fnmatch(rel, pattern) or fnmatch.fnmatch(rel, "*/" + pattern):
|
||||
hits.append(path)
|
||||
if not hits:
|
||||
continue
|
||||
hits.sort(key=lambda p: (len(p.relative_to(folder).parts), str(p)))
|
||||
if len(hits) > 1 and expected:
|
||||
for hit in hits:
|
||||
if sha256_of(hit) == expected:
|
||||
if log is not None:
|
||||
log.append("%s: %d copies match, took %s (the expected one)" % (pattern, len(hits), hit.relative_to(folder).as_posix()))
|
||||
return hit
|
||||
return hits[0]
|
||||
raise RuntimeError("no file matches %s" % " | ".join(patterns))
|
||||
|
||||
|
||||
def pack_tar(tree, out):
|
||||
with tarfile.open(out, "w:xz", preset=9) as archive:
|
||||
for path in sorted(p for p in Path(tree).rglob("*") if p.is_file()):
|
||||
info = archive.gettarinfo(str(path), path.relative_to(tree).as_posix())
|
||||
info.mtime, info.uid, info.gid, info.uname, info.gname, info.mode = 0, 0, 0, "", "", 0o644
|
||||
with open(path, "rb") as source:
|
||||
archive.addfile(info, source)
|
||||
|
||||
|
||||
def pack_zip(tree, out):
|
||||
with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as archive:
|
||||
for path in sorted(p for p in Path(tree).rglob("*") if p.is_file()):
|
||||
info = zipfile.ZipInfo(path.relative_to(tree).as_posix(), date_time=(1980, 1, 1, 0, 0, 0))
|
||||
info.compress_type = zipfile.ZIP_DEFLATED
|
||||
info.external_attr = 0o644 << 16
|
||||
archive.writestr(info, path.read_bytes())
|
||||
|
||||
|
||||
def build_one(name, recipe, expect, sources, out_dir):
|
||||
if recipe.get("skip"):
|
||||
return None, recipe["skip"]
|
||||
tree = Path(tempfile.mkdtemp(prefix="bundle-"))
|
||||
try:
|
||||
for item in recipe["files"]:
|
||||
patterns = item["from"] if isinstance(item["from"], list) else [item["from"]]
|
||||
source = next(s for s in recipe["sources"] if s["name"] == patterns[0].split(":", 1)[0])
|
||||
folder = sources.open(source)
|
||||
src = find(folder, [p.split(":", 1)[1] for p in patterns], expect.get(name, {}).get(item["to"]), sources.log)
|
||||
target = tree / item["to"]
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(src, target)
|
||||
built = {p.relative_to(tree).as_posix(): sha256_of(p) for p in tree.rglob("*") if p.is_file()}
|
||||
wanted = expect.get(name, {})
|
||||
accepted = ""
|
||||
if built != wanted:
|
||||
missing = sorted(set(wanted) - set(built))
|
||||
extra = sorted(set(built) - set(wanted))
|
||||
differ = sorted(k for k in set(wanted) & set(built) if wanted[k] != built[k])
|
||||
if recipe.get("accept_different") and not missing and not extra:
|
||||
accepted = "; differs from the hand-made bundle in %d file(s), accepted: %s" % (len(differ), recipe["accept_different"])
|
||||
else:
|
||||
return False, "does not match the hand-made bundle: missing %s, extra %s, different %s" % (missing, extra, differ)
|
||||
out = Path(out_dir) / name
|
||||
if name.endswith(".tar.xz"):
|
||||
pack_tar(tree, out)
|
||||
elif name.endswith(".zip"):
|
||||
pack_zip(tree, out)
|
||||
else:
|
||||
shutil.copyfile(tree / recipe["files"][0]["to"], out)
|
||||
return True, "%d files, %d bytes, %s%s" % (len(built), out.stat().st_size, sha256_of(out), accepted)
|
||||
finally:
|
||||
shutil.rmtree(tree, ignore_errors=True)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--recipes", required=True)
|
||||
parser.add_argument("--expect", required=True)
|
||||
parser.add_argument("--out", required=True)
|
||||
parser.add_argument("--cache", default=None)
|
||||
parser.add_argument("--source-dir", default=None)
|
||||
parser.add_argument("--unpacker", default="droiddeck-msi-install")
|
||||
parser.add_argument("--only", nargs="*")
|
||||
args = parser.parse_args()
|
||||
recipes = json.loads(Path(args.recipes).read_text("utf-8"))
|
||||
expect = json.loads(Path(args.expect).read_text("utf-8"))
|
||||
Path(args.out).mkdir(parents=True, exist_ok=True)
|
||||
sources = Sources(args.cache or tempfile.mkdtemp(prefix="sources-"), args.source_dir, args.unpacker)
|
||||
names = [n for n in recipes if n != "_" and (not args.only or n in args.only)]
|
||||
results = {}
|
||||
for name in names:
|
||||
print("== %s" % name, flush=True)
|
||||
try:
|
||||
ok, note = build_one(name, recipes[name], expect, sources, args.out)
|
||||
except Exception as error: # noqa: BLE001
|
||||
ok, note = False, "failed: %s" % error
|
||||
results[name] = (ok, note)
|
||||
print(" %s %s" % ("built" if ok else ("skipped" if ok is None else "REFUSED"), note), flush=True)
|
||||
print("\n== sources")
|
||||
for line in sources.log:
|
||||
print(" " + line)
|
||||
Path(args.out, "BUILT.md").write_text(
|
||||
"# DLL bundles built from the vendors' packages\n\n| bundle | result |\n|---|---|\n" +
|
||||
"".join("| %s | %s %s |\n" % (n, "built:" if ok else ("skipped:" if ok is None else "REFUSED:"), note) for n, (ok, note) in results.items()) +
|
||||
"\n## Sources\n\n" + "".join("- %s\n" % line for line in sources.log), "utf-8")
|
||||
bad = [n for n, (ok, _) in results.items() if ok is False]
|
||||
skipped = [n for n, (ok, _) in results.items() if ok is None]
|
||||
print("\n%d built, %d skipped, %d refused%s" % (len(results) - len(bad) - len(skipped), len(skipped), len(bad), ": " + " ".join(bad) if bad else ""))
|
||||
return 1 if bad else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,226 @@
|
||||
{
|
||||
"amstream__libs.tar.xz": {
|
||||
"win32/amstream.dll": "26012b03ad7c0802a2f26bf89c6510a78f6f4ae44d5e5eed164e22db7db334f0",
|
||||
"win64/amstream.dll": "fed6c36838e0a11cd0c003eeeaf8ef0e1c1f005c03a80a89165fc2da0c9693d1"
|
||||
},
|
||||
"atmlib__libs.tar.xz": {
|
||||
"win32/atmlib.dll": "dd521f1af21c53158ceabc1b6d95a9ef978dff84410c568ed5763b601c9ed2d8"
|
||||
},
|
||||
"d3dcompiler_42__libs.tar.xz": {
|
||||
"win32/d3dcompiler_42.dll": "6c976311406c23aa71018d274da0ecdef43b6e3a3b0b01e941a5e8e4e974386c",
|
||||
"win64/d3dcompiler_42.dll": "4d61e843aa86d8801a60d01b8e872379c5b2503795a2bcbe6141978396ade00b"
|
||||
},
|
||||
"d3dcompiler_46__libs.tar.xz": {
|
||||
"win32/d3dcompiler_46.dll": "58d9a00888af693b2a5222fe74cfded32ce83e74f85b474f1cbe5987217b5a9d",
|
||||
"win64/d3dcompiler_46.dll": "b1a06f7aa52439a948a152bfd3301d9b595c78969bb77c9741bba935139f58a0"
|
||||
},
|
||||
"devenum__libs.tar.xz": {
|
||||
"win32/devenum.dll": "ab49f2ebb9f99b640c14a4a1d830b35685aa758c7b1f5c62d77fdb6e09081387",
|
||||
"win64/devenum.dll": "ab49f2ebb9f99b640c14a4a1d830b35685aa758c7b1f5c62d77fdb6e09081387"
|
||||
},
|
||||
"dx8vb__libs.tar.xz": {
|
||||
"win32/dx8vb.dll": "b1c26cae457ea67bf2d54f8762c7ff98230935818e26f2cf9ae3350ea61dd91d"
|
||||
},
|
||||
"gdiplus__libs.tar.xz": {
|
||||
"win32/gdiplus.dll": "c07a936009fb275d56dc451c7d05b5f5a3f4dcd985c7f02dee57d674034c602b",
|
||||
"win64/gdiplus.dll": "980ea189929d95eb36e35980fff0c81f7b78de9422771fde8f4ac7a779f5bd89"
|
||||
},
|
||||
"gmdls__libs.tar.xz": {
|
||||
"win64/drivers/gm.dls": "3229b09b9d7d9f3f4793b0d9b34fe6abc75cfa4a2503c0c90f43ff651ba7f2c0",
|
||||
"win32/drivers/gm.dls": "3229b09b9d7d9f3f4793b0d9b34fe6abc75cfa4a2503c0c90f43ff651ba7f2c0"
|
||||
},
|
||||
"iertutil__libs.tar.xz": {
|
||||
"win32/iertutil.dll": "f5cd7de4b315f318d495c4e28292fa47b96640d044502131934190ec58f3d1d5",
|
||||
"win64/iertutil.dll": "89b894eccf65704d00d30ea3bd45b184bfab8345b779f9ae2be66b9fc7226f72"
|
||||
},
|
||||
"mediafoundation__libs.tar.xz": {
|
||||
"win32/mf.dll": "045846267bcb1d9c4931a4871c887d8496e92df655936dd5d40375e09c950510",
|
||||
"win64/mf.dll": "dbf0458bed308078edd82c81e6d1209e111c5bfa5d64dd068cedfd9e0a6c05ab"
|
||||
},
|
||||
"mfc40__libs.tar.xz": {
|
||||
"win32/mfc40.dll": "b44b89aa6d5dc1790e1e6043ff2569f1ec31142e7659c42735a3e04c2256ffed",
|
||||
"win32/mfc40u.dll": "02b8e9ee0b44526f74c3e6d072a2c4ab417c958774e6cbede9064ea66110e123"
|
||||
},
|
||||
"mfc42__libs.tar.xz": {
|
||||
"win32/mfc42u.dll": "ea9227556f96415983065a476207cff2b36056b4010a358837a35c21cf32c877"
|
||||
},
|
||||
"msasn1__libs.tar.xz": {
|
||||
"win64/msasn1.dll": "8d2b6750e90a27bfa916520cc3a52fbeac317aea40662bc0d9b71e4de34c1207"
|
||||
},
|
||||
"msftedit__libs.tar.xz": {
|
||||
"win32/msftedit.dll": "04ed66befdb822181ebd1d84cbf0b17aaadf8455ae742f44d7adcb26ab07bdad",
|
||||
"win64/msftedit.dll": "e15ed4fefc3010c213694331ddfdc03767682325c898d773ab243e2dc8b08461"
|
||||
},
|
||||
"msls31__libs.tar.xz": {
|
||||
"win32/msls31.dll": "841fde9b24476a7ed364a3e4a1470ac9b7358bc92f29fca4a06aab557d140850"
|
||||
},
|
||||
"mspatcha__libs.tar.xz": {
|
||||
"win32/mspatcha.dll": "202366e216cf3139ae49125a1614dc07dd2068f7a13d9570e5336f7532dd5e71"
|
||||
},
|
||||
"msxml6__libs.tar.xz": {
|
||||
"win32/msxml6.dll": "f4174184a7420fe3d902836dc6c1ed0e54c28b5deb94ba08637df041b0926d95",
|
||||
"win32/msxml6r.dll": "7ad08fdda21796679d070570713f91d798419b288412a0740e67c6b72d3febff",
|
||||
"win64/msxml6.dll": "59d4bc37687ff06a26d202b284d048bdb6a14802d4e38f7923f2a1c8653e3889",
|
||||
"win64/msxml6r.dll": "5647540a43d3b9f764e4d9bce0d14b5d9e17af59b394b7abfea5377e3dce699f"
|
||||
},
|
||||
"oalinst_dll__libs.tar.xz": {
|
||||
"win64/OpenAL32.dll": "3963b06e319180700be0bcc0d027336664bb3a4975876b4a5442f10746fab5b4",
|
||||
"win32/OpenAL32.dll": "606cf8b4c22c7ae00813585279ac045115907bc66d1483eb26355907021ea4b0"
|
||||
},
|
||||
"powershell__libs.tar.xz": {
|
||||
"win32/powershell.exe": "212236f43b05c5e803b8bfb05c3137456ba5f799df9a442a3390fda755a35ea8",
|
||||
"win64/powershell.exe": "f03523112548531e513cc64872d426b0752f96f653cdcd6785567c6ef65f15f7",
|
||||
"profile.ps1": "9fdbd47ca304072b8444dda2f4338c61dab227248837d6952593bb2a8b95c80b"
|
||||
},
|
||||
"qasf__libs.tar.xz": {
|
||||
"win32/qasf.dll": "baf44e2a33f86d8a5ec848804afcb7771d55d42a443980fce45bfecf71374dc8",
|
||||
"win64/qasf.dll": "45f6313ef38d5488cef7b051895f19446b5c8d6c1de0847911389bcc84d0a608"
|
||||
},
|
||||
"qcap__libs.tar.xz": {
|
||||
"win32/qcap.dll": "aff680e62d989a62cbdec2bf70b6d17f7615f9826eaee0c8a524af80f9ff862d",
|
||||
"win64/qcap.dll": "44799a5f70c8610fdf371b159c7f7035cbf97038eed09e8bce0bdb90b63b0d9e"
|
||||
},
|
||||
"qdvd__libs.tar.xz": {
|
||||
"win32/qdvd.dll": "b6a8105e4432e1778db2a832e1f93bc34f0c59ef955945dd8bbae835278598b9",
|
||||
"win64/qdvd.dll": "7218f46b6b42bce78b2e1524cb21d9622bfe9b7bf544fb95aa17df373202fa94"
|
||||
},
|
||||
"qedit__libs.tar.xz": {
|
||||
"win32/qedit.dll": "38fd8f09cee1bb17782c6d54222b32f41acbd4ea62af4104e5f6efc2c27cbc99",
|
||||
"win64/qedit.dll": "e90693ecc0c5e146b13ef65c33ca1587b2d923a0f3cbbd15bcb65f17f8f9795f"
|
||||
},
|
||||
"quartz__libs.tar.xz": {
|
||||
"win32/quartz.dll": "fa52a0d0647413deeef57c5cb632f73a97a48588c16877fc1cc66404c3c21a2b",
|
||||
"win64/quartz.dll": "701e6047d6cc30feada589eb65cefa95fd708031dcb8417d78ef2639ffa3efdd"
|
||||
},
|
||||
"riched20__libs.tar.xz": {
|
||||
"win32/riched20.dll": "c741226a0465a8c4edcbe0f3af54de02e21931122afe3a4dad8d49553fececcc"
|
||||
},
|
||||
"sqlite3.zip": {
|
||||
"sqlite3.dll": "ec0debfaaa036fe1421f09da809848820f9a36d35edf09317f81a776379bc42b",
|
||||
"sqlite3.def": "02e76d26e9afcb06313980ce577144a1d928ed39c4d189ad8652dca34712fc77"
|
||||
},
|
||||
"urlmon__libs.tar.xz": {
|
||||
"win32/urlmon.dll": "669bed2f3a2cffb5e25fe6cf6956e61036fa1d9127ae59f05b22cfd57cea3e2a",
|
||||
"win64/urlmon.dll": "2590e88cab52fcc1b24cb262d293131c6280a5f234e0c130e77aa8697efa3b5f"
|
||||
},
|
||||
"vcredist2005_dll__libs.tar.xz": {
|
||||
"win64/msvcm80.dll": "835195907e9f3731a07e590e4bf15a3ebce17f53d3bb6cd6ef7cfb26fbde1f55",
|
||||
"win64/msvcp80.dll": "40f6502679cee0b657b0005278fbe7213bdda6deaacf868058e17737c182e1b4",
|
||||
"win64/msvcr80.dll": "690f12c490bee2bf17ab7b6804e6e9b96f51c304350ccde80fe5c7eefa89720e",
|
||||
"win32/msvcm80.dll": "7dcb3c45938d31854e46b5e5b0e16d538e29230d1bc81086d40c8db3bdf510bc",
|
||||
"win32/msvcp80.dll": "372af797353f9335915cd06d4076bab8410775dcaf2dac0593197d7c41bbffb2",
|
||||
"win32/msvcr80.dll": "c3ac989c8489a23bb96400b1856f5325ffc67e844f04651ea5d61bc20a991c6d"
|
||||
},
|
||||
"vcredist2008_dll__libs.tar.xz": {
|
||||
"win64/msvcm90.dll": "47b4e80f1173532365fceedaeeb495bdaefac4d362eb62bb92f4fea589d2c104",
|
||||
"win64/msvcp90.dll": "b3f4fda27430acc6d6bc1c3cbd518b9cae5ba0f22ab8917578a7f16270f94c8b",
|
||||
"win64/msvcr90.dll": "5330682ae9c08e5f2e30c5e256b91028389bbbddaa8c38950df76616fca854ff",
|
||||
"win32/msvcm90.dll": "7a74da389fbd10a710c294c2e914dc6f18e05f028f07958a2fa53ac44f0e4b90",
|
||||
"win32/msvcp90.dll": "06918cf99ad26cd6cf106881c0d5bdb212dc0bac4549805c9f5906e3d03d152c",
|
||||
"win32/msvcr90.dll": "8e7fe1a1f3550c479ffd86a77bc9d10686d47f8727025bb891d8f4f0259354c8"
|
||||
},
|
||||
"vcredist2010_dll__libs.tar.xz": {
|
||||
"win64/msvcp100.dll": "5bb420fbe28315f2117376052bb8488ce84a3398dda65005b8ae1f792017e9a8",
|
||||
"win64/msvcr100.dll": "1d1a1ae540ba132f998d60d3622f0297b6e86ae399332c3b47462d7c0f560a36",
|
||||
"win32/msvcp100.dll": "d769fafa2b3232de9fa7153212ba287f68e745257f1c00fafb511e7a02de7adf",
|
||||
"win32/msvcr100.dll": "60c06e0fa4449314da3a0a87c1a9d9577df99226f943637e06f61188e5862efa"
|
||||
},
|
||||
"vcredist2012_dll__libs.tar.xz": {
|
||||
"win64/msvcp110.dll": "27f394ae01d12f851f1dee3632dee3c5afa1d267f7a96321d35fd43105b035ad",
|
||||
"win64/msvcr110.dll": "ae996edb9b050677c4f82d56092efdc75f0addc97a14e2c46753e2db3f6bd732",
|
||||
"win32/msvcp110.dll": "c8d5572ca8d7624871188f0acabc3ae60d4c5a4f6782d952b9038de3bc28b39a",
|
||||
"win32/msvcr110.dll": "b30160e759115e24425b9bcdf606ef6ebce4657487525ede7f1ac40b90ff7e49"
|
||||
},
|
||||
"vcredist2013_dll__libs.tar.xz": {
|
||||
"win64/msvcp120.dll": "3b92e964404e3f94531e7d7c4c7419561d9eca6accd98dc3979c9e3596db444c",
|
||||
"win64/msvcr120.dll": "753ac30c30aae62415cc225e3d057b8b6254afe280696e0a43f1a7c3132632a6",
|
||||
"win32/msvcp120.dll": "0e535e147d253fc645ccb9406c888aee374ab1f3f9d99dc057a2b19cf401458b",
|
||||
"win32/msvcr120.dll": "82225ec7e2da43a7a72a3d523698747512523afa488767ca6839c63a7a5706fa"
|
||||
},
|
||||
"vcredist2022_dll__libs.tar.xz": {
|
||||
"win64/concrt140.dll": "b2faf3b85b23c840b654e57d5497a0ad31acd02fb01856cad4725a1715d5f78e",
|
||||
"win64/msvcp140.dll": "def46aa6a8f72f27bafac0c43334419486a4d1dcdb6c479a8ef7034b3e1fa4cb",
|
||||
"win64/msvcp140_1.dll": "2dd670f874562fbdca5b022df1943d70a57ba91fde559280e3a1daebe4db2380",
|
||||
"win64/msvcp140_2.dll": "1d60da3ac2b06482912ca852fa7047436e6e474b4cfffa3bf77f4598cfbf454c",
|
||||
"win64/msvcp140_atomic_wait.dll": "e7963645e0d1db08e300614d4c5fa7194bd8173e9ab7a5558859e6b232ed3241",
|
||||
"win64/msvcp140_codecvt_ids.dll": "ae8d922b00cdd93e3ebecc37beb46c800f383ebdeb9f9e5b84e04a72428b6fb3",
|
||||
"win64/vcamp140.dll": "617f6031929ec70a084a9dd1f01e037494a74d8c66703b7fb53e80547f90af28",
|
||||
"win64/vccorlib140.dll": "6b8d8a76c3e6664293407553650e60b94df9aaafc7c92057ea83032bd228e44f",
|
||||
"win64/vcomp140.dll": "31af29c03643f8396a6f26bcd601c6369d26493d7d78b714827ab2801bd284c7",
|
||||
"win64/vcruntime140.dll": "184146852727a9db4eea06178716bec3cdbb1015c911f6b0f915b184ad7775b2",
|
||||
"win64/vcruntime140_1.dll": "f4d66bdd6789aaed53341dc3153655e0ef2a9201c3652400443896a592e795b6",
|
||||
"win32/concrt140.dll": "1a084b2275c02b4df2b0ae395a5312a3ad3143f5ace9ba08cfd4fc42cdf9bf61",
|
||||
"win32/msvcp140.dll": "546ee2af2ffff02a34dbc1139bc6eb0eb5d67d83b3be782cfead374d29c8e01e",
|
||||
"win32/msvcp140_1.dll": "c19ef5e0fc2de4207800c3e7642cb18a2bf65ec8b7bc1833922c2bd4bc8bfa8e",
|
||||
"win32/msvcp140_2.dll": "8511395c8b36d0473aa1da9bd11a4cacc03511f77f837e997ca51047f29e145e",
|
||||
"win32/msvcp140_atomic_wait.dll": "fc5d35783a30822a1d418ee8f08130c450f4465c1dd68ce09241ab77e33c1931",
|
||||
"win32/msvcp140_codecvt_ids.dll": "79dcb9ef3986802cc9881ec80ee84d9f6a2ab075585da0f87561931821c0a9e7",
|
||||
"win32/vcamp140.dll": "4c2c17577b9dbc7becd8caa0aff01b1d04534e0a1bb5712bcff8780dc092b39c",
|
||||
"win32/vccorlib140.dll": "62ce34167c24c6f895da00acb5b65a8fb4ff392da5bdbeeece875ac3939df187",
|
||||
"win32/vcomp140.dll": "b1df4f47c8746e49b0b7db55fc5d391a1442de04863ccba497530003182b7576",
|
||||
"win32/vcruntime140.dll": "87fc734e0f2884985514edace58cf649a8ad67cb058dc7b7a4068f77af86810a"
|
||||
},
|
||||
"vcredist6__libs.tar.xz": {
|
||||
"win32/asycfilt.dll": "de83c9d9203050b40c098e4143ef8f577aa90016c7a64d4f2931b57a4c43e566",
|
||||
"win32/comcat.dll": "96d04cdfaf4f4d7b8722b139a15074975d4c244302f78034b7be65df1a92fd03",
|
||||
"win32/mfc42.dll": "ec63a85030c60716acdcf060abfaa95a6a3528631622fa60e7d17fbea2f751f9",
|
||||
"win32/mfc42u.dll": "5b3b68bb88be968a0c7c24887b437fac6d9081671b8d8376482168383ca30b8b",
|
||||
"win32/msvcirt.dll": "647f7534aaaedffa93534e4cb9b24bfcf91524828ff0364d88973be58139e255",
|
||||
"win32/msvcp60.dll": "2b6b93c2d66969eb00258e2b5ad6172decebada096e3b1b077a3380c80e4a072",
|
||||
"win32/msvcrt.dll": "887eb5ce93edb7192ca3e9220f07f9ca0f94db02af5862ebcbdfcb852db99fd1",
|
||||
"win32/oleaut32.dll": "baeb2f7c1b8be56738d34e1d1ddf8e0eebd3a633215dc1575e14656be38b939d",
|
||||
"win32/olepro32.dll": "789c3c45eda1749bd939f4a96616e1e9ef1b7dcc62a2889f65088954c64d0938",
|
||||
"win32/atla.dll": "e911ff1063d16348f353ee9001fbd43e04a3c8514028d964967d32dd87d83cca",
|
||||
"win32/stdole2.tlb": "6ea525bface5467c1045c3708f339a4b92a3a273f70656e061c7f7322c56d667"
|
||||
},
|
||||
"vcredist6sp6_inner.exe": {
|
||||
"vcredist6sp6_inner.exe": "ce05a2ad669fdc331cf43b4637f7c596c279685bc45eab005c15eb5ea495d5ea"
|
||||
},
|
||||
"winhttp__libs.tar.xz": {
|
||||
"win32/winhttp.dll": "5b40a36e68c7b428001e7ece8de43fb5effcc8a8a63671de6735c35cfd17aed2"
|
||||
},
|
||||
"wininet__libs.tar.xz": {
|
||||
"win32/wininet.dll": "4b01872788057ecf5a1ac6647512f5671213f76651ff8e808f1310a1240f35a2",
|
||||
"win64/wininet.dll": "e3ebf44621efc6381baae0f0efc13c356dcb6ee31bb258137edb3cc3e18549b5"
|
||||
},
|
||||
"xaudio2.7__libs.tar.xz": {
|
||||
"win32/XAudio2_2.dll": "9c7897b4ee1bcd190b1c0b7b77e64ee731d234764683a1e2286af70d86b62753",
|
||||
"win32/XAPOFX1_1.dll": "5ab63c0f040fdf65e681eba4daa55ed83e89ea10c426dc2fc763da0fc94f3ace",
|
||||
"win32/XAudio2_5.dll": "c596de2ab8394fb62538fef0b4657317f4ead50a6d798c5d066e25e334576c27",
|
||||
"win32/XAPOFX1_3.dll": "c6956ac2ee59f71e86784138b5443de6970a1274ac161945b8a44dc1d535db84",
|
||||
"win32/XAudio2_6.dll": "61b57170f7c6365714396072d22cb98746718c0f44c9f0d5c62fdb1b218639c7",
|
||||
"win32/XAPOFX1_4.dll": "0aee44b12913a95840ee6431d90518b0d72c54a27392e21ee6995e2151554a06",
|
||||
"win32/XAudio2_1.dll": "32a7c6a4edef46f025a4a5c64b892e29baaae948e86d9ed49e82014eec1441de",
|
||||
"win32/XAPOFX1_0.dll": "8b396d275de2550af8ada6a1ff71f0f4870b51c8407e44044c2dde7ad6b754cc",
|
||||
"win32/XAudio2_3.dll": "04dec9d7c68962e01efec0aac0ef7a3499bc4c16e8a41bd61fe6641da48d7dcc",
|
||||
"win32/XAPOFX1_2.dll": "d1a55061bcb42f69b7cf35e2985d48e30c7a90f0bc668e90390f465b36bd0161",
|
||||
"win32/XAudio2_7.dll": "d1690b602cb317f7f1e1e13e3fc5819ad8b5b38a92d812078afb1b408ccc4b69",
|
||||
"win32/XAPOFX1_5.dll": "becfdcd6b16523573cb52df87aa7d993f1b345ba903d0618c3b36535c3800197",
|
||||
"win32/XAudio2_0.dll": "e435b73193bdf651f7ae564eba05266595ac672db45e0e22dce92d0bcb3c6513",
|
||||
"win32/XAudio2_4.dll": "bcde4317debd0052b1436a6fda60e1dcb1e308979498117fa0cb50061f38101c",
|
||||
"win64/XAudio2_2.dll": "9ac81279b1625cbd9a8b59716aac63ef68d722dae586291457d24366a5f4e78b",
|
||||
"win64/XAPOFX1_1.dll": "cd03a9ef13123c1b9ce3d92b9e9fd9b9682c9f66c167dd8a326e4bd0acff73f2",
|
||||
"win64/XAudio2_5.dll": "36c77312026a3bd5fb928f4fe599a077b08264a82f2863875c2d4619ee133b71",
|
||||
"win64/XAPOFX1_3.dll": "5f03017d3b51c1dc413952d21bf35ac660693c6f7539e8f8930ddd41d197a495",
|
||||
"win64/XAudio2_6.dll": "3240447ef7bab48ec32a19f36e990bbddfdcb0f664fd9f58449f46788b5c32dd",
|
||||
"win64/XAPOFX1_4.dll": "d42e7e91eafce25550f08a63240e23fbe3ddd278b852e36e4eab05bc42f5ec7d",
|
||||
"win64/XAudio2_1.dll": "7c7c67e2255217d12c5c2afe53556c0f131049be7db4043aed9c1379b9ca9aa1",
|
||||
"win64/XAPOFX1_0.dll": "671c24f825629381870567d726b7dbc3890d665648043c041eb1100163c49e12",
|
||||
"win64/XAudio2_4.dll": "df7ddbea32acecc7fd6743002d211506ab75cb1685b4d2b708ea160c610a6f0d",
|
||||
"win64/XAudio2_7.dll": "f8636ccc37bbfc84107992b60e4226eb7237417112267ed64b08f72983ac4314",
|
||||
"win64/XAPOFX1_5.dll": "e47dcf74d50403b376c562e4121e359e5886e42fcc60b3fa8ba53e6826854c49",
|
||||
"win64/XAudio2_0.dll": "eed9479428df33f7a9be5eca9806c2e3412509274638d13032c6c0ed1a3a9ef1",
|
||||
"win64/XAudio2_3.dll": "d8e9c006edcf959f414ae7e3ff2e11af0885bc84cedc8249fdc1c5e71cdca8ba",
|
||||
"win64/XAPOFX1_2.dll": "f16e5795fac7a97fc1ffb5d049c011ee5f0dcdc4c6393664b6a9d94f949fdf0f"
|
||||
},
|
||||
"xinput__libs.tar.xz": {
|
||||
"win32/xinput1_1.dll": "69cd725c53e0302e75db20e9a3e4b33f58dceaa2e6ea4938b2733df8bc289a71",
|
||||
"win32/xinput1_3.dll": "8d540d484ea41e374fd0107d55d253f87ded4ce780d515d8fd59bbe8c98970a7",
|
||||
"win32/xinput1_2.dll": "9714d146a785d458f0de8fef387d82c9f8e101c02407a0cbeb06f02a69518eec",
|
||||
"win32/xinput9_1_0.dll": "9c14c5afd282be3e16117ff35d9896a96bdef497c45da950a730d279bd0e0cf6",
|
||||
"win64/xinput1_1.dll": "87c7f921bbf171fa04f7c2adaf8b25d4bf152d12c6b99771da42f3af07f78987",
|
||||
"win64/xinput1_3.dll": "756cad002e1553cfa1a91ebe8c1b9380ffabe0b4b1916c4a4db802396ddfbef8",
|
||||
"win64/xinput1_2.dll": "77055b6a144c4017acd0a8b34520cf9a84be6b9c75db40f7a708b9f4e38aa138",
|
||||
"win64/xinput9_1_0.dll": "81c6dfcd7bcb4f2b2580a98c8e66a3eb624b44f48b80f42a1f8134e448171cc9"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env python3
|
||||
"""What a release's assets must hash to, read from the catalog that points at them.
|
||||
|
||||
Prints one JSON object {asset name: {"sha256": ..., "md5": ..., "size": ...}} (each key optional)
|
||||
for the release tag given, so the mirror workflow can refuse an asset that does not match what
|
||||
the catalog promises. Catalogs: components.json (Bottles manifests: md5 as file_checksum and
|
||||
file_size per step), desktop.json / steam-seed.json (packages with sha256 and size), linuxfs.json
|
||||
(one package with sha256 and size). For the recordings release, each <name>.snapshot.json names
|
||||
its files archive with a sha256 and size; the snapshot files themselves have no outer checksum.
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def basename(url):
|
||||
return url.split("?")[0].rstrip("/").rsplit("/", 1)[-1]
|
||||
|
||||
|
||||
def under(url, tag):
|
||||
return "/releases/download/%s/" % tag in url
|
||||
|
||||
|
||||
def add(out, name, **fields):
|
||||
"""One candidate per catalog step: the Bottles manifests describe the vendor's original file
|
||||
(its md5 and size), which is what a mirrored .exe/.msi is but not what a repacked .tar.xz is,
|
||||
and two steps may describe the same asset differently. An asset passes when it matches any
|
||||
of its candidates; an asset with no usable candidate is mirrored on its own sha256 alone."""
|
||||
candidate = {}
|
||||
for key, value in fields.items():
|
||||
if value in (None, "", 0, "0"):
|
||||
continue
|
||||
if key == "size":
|
||||
value = int(value)
|
||||
elif key == "md5" and not (isinstance(value, str) and len(value) == 32):
|
||||
continue
|
||||
elif key == "sha256" and not (isinstance(value, str) and len(value) == 64):
|
||||
continue
|
||||
candidate[key] = value
|
||||
if not candidate or not ({"md5", "sha256"} & set(candidate)):
|
||||
out.setdefault(name, [])
|
||||
return
|
||||
candidates = out.setdefault(name, [])
|
||||
if candidate not in candidates:
|
||||
candidates.append(candidate)
|
||||
|
||||
|
||||
def from_components(doc, tag, out):
|
||||
for comp in doc.get("components", []):
|
||||
for step in comp.get("steps", []):
|
||||
url = step.get("url") or (step.get("environment") or {}).get("url") or ""
|
||||
if not isinstance(url, str) or not under(url, tag):
|
||||
continue
|
||||
add(out, basename(url), md5=step.get("file_checksum") or (step.get("environment") or {}).get("file_checksum"),
|
||||
size=step.get("file_size") or (step.get("environment") or {}).get("file_size"))
|
||||
|
||||
|
||||
def from_packages(doc, tag, out):
|
||||
packages = doc.get("packages") or ([doc] if doc.get("url") else [])
|
||||
for pkg in packages:
|
||||
url = pkg.get("url", "")
|
||||
if under(url, tag):
|
||||
add(out, basename(url), sha256=pkg.get("sha256"), size=pkg.get("size") or pkg.get("file_size"))
|
||||
|
||||
|
||||
def from_snapshots(folder, out):
|
||||
for path in Path(folder).glob("*.snapshot.json"):
|
||||
doc = json.loads(path.read_text("utf-8"))
|
||||
archive = doc.get("files_archive")
|
||||
if archive and archive.get("name"):
|
||||
add(out, archive["name"], sha256=archive.get("sha256"), size=archive.get("size"))
|
||||
|
||||
|
||||
def main():
|
||||
tag = sys.argv[1]
|
||||
out = {}
|
||||
for source in sys.argv[2:]:
|
||||
path = Path(source)
|
||||
if path.is_dir():
|
||||
from_snapshots(path, out)
|
||||
continue
|
||||
doc = json.loads(path.read_text("utf-8"))
|
||||
if "components" in doc:
|
||||
from_components(doc, tag, out)
|
||||
else:
|
||||
from_packages(doc, tag, out)
|
||||
json.dump(out, sys.stdout, indent=1, sort_keys=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"_": "Vendor links for assets whose Bottles manifest (pinned 2026-06-23) points at the wrong file; the catalog carries the right checksum. The mirror fetches these from the vendor and verifies them like any other.",
|
||||
"vcredist2010__vcredist_x86.exe": {
|
||||
"url": "https://download.microsoft.com/download/C/6/D/C6D0FD4E-9E53-4897-9B91-836EBA2AACD3/vcredist_x86.exe",
|
||||
"why": "the manifest's 2010 x86 is the original release (10.0.30319) while its x64 is SP1 (10.0.40219); this is SP1 x86"
|
||||
},
|
||||
"vcredist2012__vcredist_x86.exe": {
|
||||
"url": "https://download.microsoft.com/download/1/6/B/16B06F60-3B20-4FF2-B699-5E9B7962F9AE/VSU_4/vcredist_x86.exe",
|
||||
"why": "the manifest's 2012 x86 link is the 2010 installer; this is 2012 Update 4 x86 (11.0.61030)"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env python3
|
||||
"""upstream.py <components.json> <bottles manifests dir> <tag>: the vendor's own download link
|
||||
for every asset of the release that a Bottles dependency manifest describes with the same md5.
|
||||
Prints {asset name: url}. The manifests are checked out at a pinned commit by the workflow, so
|
||||
the link is the one the catalog's checksum was taken from, not whatever the manifest says today."""
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
def manifest_steps(root):
|
||||
"""md5 -> (component, url) over every step of every manifest (plain line scan: the files are
|
||||
simple YAML lists of steps)."""
|
||||
found = {}
|
||||
for path in glob.glob(os.path.join(root, "**", "*.yml"), recursive=True):
|
||||
name = os.path.basename(path)[:-4]
|
||||
current = {}
|
||||
|
||||
def flush():
|
||||
if current.get("url") and current.get("md5"):
|
||||
found.setdefault(current["md5"].lower(), (name, current["url"]))
|
||||
|
||||
for line in open(path, encoding="utf-8", errors="replace"):
|
||||
if re.match(r"\s*-\s*action:", line):
|
||||
flush()
|
||||
current = {}
|
||||
match = re.match(r"\s*url:\s*(\S+)", line)
|
||||
if match:
|
||||
current["url"] = match.group(1)
|
||||
match = re.match(r"\s*file_checksum:\s*(\S+)", line)
|
||||
if match:
|
||||
current["md5"] = match.group(1)
|
||||
flush()
|
||||
return found
|
||||
|
||||
|
||||
def main():
|
||||
catalog, root, tag = sys.argv[1:4]
|
||||
doc = json.load(open(catalog, encoding="utf-8"))
|
||||
steps = manifest_steps(root)
|
||||
out = {}
|
||||
for comp in doc.get("components", []):
|
||||
for step in comp.get("steps", []):
|
||||
env = step.get("environment") or {}
|
||||
url = step.get("url") or env.get("url") or ""
|
||||
if "/releases/download/%s/" % tag not in str(url):
|
||||
continue
|
||||
md5 = str(step.get("file_checksum") or env.get("file_checksum") or "").lower()
|
||||
asset = url.rsplit("/", 1)[-1]
|
||||
if len(md5) == 32 and md5 in steps and asset not in out:
|
||||
out[asset] = steps[md5][1]
|
||||
overrides = os.path.join(os.path.dirname(os.path.abspath(__file__)), "upstream-overrides.json")
|
||||
if os.path.isfile(overrides):
|
||||
for asset, entry in json.load(open(overrides, encoding="utf-8")).items():
|
||||
if asset != "_" and isinstance(entry, dict) and entry.get("url"):
|
||||
out[asset] = entry["url"]
|
||||
json.dump(out, sys.stdout, indent=1, sort_keys=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python3
|
||||
"""verify.py <file> <asset name> <expectations.json>: exit 0 when the file matches one of the
|
||||
catalog's candidates for that asset (md5 or sha256, and size when given), or when the catalog
|
||||
has none; exit 1 on a mismatch. Prints the file's sha256, md5 and size either way."""
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main():
|
||||
path, name, expectations = Path(sys.argv[1]), sys.argv[2], json.loads(Path(sys.argv[3]).read_text("utf-8"))
|
||||
sha, md5 = hashlib.sha256(), hashlib.md5()
|
||||
with path.open("rb") as source:
|
||||
for block in iter(lambda: source.read(1 << 20), b""):
|
||||
sha.update(block)
|
||||
md5.update(block)
|
||||
size = path.stat().st_size
|
||||
actual = {"sha256": sha.hexdigest(), "md5": md5.hexdigest(), "size": size}
|
||||
candidates = expectations.get(name, [])
|
||||
print("%s sha256=%s md5=%s size=%d" % (name, actual["sha256"], actual["md5"], size))
|
||||
if not candidates:
|
||||
print(" no catalog checksum for %s: mirrored on its own sha256" % name)
|
||||
return 0
|
||||
for candidate in candidates:
|
||||
if all(actual.get(key) == str(value).lower() for key, value in candidate.items() if key in ("md5", "sha256")):
|
||||
# The hash is the check; a catalog size that disagrees with a matching hash is the
|
||||
# catalog's typo (the Bottles manifests carry a few), noted but not fatal.
|
||||
if "size" in candidate and candidate["size"] != size:
|
||||
print(" matches the catalog's %s; the catalog's size %d is wrong (file is %d)" % (
|
||||
"sha256" if "sha256" in candidate else "md5", candidate["size"], size))
|
||||
else:
|
||||
print(" matches the catalog (%s)" % ", ".join("%s=%s" % (k, str(v)[:12]) for k, v in candidate.items()))
|
||||
return 0
|
||||
print("::error::%s does not match any catalog candidate: %s" % (name, candidates), file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,2 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
@@ -0,0 +1,260 @@
|
||||
{
|
||||
"_": "How each installer is run unattended in a fresh Wine prefix to record what it leaves behind (tools/snapshot/record.py). 'installer'+'args', or 'installers' [{asset, args}] run in order, name assets on the system-libraries-v1 release. 'arch' = prefix: win64, or win32 for installers that refuse a 64-bit prefix (.NET 2.0/3.5, Jet); a win32 recording is laid out as on 64-bit Windows unless 'remap' is false. 'timeout' seconds per installer. 'winver' = Windows version set before recording (not recorded). 'after' = recipes installed first, so this one records only its own delta (the app installs them first too). 'host_files' = the placed files travel with the recording (needed when no extractor opens the installer; only where the licence allows, 'licence' says why). 'dll_overrides' = Wine overrides for the run itself (never recorded). 'extract_to' + 'register' = instead of running a wizard with no silent mode, unpack the installer into that drive_c folder and regsvr32 those files (paths under drive_c). 'pre_reg_delete' = registry keys removed before recording (a fresh Wine prefix may claim a .NET it does not have).",
|
||||
"K-Lite": {
|
||||
"installer": "K-Lite.exe",
|
||||
"args": [
|
||||
"/VERYSILENT",
|
||||
"/SUPPRESSMSGBOXES",
|
||||
"/NORESTART",
|
||||
"/SP-"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 900,
|
||||
"host_files": true,
|
||||
"licence": "The pack's installer is Inno Setup 6.4.2, which no extractor reads yet, so the placed files travel with the recording. What this edition places is open source (LAV Filters, MPC-HC, MPC Video Renderer, DirectVobSub: GPL) plus Icaros, which its author distributes freely.",
|
||||
"notes": "The pack registers its DirectShow filters with regsvr32 during setup; those CLSID / filter entries are the point of the recording."
|
||||
},
|
||||
"lavfilters741": {
|
||||
"installer": "lavfilters741.exe",
|
||||
"args": [
|
||||
"/VERYSILENT",
|
||||
"/SUPPRESSMSGBOXES",
|
||||
"/NORESTART",
|
||||
"/SP-"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 600,
|
||||
"host_files": true,
|
||||
"licence": "LAV Filters is GPL-2.0; the files travel with the recording so a device needs no Inno Setup extractor."
|
||||
},
|
||||
"lavfilters702": {
|
||||
"installer": "lavfilters702.exe",
|
||||
"args": [
|
||||
"/VERYSILENT",
|
||||
"/SUPPRESSMSGBOXES",
|
||||
"/NORESTART",
|
||||
"/SP-"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 600,
|
||||
"host_files": true,
|
||||
"licence": "LAV Filters is GPL-2.0; the files travel with the recording so a device needs no Inno Setup extractor."
|
||||
},
|
||||
"ffdshow": {
|
||||
"installer": "ffdshow.exe",
|
||||
"args": [
|
||||
"/VERYSILENT",
|
||||
"/SUPPRESSMSGBOXES",
|
||||
"/NORESTART",
|
||||
"/SP-"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 600,
|
||||
"host_files": true,
|
||||
"licence": "ffdshow is GPL-2.0; the files travel with the recording so a device needs no Inno Setup extractor."
|
||||
},
|
||||
"dirac": {
|
||||
"installer": "dirac.exe",
|
||||
"arch": "win32",
|
||||
"timeout": 600,
|
||||
"extract_to": "Program Files/Dirac",
|
||||
"register": [
|
||||
"Program Files/Dirac/DiracSplitter-Dirac.ax",
|
||||
"Program Files/Dirac/DiracSplitter-libschroedinger.ax"
|
||||
],
|
||||
"notes": "NSIS wizard with no working silent mode (winetricks clicks through it with AutoHotkey): unpacked into Program Files/Dirac and the two filters registered with regsvr32, in a win32 prefix laid out as on 64-bit Windows. 7-Zip opens the installer on the device. The filters link the VC++ 2008 runtime, installed first (and a catalog dependency).",
|
||||
"after": [
|
||||
"vcredist2008"
|
||||
]
|
||||
},
|
||||
"quicktime72": {
|
||||
"installer": "quicktime72.exe",
|
||||
"args": [
|
||||
"/qn",
|
||||
"ALLUSERS=1",
|
||||
"DESKTOP_SHORTCUTS=0",
|
||||
"QTTaskRunFlags=0",
|
||||
"QTINFO.BISQTPRO=1",
|
||||
"SCHEDULE_ASUW=0",
|
||||
"REBOOT_REQUIRED=No"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 900,
|
||||
"dll_overrides": "winemenubuilder.exe=d",
|
||||
"notes": "Apple's redistribution terms do not allow hosting the files: a device takes them from the installer (a cabinet wrapper 7-Zip and cabextract open)."
|
||||
},
|
||||
"oalinst": {
|
||||
"installer": "oalinst.exe",
|
||||
"args": [
|
||||
"/s"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 300,
|
||||
"notes": "NSIS; the DLL-copy twin oalinst_dll already covers most games."
|
||||
},
|
||||
"VulkanRT": {
|
||||
"installer": "VulkanRT.exe",
|
||||
"args": [
|
||||
"/S"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 300,
|
||||
"notes": "NSIS (the catalog's /passive is an MSI switch and shows the wizard)."
|
||||
},
|
||||
"vcredist6sp6": {
|
||||
"installer": "vcredist6sp6_inner.exe",
|
||||
"args": [
|
||||
"/Q"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 300,
|
||||
"host_files": true,
|
||||
"licence": "Microsoft Visual C++ 6 runtime redistributable: licensed for redistribution with applications."
|
||||
},
|
||||
"jet40": {
|
||||
"installer": "jet40.exe",
|
||||
"args": [
|
||||
"/q"
|
||||
],
|
||||
"arch": "win32",
|
||||
"timeout": 600,
|
||||
"host_files": true,
|
||||
"licence": "Microsoft Jet 4.0 SP8 redistributable: licensed for redistribution with applications.",
|
||||
"notes": "32-bit only IExpress package (jetsetup.exe inside, INF-driven); /q as winetricks runs it; recorded in a win32 prefix and laid out as on 64-bit Windows."
|
||||
},
|
||||
"mdac28": {
|
||||
"installer": "mdac28.EXE",
|
||||
"args": [
|
||||
"/q",
|
||||
"/C:setup /qnt"
|
||||
],
|
||||
"arch": "win64",
|
||||
"winver": "win2k",
|
||||
"timeout": 900,
|
||||
"host_files": true,
|
||||
"licence": "Microsoft Data Access Components 2.8 redistributable: licensed for redistribution with applications.",
|
||||
"notes": "The setup refuses to run on anything after Windows 2000, so the recording's prefix says win2k (the catalog's win98 is the 32-bit Wine convention)."
|
||||
},
|
||||
"dotnet20": {
|
||||
"installer": "dotnet20__dotnetfx.exe",
|
||||
"args": [
|
||||
"/q",
|
||||
"/c:install.exe /q"
|
||||
],
|
||||
"arch": "win32",
|
||||
"winver": "winxp",
|
||||
"timeout": 1800,
|
||||
"host_files": true,
|
||||
"licence": "Microsoft .NET Framework 2.0 redistributable: licensed for redistribution with applications.",
|
||||
"notes": "32-bit only installer (the x64 one refuses Wine); recorded in a win32 prefix and laid out as on 64-bit Windows."
|
||||
},
|
||||
"dotnet20sp1": {
|
||||
"installer": "dotnet20sp1__NetFx20SP1_x86.exe",
|
||||
"args": [
|
||||
"/q"
|
||||
],
|
||||
"arch": "win32",
|
||||
"winver": "win2k",
|
||||
"after": [
|
||||
"dotnet20"
|
||||
],
|
||||
"timeout": 600,
|
||||
"host_files": true,
|
||||
"licence": "Microsoft .NET Framework 2.0 SP1 redistributable: licensed for redistribution with applications.",
|
||||
"notes": "32-bit only; recorded on top of dotnet20 in a win32 prefix and laid out as on 64-bit Windows. As winetricks does: Windows 2000 for the installer, ngen/regsvcs/mscorsvw stubbed.",
|
||||
"dll_overrides": "ngen.exe,regsvcs.exe,mscorsvw.exe=b",
|
||||
"pre_reg_delete": [
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v3.0",
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v3.5",
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v4"
|
||||
]
|
||||
},
|
||||
"dotnet35": {
|
||||
"installer": "dotnet35.exe",
|
||||
"args": [
|
||||
"/lang:ENU",
|
||||
"/q",
|
||||
"/norestart"
|
||||
],
|
||||
"arch": "win32",
|
||||
"winver": "winxp",
|
||||
"timeout": 3000,
|
||||
"host_files": true,
|
||||
"licence": "Microsoft .NET Framework 3.5 redistributable: licensed for redistribution with applications.",
|
||||
"notes": "The full redistributable (carries 2.0 SP2 and 3.0 SP2); 32-bit only, recorded in a win32 prefix and laid out as on 64-bit Windows. As winetricks does: mscoree/mscorwks native (no Wine stand-ins), Windows XP.",
|
||||
"dll_overrides": "mscoree,mscorwks=n",
|
||||
"pre_reg_delete": [
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v3.0",
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v3.5",
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v4"
|
||||
]
|
||||
},
|
||||
"dotnet35sp1": {
|
||||
"installer": "dotnet35sp1.exe",
|
||||
"args": [
|
||||
"/lang:ENU",
|
||||
"/q",
|
||||
"/norestart"
|
||||
],
|
||||
"arch": "win32",
|
||||
"winver": "winxp",
|
||||
"timeout": 3000,
|
||||
"host_files": true,
|
||||
"licence": "Microsoft .NET Framework 3.5 SP1 redistributable: licensed for redistribution with applications.",
|
||||
"notes": "The full redistributable (carries 2.0 SP2 and 3.0 SP2); 32-bit only, recorded in a win32 prefix and laid out as on 64-bit Windows. As winetricks does: mscoree/mscorwks native (no Wine stand-ins), Windows XP.",
|
||||
"dll_overrides": "mscoree,mscorwks=n;ngen.exe=n",
|
||||
"pre_reg_delete": [
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v3.0",
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v3.5",
|
||||
"HKLM\\Software\\Microsoft\\NET Framework Setup\\NDP\\v4"
|
||||
]
|
||||
},
|
||||
"aairruntime": {
|
||||
"installer": "aairruntime.exe",
|
||||
"args": [
|
||||
"-silent",
|
||||
"-eulaAccepted"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 600,
|
||||
"notes": "Adobe's terms do not allow hosting the files; the installer is an archive 7-Zip can open."
|
||||
},
|
||||
"art2kmin": {
|
||||
"installer": "art2kmin.exe",
|
||||
"args": [
|
||||
"/q"
|
||||
],
|
||||
"arch": "win32",
|
||||
"timeout": 900,
|
||||
"notes": "Access 2000 runtime: a 7-Zip-openable wrapper around Setup.exe + Data1.msi. Not recorded: the app's MSI engine lays Data1.msi out directly once art2kmin leaves its setup-program list."
|
||||
},
|
||||
"webview2": {
|
||||
"installer": "webview2.exe",
|
||||
"args": [
|
||||
"/silent",
|
||||
"/install"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 1200,
|
||||
"notes": "Microsoft Edge WebView2 Evergreen standalone installer. Recorded once: 656 files, 576 of them inside a payload no extractor opens, and Microsoft's terms do not allow hosting them. Not linkable; left for a runtime that can run the installer."
|
||||
},
|
||||
"gfw": {
|
||||
"installer": "gfw.exe",
|
||||
"args": [
|
||||
"/nodotnet",
|
||||
"/q"
|
||||
],
|
||||
"arch": "win64",
|
||||
"timeout": 1200,
|
||||
"notes": "Games for Windows Live client + redistributable (gfwlivesetupmin, two MSIs inside). The files come from the installer on the device; signing in needs Microsoft's live servers, which nothing here records."
|
||||
},
|
||||
"vcredist2008": {
|
||||
"installer": "vcredist2008__vcredist_x86.exe",
|
||||
"args": [
|
||||
"/q"
|
||||
],
|
||||
"arch": "win32",
|
||||
"timeout": 600,
|
||||
"notes": "Prerequisite only (dirac's filters link msvcr90): the app installs vcredist2008 through its own MSI engine; this recipe exists so a recording can be made on top of it."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,707 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Records what a Windows installer leaves behind in a Wine prefix, without keeping the installer's files.
|
||||
|
||||
record.py --component NAME --installer FILE --out DIR [--recipes recipes.json] [--extract DIR]
|
||||
|
||||
A fresh 64-bit prefix is made, its files and registry noted, the installer run unattended (the
|
||||
switches in recipes.json), and the prefix noted again. The difference is the recording:
|
||||
|
||||
<out>/<component>.snapshot.json
|
||||
"files": every file the installer placed under drive_c, with size and sha256, and where
|
||||
the same bytes sit inside the installer when an extractor (innoextract, 7-Zip)
|
||||
can reach them ("source"), so a device pulls them from the vendor's own
|
||||
installer instead of a copy we host. A file the installer generated (no
|
||||
matching bytes inside it) is carried inline when small ("data", base64), else
|
||||
listed as "missing" for the recipe to deal with.
|
||||
"registry": every value the installer added or changed, in the shape droiddeck-wincomponents
|
||||
writes into a prefix (hive, key, name, type, data), the user's folder rewritten
|
||||
to Proton's steamuser and Wine's own bookkeeping left out.
|
||||
<out>/<component>.summary.md a readable account of the run.
|
||||
|
||||
The point of the recording is the registry: a self-registering DLL (a DirectShow filter) decides
|
||||
its own CLSID and filter entries at run time, so nothing but running it tells us what they are.
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
INLINE_FILE_LIMIT = 256 * 1024
|
||||
INLINE_TOTAL_LIMIT = 2 * 1024 * 1024
|
||||
|
||||
# Folders whose contents are the installer's own litter, not the product.
|
||||
SKIP_DIRS = ("windows/temp", "windows/installer", "users/*/temp", "users/*/appdata/local/temp",
|
||||
"users/*/appdata/local/microsoft/windows/inetcache", "programdata/microsoft/windows/start menu",
|
||||
"users/*/appdata/roaming/microsoft/windows/start menu", "users/*/desktop", "users/*/start menu",
|
||||
"windows/logs", "windows/prefetch")
|
||||
SKIP_FILE_SUFFIXES = (".log", ".lnk", ".url", ".tmp")
|
||||
|
||||
# Registry keys Wine or the prefix maintain by themselves between two snapshots.
|
||||
SKIP_KEY_PREFIXES = (
|
||||
"software\\wine\\", "software\\microsoft\\windows\\currentversion\\explorer\\",
|
||||
"software\\microsoft\\windows nt\\currentversion\\profilelist", "system\\currentcontrolset\\control\\session manager\\environment\\",
|
||||
"software\\microsoft\\windows\\currentversion\\installer\\userdata", "software\\microsoft\\cryptography\\",
|
||||
"software\\microsoft\\windows\\currentversion\\shell extensions\\cached", "software\\classes\\local settings\\",
|
||||
"software\\microsoft\\windows\\currentversion\\explorer", "software\\microsoft\\rpc\\",
|
||||
"software\\microsoft\\windows nt\\currentversion\\fonts", "software\\microsoft\\windows nt\\currentversion\\winlogon",
|
||||
"control panel\\", "environment", "volatile environment", "software\\microsoft\\windows\\currentversion\\uninstall\\{",
|
||||
"console", "console\\",
|
||||
)
|
||||
SKIP_VALUE_NAMES = {"installdate", "installtime", "lastwritetime", "installsource", "sourcelist", "lastusedsource",
|
||||
"estimatedsize", "modified", "installlocation"}
|
||||
|
||||
|
||||
def sha256_of(path):
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as source:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def run(cmd, env=None, timeout=None, check=False):
|
||||
return subprocess.run(cmd, env=env, timeout=timeout, check=check, capture_output=True, text=True, errors="replace")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- files
|
||||
|
||||
SKIP_DIR_PATTERNS = [re.compile("^" + re.escape(p).replace("\\*", "[^/]+") + "(/|$)") for p in SKIP_DIRS]
|
||||
|
||||
|
||||
def skipped_path(rel):
|
||||
lower = rel.lower()
|
||||
return lower.endswith(SKIP_FILE_SUFFIXES) or any(pattern.match(lower) for pattern in SKIP_DIR_PATTERNS)
|
||||
|
||||
|
||||
def snapshot_files(drive_c):
|
||||
found = {}
|
||||
for root, dirs, files in os.walk(drive_c):
|
||||
for name in files:
|
||||
path = Path(root) / name
|
||||
if path.is_symlink():
|
||||
continue
|
||||
rel = path.relative_to(drive_c).as_posix()
|
||||
if skipped_path(rel):
|
||||
continue
|
||||
try:
|
||||
found[rel] = (path.stat().st_size, sha256_of(path))
|
||||
except OSError:
|
||||
continue
|
||||
return found
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- registry
|
||||
|
||||
def unescape(text):
|
||||
out, i = [], 0
|
||||
while i < len(text):
|
||||
ch = text[i]
|
||||
if ch == "\\" and i + 1 < len(text):
|
||||
nxt = text[i + 1]
|
||||
if nxt == "x" and i + 5 < len(text):
|
||||
try:
|
||||
out.append(chr(int(text[i + 2:i + 6], 16)))
|
||||
i += 6
|
||||
continue
|
||||
except ValueError:
|
||||
pass
|
||||
out.append({"n": "\n", "r": "\r", "0": "\0", "t": "\t"}.get(nxt, nxt))
|
||||
i += 2
|
||||
else:
|
||||
out.append(ch)
|
||||
i += 1
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def parse_reg(path):
|
||||
"""A Wine registry file as {key: {name: raw}}; name "" is the default value; a key with no
|
||||
values is {}. Multi-line hex values are joined."""
|
||||
keys = {}
|
||||
current = None
|
||||
pending = None
|
||||
try:
|
||||
lines = Path(path).read_text("utf-8", errors="replace").split("\n")
|
||||
except FileNotFoundError:
|
||||
return keys
|
||||
for raw_line in lines:
|
||||
line = raw_line.rstrip("\r")
|
||||
if pending is not None:
|
||||
pending += line.strip()
|
||||
if pending.endswith("\\"):
|
||||
pending = pending[:-1]
|
||||
continue
|
||||
name, raw = pending_name, pending
|
||||
keys[current][name] = raw
|
||||
pending = None
|
||||
continue
|
||||
if not line or line.startswith(";") or line.startswith("#"):
|
||||
continue
|
||||
if line.startswith("["):
|
||||
end = line.rfind("]")
|
||||
key = line[1:end].replace("\\\\", "\\")
|
||||
current = key
|
||||
keys.setdefault(current, {})
|
||||
continue
|
||||
if current is None:
|
||||
continue
|
||||
if line.startswith("@="):
|
||||
name, raw = "", line[2:]
|
||||
elif line.startswith('"'):
|
||||
end = 1
|
||||
while end < len(line):
|
||||
if line[end] == "\\":
|
||||
end += 2
|
||||
continue
|
||||
if line[end] == '"':
|
||||
break
|
||||
end += 1
|
||||
name = unescape(line[1:end])
|
||||
raw = line[end + 2:] if line[end + 1:end + 2] == "=" else ""
|
||||
else:
|
||||
continue
|
||||
if raw.endswith("\\"):
|
||||
pending, pending_name = raw[:-1], name
|
||||
continue
|
||||
keys[current][name] = raw
|
||||
return keys
|
||||
|
||||
|
||||
def decode(raw):
|
||||
"""A Wine registry value as (type, data) in droiddeck-wincomponents' vocabulary, or None."""
|
||||
if raw.startswith('"') and raw.endswith('"'):
|
||||
return "sz", unescape(raw[1:-1])
|
||||
if raw.startswith("dword:"):
|
||||
try:
|
||||
return "dword", int(raw[6:], 16)
|
||||
except ValueError:
|
||||
return None
|
||||
if raw.startswith("str(2):") or raw.startswith("str(7):"):
|
||||
text = unescape(raw[7:].strip('"'))
|
||||
return ("expand_sz", text) if raw.startswith("str(2)") else ("multi_sz", [s for s in text.split("\0") if s])
|
||||
match = re.match(r"hex(\((\d+)\))?:(.*)$", raw)
|
||||
if match:
|
||||
kind = int(match.group(2) or 3)
|
||||
data = bytes.fromhex(match.group(3).replace(",", "").replace("\\", ""))
|
||||
if kind in (2, 7, 1):
|
||||
text = data.decode("utf-16-le", errors="replace")
|
||||
if kind == 7:
|
||||
return "multi_sz", [s for s in text.split("\0") if s]
|
||||
return ("expand_sz" if kind == 2 else "sz"), text.rstrip("\0")
|
||||
if kind == 4 and len(data) == 4:
|
||||
return "dword", int.from_bytes(data, "little")
|
||||
return "binary", data.hex()
|
||||
return None
|
||||
|
||||
|
||||
def skipped_key(key):
|
||||
lower = key.lower()
|
||||
return any(lower.startswith(prefix) or lower == prefix.rstrip("\\") for prefix in SKIP_KEY_PREFIXES) or "\\volatile" in lower
|
||||
|
||||
|
||||
def diff_registry(before, after, hive, home_user):
|
||||
"""The values in [after] that [before] lacks or has differently, as registry.json entries."""
|
||||
entries = []
|
||||
home_pattern = re.compile(re.escape("C:\\users\\" + home_user), re.IGNORECASE) if home_user else None
|
||||
for key, values in after.items():
|
||||
if skipped_key(key):
|
||||
continue
|
||||
old = before.get(key)
|
||||
if old is None and not values:
|
||||
entries.append({"hive": hive, "key": key, "type": "key"})
|
||||
continue
|
||||
for name, raw in values.items():
|
||||
if name.lower() in SKIP_VALUE_NAMES:
|
||||
continue
|
||||
if old is not None and old.get(name) == raw:
|
||||
continue
|
||||
decoded = decode(raw)
|
||||
if decoded is None:
|
||||
continue
|
||||
kind, data = decoded
|
||||
if isinstance(data, str):
|
||||
if "Z:\\" in data or "z:\\" in data:
|
||||
continue
|
||||
if home_pattern:
|
||||
data = home_pattern.sub("C:\\\\users\\\\steamuser", data)
|
||||
elif isinstance(data, list) and home_pattern:
|
||||
data = [home_pattern.sub("C:\\\\users\\\\steamuser", item) for item in data]
|
||||
entries.append({"hive": hive, "key": key, "name": name, "type": kind, "data": data})
|
||||
return entries
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 32-bit recordings in the 64-bit layout
|
||||
|
||||
# A 32-bit installer that refuses a 64-bit prefix (.NET 2.0 and 3.5, Jet) is recorded in a win32
|
||||
# prefix, then translated into what the same installer leaves on 64-bit Windows: files move the
|
||||
# way the file-system redirector moves them (system32 -> syswow64, Program Files -> Program
|
||||
# Files (x86)), and the registry is replayed into a fresh 64-bit prefix through Wine's own 32-bit
|
||||
# regedit, so Wine's WOW64 registry redirection (Wow6432Node, the Classes subkeys) decides where
|
||||
# each key lands rather than a hand-written table.
|
||||
SYSTEM32_SHARED = ("catroot", "catroot2", "driverstore", "drivers", "etc", "logfiles", "spool")
|
||||
|
||||
|
||||
def remap_file(rel):
|
||||
parts = rel.split("/")
|
||||
lower = [p.lower() for p in parts]
|
||||
if lower[:2] == ["windows", "system32"] and not (len(lower) > 2 and lower[2] in SYSTEM32_SHARED):
|
||||
return "/".join(["windows", "syswow64"] + parts[2:])
|
||||
if lower[:1] == ["program files"]:
|
||||
return "/".join(["Program Files (x86)"] + parts[1:])
|
||||
if lower[:2] == ["programdata", "microsoft"] or lower[:1] == ["users"]:
|
||||
return rel
|
||||
return rel
|
||||
|
||||
|
||||
def reg_escape(text):
|
||||
return text.replace("\\", "\\\\").replace('"', '\\"')
|
||||
|
||||
|
||||
def encode_value(kind, data):
|
||||
if kind == "sz":
|
||||
return '"%s"' % reg_escape(data)
|
||||
if kind == "dword":
|
||||
return "dword:%08x" % int(data)
|
||||
if kind == "expand_sz":
|
||||
return "hex(2):" + ",".join("%02x" % b for b in (data + "\0").encode("utf-16-le"))
|
||||
if kind == "multi_sz":
|
||||
return "hex(7):" + ",".join("%02x" % b for b in ("\0".join(list(data) + [""]) + "\0").encode("utf-16-le"))
|
||||
if kind == "binary":
|
||||
return "hex:" + ",".join(data[i:i + 2] for i in range(0, len(data), 2))
|
||||
return None
|
||||
|
||||
|
||||
def write_reg(entries, path):
|
||||
"""The recorded values as a .reg file regedit imports (UTF-16, as Windows writes them)."""
|
||||
roots = {"HKLM": "HKEY_LOCAL_MACHINE", "HKCU": "HKEY_CURRENT_USER"}
|
||||
lines = ["Windows Registry Editor Version 5.00", ""]
|
||||
by_key = {}
|
||||
for entry in entries:
|
||||
by_key.setdefault((entry["hive"], entry["key"]), []).append(entry)
|
||||
for (hive, key), values in by_key.items():
|
||||
lines.append("[%s\\%s]" % (roots[hive], key))
|
||||
for entry in values:
|
||||
if entry.get("type") == "key":
|
||||
continue
|
||||
raw = encode_value(entry["type"], entry["data"])
|
||||
if raw is None:
|
||||
continue
|
||||
lines.append(("@=%s" if entry["name"] == "" else '"%s"=%%s' % reg_escape(entry["name"])) % raw if entry["name"] else "@=" + raw)
|
||||
lines.append("")
|
||||
Path(path).write_text("\n".join(lines) + "\n", "utf-16")
|
||||
|
||||
|
||||
def replay_into_win64(entries, work, home_user, notes):
|
||||
"""Imports the 32-bit recording's registry into a fresh 64-bit prefix with Wine's 32-bit
|
||||
regedit and records what landed, Wow6432Node and all."""
|
||||
prefix = work / "pfx64"
|
||||
env = wine_env(prefix, "win64", "")
|
||||
print("== making a fresh win64 prefix to replay the registry into", flush=True)
|
||||
result = run(["wineboot", "-u"], env=env, timeout=600)
|
||||
wait_wine(env, 300)
|
||||
if result.returncode != 0:
|
||||
notes.append("the 64-bit replay prefix could not be made; registry left in the 32-bit layout")
|
||||
return entries
|
||||
before = {"HKLM": parse_reg(prefix / "system.reg"), "HKCU": parse_reg(prefix / "user.reg")}
|
||||
reg_file = work / "recording.reg"
|
||||
write_reg(entries, reg_file)
|
||||
dos_path = "Z:" + str(reg_file.resolve()).replace("/", "\\")
|
||||
result = run(["wine", "C:\\windows\\syswow64\\regedit.exe", "/S", dos_path], env=env, timeout=600)
|
||||
wait_wine(env, 300)
|
||||
subprocess.run(["wineserver", "-k"], env=env)
|
||||
time.sleep(2)
|
||||
if result.returncode != 0:
|
||||
notes.append("32-bit regedit import exit %s: %s" % (result.returncode, result.stderr.strip()[-300:]))
|
||||
after = {"HKLM": parse_reg(prefix / "system.reg"), "HKCU": parse_reg(prefix / "user.reg")}
|
||||
replayed = diff_registry(before["HKLM"], after["HKLM"], "HKLM", home_user) + \
|
||||
diff_registry(before["HKCU"], after["HKCU"], "HKCU", home_user)
|
||||
# Only what the recording wrote: Wine stirs a few keys of its own in a fresh prefix meanwhile.
|
||||
wanted = {(e["hive"], e["key"].lower()) for e in entries}
|
||||
def recorded_key(e):
|
||||
plain = e["key"].lower().replace("\\wow6432node", "")
|
||||
return (e["hive"], plain) in wanted or (e["hive"], e["key"].lower()) in wanted
|
||||
replayed = [e for e in replayed if recorded_key(e)]
|
||||
wow = sum(1 for e in replayed if "wow6432node" in e["key"].lower())
|
||||
notes.append("registry replayed through Wine's 32-bit regedit into a 64-bit prefix: %d values in, %d out, %d under Wow6432Node" % (
|
||||
sum(1 for e in entries if e.get("type") != "key"), sum(1 for e in replayed if e.get("type") != "key"), wow))
|
||||
print(" replay: %d values in, %d out, %d under Wow6432Node" % (len(entries), len(replayed), wow), flush=True)
|
||||
return replayed
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the run
|
||||
|
||||
def wine_env(prefix, arch, overrides):
|
||||
env = dict(os.environ)
|
||||
env.update({"WINEPREFIX": str(prefix), "WINEARCH": arch, "WINEDEBUG": "-all",
|
||||
"WINEDLLOVERRIDES": "winemenubuilder.exe=d" + (";" + overrides if overrides else "")})
|
||||
env.pop("DISPLAY", None)
|
||||
return env
|
||||
|
||||
|
||||
def wait_wine(env, timeout):
|
||||
try:
|
||||
subprocess.run(["wineserver", "-w"], env=env, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
subprocess.run(["wineserver", "-k"], env=env)
|
||||
time.sleep(3)
|
||||
|
||||
|
||||
NESTED_SUFFIXES = (".7z", ".zip", ".cab", ".exe", ".msi", ".rar", ".xz", ".gz", ".tar")
|
||||
|
||||
|
||||
def open_nested(dest, depth=0):
|
||||
"""Archives inside an extraction (K-Lite carries its filters as 7-Zip payloads) are opened in
|
||||
place, one folder per archive, two levels deep."""
|
||||
if depth > 1 or not shutil.which("7z"):
|
||||
return 0
|
||||
opened = 0
|
||||
for path in sorted(p for p in dest.rglob("*") if p.is_file() and p.suffix.lower() in NESTED_SUFFIXES):
|
||||
if path.name.endswith(".nested") or path.stat().st_size < 1024:
|
||||
continue
|
||||
probe = run(["7z", "l", "-ba", str(path)], timeout=300)
|
||||
if probe.returncode != 0 or not probe.stdout.strip():
|
||||
continue
|
||||
folder = path.with_name(path.name + ".nested")
|
||||
folder.mkdir(exist_ok=True)
|
||||
run(["7z", "x", "-y", "-bd", "-bso0", "-bsp0", "-o" + str(folder), str(path)], timeout=1800)
|
||||
if any(folder.rglob("*")):
|
||||
opened += 1 + open_nested(folder, depth + 1)
|
||||
else:
|
||||
shutil.rmtree(folder, ignore_errors=True)
|
||||
return opened
|
||||
|
||||
|
||||
def extract_installer(installer, dest, notes):
|
||||
"""Opens the installer with innoextract or 7-Zip; returns {sha256: relative path} of what came out."""
|
||||
found = {}
|
||||
tried = []
|
||||
if shutil.which("innoextract"):
|
||||
tried.append("innoextract")
|
||||
listing = run(["innoextract", "-l", str(installer)], timeout=600)
|
||||
said = (listing.stderr + listing.stdout).strip().splitlines()
|
||||
warnings = [line for line in said if "warning" in line.lower() or "error" in line.lower()]
|
||||
print(" innoextract: %d listed, %s" % (sum(1 for line in said if line.startswith(" - ")), "; ".join(warnings[:3]) or "no warnings"), flush=True)
|
||||
if warnings:
|
||||
notes.append("innoextract: " + "; ".join(warnings[:3]))
|
||||
result = run(["innoextract", "-q", "-m", "-d", str(dest), str(installer)], timeout=1800)
|
||||
if result.returncode != 0 or not any(dest.rglob("*")):
|
||||
notes.append("innoextract extracted nothing (exit %s)" % result.returncode)
|
||||
shutil.rmtree(dest, ignore_errors=True)
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
if not any(dest.rglob("*")) and shutil.which("7z"):
|
||||
tried.append("7z")
|
||||
result = run(["7z", "x", "-y", "-bd", "-bso0", "-bsp0", "-o" + str(dest), str(installer)], timeout=1800)
|
||||
if not any(dest.rglob("*")):
|
||||
notes.append("7-Zip extracted nothing (exit %s)" % result.returncode)
|
||||
top = sum(1 for p in dest.rglob("*") if p.is_file())
|
||||
nested = open_nested(dest)
|
||||
for path in dest.rglob("*"):
|
||||
if path.is_file():
|
||||
found.setdefault(sha256_of(path), path.relative_to(dest).as_posix())
|
||||
print(" extraction: %d files at the top, %d nested archives opened, %d files in all" % (top, nested, len(found)), flush=True)
|
||||
notes.append("extraction: %d files, %d nested archives opened (%s)" % (len(found), nested, ", ".join(tried) or "no extractor"))
|
||||
return found, tried
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--component", required=True)
|
||||
parser.add_argument("--installer", help="the installer file (single-installer recipes)")
|
||||
parser.add_argument("--installers-dir", help="folder holding every recipe's installer assets by name")
|
||||
parser.add_argument("--out", required=True)
|
||||
parser.add_argument("--recipes", default=str(Path(__file__).with_name("recipes.json")))
|
||||
parser.add_argument("--extract", help="where the installer's own extraction goes (default: a temp dir)")
|
||||
args = parser.parse_args()
|
||||
|
||||
recipes = json.loads(Path(args.recipes).read_text("utf-8"))
|
||||
recipe = recipes.get(args.component)
|
||||
if not recipe:
|
||||
print("no recipe for %s" % args.component, file=sys.stderr)
|
||||
return 2
|
||||
out = Path(args.out)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
arch = recipe.get("arch", "win64")
|
||||
timeout = int(recipe.get("timeout", 600))
|
||||
notes = []
|
||||
assets = Path(args.installers_dir).resolve() if args.installers_dir else None
|
||||
|
||||
def runs_of(name):
|
||||
"""A recipe's installers as (path, args) pairs, from --installers-dir or --installer."""
|
||||
r = recipes[name]
|
||||
items = r.get("installers") or [{"asset": r.get("installer"), "args": r.get("args", [])}]
|
||||
found = []
|
||||
for item in items:
|
||||
path = assets / item["asset"] if assets else Path(args.installer).resolve()
|
||||
if not path.is_file():
|
||||
raise SystemExit("installer missing: %s" % path)
|
||||
found.append((path, list(item.get("args", []))))
|
||||
return found
|
||||
|
||||
def run_installer(path, run_args, limit, label):
|
||||
print("== running %s %s" % (path.name, " ".join(run_args)), flush=True)
|
||||
cmd = ["xvfb-run", "-a", "-s", "-screen 0 1280x800x24", "wine", str(path)] + run_args
|
||||
started_at = time.time()
|
||||
try:
|
||||
result = run(cmd, env=env, timeout=limit)
|
||||
exit_code = result.returncode
|
||||
if result.stderr.strip():
|
||||
notes.append("%s stderr: %s" % (label, result.stderr.strip()[-400:]))
|
||||
except subprocess.TimeoutExpired:
|
||||
exit_code = "timeout"
|
||||
notes.append("%s was still running after %d s and was killed" % (label, limit))
|
||||
wait_wine(env, 180)
|
||||
subprocess.run(["wineserver", "-k"], env=env)
|
||||
time.sleep(2)
|
||||
took = int(time.time() - started_at)
|
||||
print("== %s exit %s after %d s" % (label, exit_code, took), flush=True)
|
||||
if exit_code != 0:
|
||||
# What the installer itself said: Microsoft's setups leave dd_*.txt / *.log in Temp.
|
||||
logs = []
|
||||
for pattern in ("users/*/Temp/*.txt", "users/*/Temp/*.log", "users/*/Temp/*/*.txt", "users/*/Temp/*/*.log",
|
||||
"windows/temp/*.txt", "windows/temp/*.log", "windows/temp/*/*.log", "*/*.log", "*/*.txt"):
|
||||
for log in drive_c.glob(pattern):
|
||||
try:
|
||||
if log.stat().st_mtime >= started_at - 5 and log.stat().st_size > 0:
|
||||
logs.append(log)
|
||||
except OSError:
|
||||
pass
|
||||
logs = sorted(set(logs), key=lambda l: -l.stat().st_size)[:3]
|
||||
for log in logs:
|
||||
try:
|
||||
text = log.read_bytes().decode("utf-16", errors="replace") if log.read_bytes()[:2] in (b"\xff\xfe", b"\xfe\xff") else log.read_text("utf-8", errors="replace")
|
||||
except OSError:
|
||||
continue
|
||||
tail = "\n".join(line for line in text.splitlines() if line.strip())[-1500:]
|
||||
print(" -- %s (last lines) --\n%s" % (log.relative_to(drive_c), tail), flush=True)
|
||||
notes.append("%s log %s: %s" % (label, log.relative_to(drive_c).as_posix(), tail[-600:]))
|
||||
# The lines that name the problem, wherever they are in the log.
|
||||
telling = [line.strip() for line in text.splitlines()
|
||||
if re.search(r"block|gencomp|return value 3|error|fail|not (met|found|supported)|requires|missing", line, re.IGNORECASE)
|
||||
and "does not match requested set" not in line][:25]
|
||||
if telling:
|
||||
print(" -- %s (telling lines) --\n%s" % (log.relative_to(drive_c), "\n".join(telling)), flush=True)
|
||||
return exit_code, took
|
||||
|
||||
runs = runs_of(args.component)
|
||||
installer = runs[0][0]
|
||||
|
||||
work = Path(tempfile.mkdtemp(prefix="snapshot-"))
|
||||
prefix = work / "pfx"
|
||||
env = wine_env(prefix, arch, recipe.get("dll_overrides", ""))
|
||||
print("== making a fresh %s prefix" % arch, flush=True)
|
||||
result = run(["wineboot", "-u"], env=env, timeout=600)
|
||||
wait_wine(env, 300)
|
||||
if result.returncode != 0:
|
||||
print(result.stderr[-800:], file=sys.stderr)
|
||||
return 1
|
||||
wine_version = run(["wine", "--version"], env=env).stdout.strip()
|
||||
drive_c = prefix / "drive_c"
|
||||
home_user = next((p.name for p in (drive_c / "users").iterdir() if p.is_dir() and p.name.lower() not in ("public", "default")), "")
|
||||
# The Windows version the installer wants to see (MDAC refuses anything after 2000), set before
|
||||
# the first snapshot so it is not recorded; prerequisites likewise, so a service pack records its delta.
|
||||
winver = recipe.get("winver")
|
||||
if winver:
|
||||
# winecfg writes the whole set: the version APIs and the registry keys an installer reads
|
||||
# directly (CurrentVersion, CSDVersion, Control\Windows\CSDVersion). .NET 3.5's "Windows XP
|
||||
# Service Pack 2" check reads the latter, which the HKCU\Software\Wine\Version shortcut
|
||||
# never writes.
|
||||
result = run(["winecfg", "-v", winver], env=env, timeout=300)
|
||||
wait_wine(env, 120)
|
||||
if result.returncode != 0:
|
||||
run(["wine", "reg", "add", "HKCU\\Software\\Wine", "/v", "Version", "/d", winver, "/f"], env=env, timeout=300)
|
||||
wait_wine(env, 120)
|
||||
notes.append("recorded with the Windows version set to %s (winecfg)" % winver)
|
||||
for key in recipe.get("pre_reg_delete", []):
|
||||
run(["wine", "reg", "delete", key, "/f"], env=env, timeout=300)
|
||||
wait_wine(env, 120)
|
||||
notes.append("deleted %s before recording" % key)
|
||||
for prerequisite in recipe.get("after", []):
|
||||
for path, run_args in runs_of(prerequisite):
|
||||
run_installer(path, run_args, int(recipes[prerequisite].get("timeout", 600)), "prerequisite " + prerequisite)
|
||||
notes.append("recorded on top of %s" % prerequisite)
|
||||
|
||||
print("== noting the prefix before", flush=True)
|
||||
files_before = snapshot_files(drive_c)
|
||||
reg_before = {"HKLM": parse_reg(prefix / "system.reg"), "HKCU": parse_reg(prefix / "user.reg")}
|
||||
|
||||
status, elapsed = None, 0
|
||||
if recipe.get("extract_to"):
|
||||
# No silent mode (dirac's NSIS wizard): unpack the installer where its wizard would have put
|
||||
# the files and register the filters it would have registered, with Wine's own regsvr32.
|
||||
target = drive_c / recipe["extract_to"]
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
started_at = time.time()
|
||||
result = run(["7z", "x", "-y", "-bso0", "-bsp0", "-o" + str(target), str(installer)], timeout=600)
|
||||
for litter in ("$PLUGINSDIR", "$TEMP", "$R0"):
|
||||
shutil.rmtree(target / litter, ignore_errors=True)
|
||||
status = result.returncode
|
||||
notes.append("unpacked into %s instead of running the wizard (no silent mode)" % recipe["extract_to"])
|
||||
for rel in recipe.get("register", []):
|
||||
dos = "C:\\" + rel.replace("/", "\\")
|
||||
print("== regsvr32 %s" % dos, flush=True)
|
||||
result = run(["xvfb-run", "-a", "-s", "-screen 0 1280x800x24", "wine", "regsvr32", "/s", dos], env=env, timeout=300)
|
||||
wait_wine(env, 120)
|
||||
notes.append("regsvr32 %s exit %s" % (rel, result.returncode))
|
||||
if result.returncode != 0 and status == 0:
|
||||
status = result.returncode
|
||||
subprocess.run(["wineserver", "-k"], env=env)
|
||||
time.sleep(2)
|
||||
elapsed = int(time.time() - started_at)
|
||||
print("== %s unpacked and registered, exit %s after %d s" % (installer.name, status, elapsed), flush=True)
|
||||
else:
|
||||
for path, run_args in runs:
|
||||
exit_code, took = run_installer(path, run_args, timeout, path.name)
|
||||
elapsed += took
|
||||
status = exit_code if status in (None, 0) else status
|
||||
|
||||
print("== noting the prefix after", flush=True)
|
||||
files_after = snapshot_files(drive_c)
|
||||
reg_after = {"HKLM": parse_reg(prefix / "system.reg"), "HKCU": parse_reg(prefix / "user.reg")}
|
||||
|
||||
placed = {rel: info for rel, info in files_after.items() if files_before.get(rel) != info}
|
||||
removed = sorted(rel for rel in files_before if rel not in files_after)
|
||||
registry = diff_registry(reg_before["HKLM"], reg_after["HKLM"], "HKLM", home_user) + \
|
||||
diff_registry(reg_before["HKCU"], reg_after["HKCU"], "HKCU", home_user)
|
||||
|
||||
# Where each recorded file is read from in the recording prefix (differs from its recorded
|
||||
# path once a 32-bit recording is moved into the 64-bit layout).
|
||||
source_of = {rel: rel for rel in placed}
|
||||
layout = arch
|
||||
if arch == "win32" and recipe.get("remap", True) and (placed or registry):
|
||||
moved = {}
|
||||
for rel, info in placed.items():
|
||||
target = remap_file(rel)
|
||||
moved[target] = info
|
||||
source_of[target] = rel
|
||||
shifted = sum(1 for rel in placed if remap_file(rel) != rel)
|
||||
placed = moved
|
||||
removed = [remap_file(rel) for rel in removed]
|
||||
# The strings the installer wrote point where it put the files; on 64-bit Windows a 32-bit
|
||||
# installer resolves "Program Files" to "Program Files (x86)" and writes that. (system32
|
||||
# may stay: the file-system redirector sends a 32-bit reader to syswow64 by itself.)
|
||||
program_files = re.compile(r"([A-Za-z]:\\)Program Files\\(?!\(x86\))", re.IGNORECASE)
|
||||
rewritten = 0
|
||||
for entry in registry:
|
||||
data = entry.get("data")
|
||||
if isinstance(data, str):
|
||||
new = program_files.sub(r"\1Program Files (x86)\\", data)
|
||||
elif isinstance(data, list):
|
||||
new = [program_files.sub(r"\1Program Files (x86)\\", item) for item in data]
|
||||
else:
|
||||
continue
|
||||
if new != data:
|
||||
entry["data"] = new
|
||||
rewritten += 1
|
||||
if rewritten:
|
||||
notes.append("%d registry strings now say Program Files (x86)" % rewritten)
|
||||
registry = replay_into_win64(registry, work, home_user, notes)
|
||||
notes.append("32-bit recording laid out as on 64-bit Windows: %d of %d files moved (syswow64, Program Files (x86))" % (shifted, len(placed)))
|
||||
layout = "win64"
|
||||
|
||||
print("== opening the installer to find the placed files inside it", flush=True)
|
||||
extract_dir = Path(args.extract) if args.extract else work / "x"
|
||||
extract_dir.mkdir(parents=True, exist_ok=True)
|
||||
inside, tried = {}, []
|
||||
for index, (path, _) in enumerate(runs):
|
||||
folder = extract_dir / ("installer%d" % index) if len(runs) > 1 else extract_dir
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
found, used = extract_installer(path, folder, notes)
|
||||
for digest, rel in found.items():
|
||||
inside.setdefault(digest, (("installer%d/" % index) if len(runs) > 1 else "") + rel)
|
||||
tried = tried or used
|
||||
|
||||
# When the installer cannot be opened (Inno Setup newer than innoextract reads, K-Lite) and
|
||||
# the recipe says its contents may be redistributed, the placed files travel as an archive
|
||||
# beside the recording instead of being pulled from the installer on the device.
|
||||
host_files = bool(recipe.get("host_files")) and bool(placed)
|
||||
files_archive = None
|
||||
if not placed:
|
||||
notes.append("the installer placed no files (exit %s): nothing to lay out" % status)
|
||||
if host_files:
|
||||
archive = out / ("%s.files.tar.xz" % args.component)
|
||||
with tempfile.TemporaryDirectory(prefix="files-") as staging:
|
||||
for rel in placed:
|
||||
target = Path(staging) / "drive_c" / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(drive_c / source_of[rel], target)
|
||||
run(["tar", "-C", staging, "-cJf", str(archive), "drive_c"], timeout=1800, check=True)
|
||||
files_archive = {"name": archive.name, "size": archive.stat().st_size, "sha256": sha256_of(archive),
|
||||
"licence": recipe.get("licence", "")}
|
||||
print(" files archive: %s (%d bytes)" % (archive.name, files_archive["size"]), flush=True)
|
||||
files, inline_total, missing = [], 0, 0
|
||||
for rel in sorted(placed):
|
||||
size, digest = placed[rel]
|
||||
entry = {"path": rel, "size": size, "sha256": digest}
|
||||
if host_files:
|
||||
entry["archived"] = True
|
||||
elif digest in inside:
|
||||
entry["source"] = inside[digest]
|
||||
elif size <= INLINE_FILE_LIMIT and inline_total + size <= INLINE_TOTAL_LIMIT:
|
||||
entry["data"] = base64.b64encode((drive_c / source_of[rel]).read_bytes()).decode("ascii")
|
||||
inline_total += size
|
||||
else:
|
||||
entry["missing"] = True
|
||||
missing += 1
|
||||
files.append(entry)
|
||||
|
||||
snapshot = {
|
||||
"schema": 1,
|
||||
"component": args.component,
|
||||
"recorded": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"wine": wine_version, "arch": arch, "layout": layout,
|
||||
"installer": {"name": installer.name, "size": installer.stat().st_size, "sha256": sha256_of(installer),
|
||||
"args": runs[0][1], "exit": status, "seconds": elapsed, "extractors": tried},
|
||||
"installers": [{"name": path.name, "size": path.stat().st_size, "sha256": sha256_of(path), "args": run_args} for path, run_args in runs],
|
||||
"after": recipe.get("after", []), "winver": winver,
|
||||
"stats": {"files": len(files), "from_installer": sum(1 for f in files if "source" in f),
|
||||
"inline": sum(1 for f in files if "data" in f), "archived": sum(1 for f in files if f.get("archived")),
|
||||
"missing": missing, "removed": len(removed), "registry": len(registry)},
|
||||
"files_archive": files_archive,
|
||||
"files": files, "removed": removed, "registry": registry, "notes": notes,
|
||||
}
|
||||
(out / ("%s.snapshot.json" % args.component)).write_text(json.dumps(snapshot, indent=1), "utf-8")
|
||||
|
||||
by_hive = {}
|
||||
for value in registry:
|
||||
by_hive.setdefault(value["hive"], set()).add(value["key"].split("\\")[0] + "\\" + value["key"].split("\\")[1] if "\\" in value["key"] else value["key"])
|
||||
summary = ["# %s" % args.component, "",
|
||||
"Recorded %s on %s (%s prefix%s). Installer `%s`, exit %s after %d s." % (
|
||||
snapshot["recorded"], wine_version, arch, ", laid out for win64" if layout != arch else "", installer.name, status, elapsed), "",
|
||||
"| | |", "|---|---|",
|
||||
"| files placed | %d |" % len(files),
|
||||
"| of which found inside the installer | %d |" % snapshot["stats"]["from_installer"],
|
||||
"| carried in the files archive beside this recording | %d |" % snapshot["stats"]["archived"],
|
||||
"| carried inline (generated, small) | %d |" % snapshot["stats"]["inline"],
|
||||
"| missing (generated, large) | %d |" % missing,
|
||||
"| files removed | %d |" % len(removed),
|
||||
"| registry values | %d |" % len(registry), "",
|
||||
"## Where the files went", ""]
|
||||
folders = {}
|
||||
for entry in files:
|
||||
top = "/".join(entry["path"].split("/")[:3])
|
||||
folders[top] = folders.get(top, 0) + 1
|
||||
summary += ["- `%s` · %d" % (top, n) for top, n in sorted(folders.items(), key=lambda item: -item[1])[:25]]
|
||||
summary += ["", "## Registry keys touched (top two levels)", ""]
|
||||
for hive, keys in sorted(by_hive.items()):
|
||||
for key in sorted(keys)[:40]:
|
||||
summary.append("- %s\\%s" % (hive, key))
|
||||
if missing:
|
||||
summary += ["", "## Missing (not inside the installer, too large to carry)", ""]
|
||||
summary += ["- `%s` (%d bytes)" % (f["path"], f["size"]) for f in files if f.get("missing")][:50]
|
||||
if notes:
|
||||
summary += ["", "## Notes", ""] + ["- " + n for n in notes]
|
||||
(out / ("%s.summary.md" % args.component)).write_text("\n".join(summary) + "\n", "utf-8")
|
||||
print("\n".join(summary[:12]), flush=True)
|
||||
shutil.rmtree(work, ignore_errors=True)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user