mirror of
https://github.com/project-dorado/dorado-hd.git
synced 2026-10-11 21:10:01 +00:00
feat(re): full 109-module Zune HD decompilation corpus with symbol recovery + analysis docs
This commit is contained in:
John Smith
@@ -142,6 +142,13 @@ packages with title / GUID / description. Regenerable from
|
||||
archive. Binaries are encrypted (AES-ECB, marketplace DRM) and
|
||||
**never** bundled; metadata only.
|
||||
|
||||
The full firmware reverse-engineering corpus (109 reconstructed ARM32 PE modules,
|
||||
66,598 functions decompiled with recovered symbols, 22,479 strings) is built by
|
||||
`scripts/ghidra_corpus.py` into the out-of-tree `zune-hd-disassembly/`. Synthesized
|
||||
analysis lives in [`docs/zune-hd-module-inventory.md`](docs/zune-hd-module-inventory.md),
|
||||
[`docs/zune-hd-api-reference.md`](docs/zune-hd-api-reference.md) and
|
||||
[`docs/zune-hd-assets.md`](docs/zune-hd-assets.md).
|
||||
|
||||
### Tests
|
||||
|
||||
157 Robolectric unit tests + a design-invariant source scanner that
|
||||
|
||||
@@ -234,7 +234,7 @@ referencing `dorado/src/Dorado.Application`); re-run it after any desktop
|
||||
|
||||
1. **M10 closeout** (S–M) — richer lock-screen Now Playing art/controls + optional sleep timer.
|
||||
2. **M5/M6 community wishlist** (M) — share-sheet + Zune-Card artwork export, EQ presets, crossfade, live-radio cache.
|
||||
3. **Corpus mining** (M) — mine the exported Ghidra decompilation for behavioral ground truth to tighten the canon.
|
||||
3. **Corpus mining** (M) — the full 109-module corpus is built (66.6k functions, symbols applied); mine the decompilation/strings for behavioral ground truth to tighten the canon.
|
||||
|
||||
Front-loads the remaining always-on surfaces, then the community wishlist and
|
||||
disassembly-driven fidelity work.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,57 @@
|
||||
# Zune HD firmware asset inventory (disassembly corpus)
|
||||
|
||||
Non-module files extracted from `PavoBaseline.Cab` (ROM file table) into the
|
||||
out-of-tree `zune-hd-disassembly/assets/`. Raw assets are provisioned by the
|
||||
device firmware and are **never committed**; this document is the synthesized
|
||||
inventory only.
|
||||
|
||||
## Fonts (`assets/fonts/`, 28 files)
|
||||
|
||||
- **Authentic Zegoe UI** (Microsoft): `ZegoeUI.ttf`, `_B`, `_Blk`, `_L`, `_SB`,
|
||||
`_SL` — the device UI typeface. Dorado-HD ships metric-compatible **Selawik**
|
||||
(OFL) instead; users may supply Zegoe locally.
|
||||
- **CJK**: `MalgunForZune.ttf` (Korean), `MeiryoForZune.ttf` (Japanese).
|
||||
- **Core CE/desktop set**: Arial, Courier New, Georgia, Impact, Times New Roman,
|
||||
Trebuchet MS, Verdana, Webdings, Wingdings (Web/CE rendering fallbacks).
|
||||
|
||||
## Multimedia / codec firmware
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `nvmm_h264dec.axf`, `nvmm_mpeg4dec.axf`, `nvmm_adtsdec.axf`, `nvmm_jpegdec.axf`, `nvmm_jpegenc.axf` | Tegra APX 2600 codec firmware images (H.264, MPEG-4, AAC-ADTS, JPEG decode/encode). |
|
||||
| `nvddk_audiomixer_core.axf`, `nvrm_avp.axf` | Audio-mixer core and the Tegra AVP (audio/video processor) firmware. |
|
||||
| `msadpcm.dll` | Microsoft ADPCM codec (WinCE). |
|
||||
| `runtimeZune.v3.1.zcp` | The **XNA Game Studio 3.1 runtime** package (4.3 MB). NX container flagged `encrypted = True` with no readable manifest — i.e. marketplace DRM (AES-ECB); it is the same container family handled by `dorado-emu/Dorado.Containers`. |
|
||||
| `TVOut.exe` | Small Windows CE ARM PE (TV-out helper). |
|
||||
|
||||
## Networking / radio firmware
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `Acs_045E_081C_3_90_ClutchFW.hex`, `Acs_045E_091A_4_26_ConcertoFW.hex` | Intel-HEX touch-controller firmware (`045E` = Microsoft VID): "Clutch" and "Concerto" controllers. |
|
||||
| `athwlan2_0.bin.z77`, `data.patch.hw2_0.bin` | Atheros Wi-Fi firmware (LZ77 "z77" compressed) + patch. |
|
||||
| `calData_sd21-055-d0513.bin` | Wi-Fi calibration data. |
|
||||
|
||||
## Identity / provisioning
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `sysroots.p7b`, `zuneca.p7b` | PKCS#7 certificate bundles (trust roots / CA chain). |
|
||||
| `zuneroots.p7b` | PGP-encrypted root key material. |
|
||||
| `zbinfs_EXT.hash` | Filesystem (binfs) hash table for the EXT partition. |
|
||||
| `hdrfirmware.bin` | Discrete HD-radio (HD-Radio) firmware blob. |
|
||||
|
||||
## UI chrome / OS resources
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `stdsm.bmp`, `viewsm.bmp`, `stdsm.2bp`, `viewsm.2bp`, `close.2bp`, `ok.2bp` | Boot / status bitmaps and `.2bp` mask pairs. |
|
||||
| `DefaultAd.png`, `Dismiss.png` / `DismissESP.png` / `DismissFRA.png`, `devicon.ico` | Marketplace/notification chrome. |
|
||||
| `Dictionary.dat` | Custom **`ZDCT`** (Zune Dictionary) data file, 1.28 MB — spelling/dictionary resource. |
|
||||
|
||||
## What this pass did not recover
|
||||
|
||||
- `runtimeZune.v3.1.zcp` payload contents (marketplace DRM; no public key).
|
||||
- `zuneroots.p7b` (PGP-encrypted).
|
||||
- Field semantics of the `XuiTouchSettings` struct (see
|
||||
`zune-hd-touch-settings.md` §4 — no PDB/symbol data exists in the corpus).
|
||||
+41
-29
@@ -139,46 +139,58 @@ In Dorado-HD, **Selawik** (OFL) is shipped by default as the metric-compatible o
|
||||
|
||||
## 7. Automated Disassembly Pipeline
|
||||
|
||||
The repo provides an end-to-end extraction and decompilation script:
|
||||
Two scripts make up the pipeline:
|
||||
|
||||
```bash
|
||||
# Extract assets and reconstruct valid ARM32 PE binaries (< 10 seconds):
|
||||
# 1. Extract assets and reconstruct the 109 ARM32 PE modules (< 10 seconds):
|
||||
python3 scripts/disassemble_zune_hd.py --extract-only
|
||||
|
||||
# Run deep Ghidra headless analysis + decompilation export on gemstone.exe and xuidll.dll:
|
||||
python3 scripts/disassemble_zune_hd.py --ghidra
|
||||
# 2. Build the reverse-engineering corpus (all modules, symbols, full decompile):
|
||||
python3 scripts/ghidra_corpus.py --tier all --jobs 8
|
||||
# or a subset: --tier 1 | --modules gemstone.exe,xuidll.dll
|
||||
```
|
||||
|
||||
The `--ghidra` step imports/analyses each shell binary and runs the
|
||||
`scripts/ghidra/ExportDecompiled.java` post-script, which writes per-function
|
||||
decompiled C files and a `<module>.functions.json` index (name, entry, size,
|
||||
thunk flag, decompiled line count) so the corpus can be grepped without
|
||||
re-running Ghidra.
|
||||
`ghidra_corpus.py` is the corpus builder. For each module it:
|
||||
|
||||
1. **recovers named exports/imports** with `rizin` (`iEj`/`iij`) into
|
||||
`ghidra/symbols/<module>.symbols.json`;
|
||||
2. **imports the module into its own headless-Ghidra project**, applies the
|
||||
recovered symbol names (`scripts/ghidra/ExportDecompiled.java`), decompiles
|
||||
**every** function, and exports per-function `.c` files plus
|
||||
`<module>.functions.json` (name/entry/size/signature/isExport/lines) and
|
||||
`<module>.strings.json` (string + owning function);
|
||||
3. records the result in `ghidra/corpus-manifest.json`.
|
||||
|
||||
`disassemble_zune_hd.py --ghidra` delegates to this builder.
|
||||
|
||||
### Coverage (2026-09-10)
|
||||
|
||||
- **109 / 109** reconstructed modules analysed (ARM32 PE, Ghidra 12.1.2);
|
||||
- **66,598** functions decompiled (gemstone 2,737; xuidll 2,288; mshtml 13,036; …);
|
||||
- **4,169** named exports and **10,525** imports recovered;
|
||||
- **22,479** strings indexed with their owning function;
|
||||
- 1 resource-only module (`shdoclc.dll`) has no code.
|
||||
|
||||
Raw decompiled C, symbol dumps and string dumps are Microsoft-derived and stay in
|
||||
the external corpus; the repository keeps only synthesized metadata
|
||||
([`zune-hd-module-inventory.md`](zune-hd-module-inventory.md),
|
||||
[`zune-hd-api-reference.md`](zune-hd-api-reference.md),
|
||||
[`zune-hd-assets.md`](zune-hd-assets.md)).
|
||||
|
||||
### Output Directory Structure (`zune-hd-disassembly/`)
|
||||
|
||||
```
|
||||
zune-hd-disassembly/
|
||||
├── firmware/
|
||||
│ ├── PavoBaseline.Cab
|
||||
│ ├── EXT.bin
|
||||
│ └── NK.bin
|
||||
├── assets/
|
||||
│ ├── fonts/
|
||||
│ │ ├── ZegoeUI.ttf, ZegoeUI_B.ttf, ZegoeUI_Blk.ttf...
|
||||
│ │ └── MeiryoForZune.ttf, MalgunForZune.ttf...
|
||||
│ ├── DefaultAd.png, Dismiss.png, Zune.png...
|
||||
│ └── runtimeZune.v3.1.zcp
|
||||
├── modules/
|
||||
│ ├── gemstone.exe, xuidll.dll, zhud_serv.dll...
|
||||
│ └── (109 reconstructed ARM32 PE binaries)
|
||||
├── firmware/ PavoBaseline.Cab, EXT.bin, NK.bin, Recovery.bin, ZBoot.bin
|
||||
├── assets/ 60 ROM files (fonts/, Tegra .axf, runtimeZune.v3.1.zcp, …)
|
||||
├── modules/ 109 reconstructed ARM32 PE binaries
|
||||
├── ghidra/
|
||||
│ ├── project/
|
||||
│ │ └── ZuneHD_Project.gpr / ZuneHD_Project.rep
|
||||
│ └── decompiled/
|
||||
│ ├── gemstone.exe/ (per-function .c)
|
||||
│ ├── gemstone.exe.functions.json
|
||||
│ ├── xuidll.dll/
|
||||
│ └── xuidll.dll.functions.json
|
||||
│ ├── symbols/ <module>.symbols.json (recovered exports/imports)
|
||||
│ ├── decompiled/ <module>/ (per-function .c), <module>.functions.json,
|
||||
│ │ <module>.strings.json
|
||||
│ ├── logs/ per-module Ghidra logs
|
||||
│ ├── corpus-manifest.json status/size/hash/counts per module
|
||||
│ ├── index.json aggregate module index
|
||||
│ └── projects/ ephemeral per-module Ghidra projects (deleted after run)
|
||||
└── README.md
|
||||
```
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
# Zune HD module inventory (disassembly corpus)
|
||||
|
||||
Generated from the out-of-tree `zune-hd-disassembly/` corpus by
|
||||
`scripts/ghidra_corpus.py` (rizin symbol recovery + full Ghidra decompilation).
|
||||
Raw decompilation/symbols stay external; **this table is synthesized metadata only.**
|
||||
|
||||
**Coverage:** 109/109 reconstructed ARM32 PE modules analysed; 66,598 functions
|
||||
decompiled; 4,169 named exports and 10,525 imports recovered.
|
||||
|
||||
| Module | Bytes | Exports | Functions | Role |
|
||||
|---|---:|---:|---:|---|
|
||||
| `AAXSDKWin.dll` | 113,152 | 74 | 360 | Audible/AAX audio SDK (spoken-word playback) |
|
||||
| `ASAFilter.dll` | 39,936 | 5 | 193 | System DLL |
|
||||
| `Dw.exe` | 60,928 | 0 | 260 | Native executable |
|
||||
| `IECEExt.dll` | 11,776 | 43 | 117 | System DLL |
|
||||
| `MtpSvc.dll` | 375,296 | 5 | 1,585 | MTP device service (USB media transfer) |
|
||||
| `ZWmtStreamer.dll` | 390,144 | 5 | 1,485 | WMA/WMV streaming (PlayReady/WMDRM) |
|
||||
| `abufilter.dll` | 43,520 | 5 | 241 | System DLL |
|
||||
| `acsserv.dll` | 74,240 | 5 | 296 | System DLL |
|
||||
| `athsrvc.dll` | 5,120 | 8 | 36 | System DLL |
|
||||
| `audible.dll` | 48,128 | 5 | 267 | System DLL |
|
||||
| `battdrvr.dll` | 10,240 | 10 | 71 | System DLL |
|
||||
| `ceshell.dll` | 278,528 | 81 | 883 | Windows CE shell |
|
||||
| `commctrl.dll` | 375,808 | 77 | 1,456 | Common controls |
|
||||
| `commdlg.dll` | 87,040 | 5 | 245 | System DLL |
|
||||
| `compclient.dll` | 54,784 | 190 | 277 | Composition client library |
|
||||
| `compositor.exe` | 68,096 | 0 | 331 | Hardware window compositor |
|
||||
| `connectionui.dll` | 38,400 | 1 | 278 | System DLL |
|
||||
| `credprov.dll` | 4,608 | 6 | 25 | System DLL |
|
||||
| `crypt32.dll` | 388,096 | 112 | 1,502 | CryptoAPI (TLS/certs) |
|
||||
| `ddraw.dll` | 25,600 | 3 | 180 | System DLL |
|
||||
| `dhcpsrv.dll` | 5,632 | 5 | 32 | System DLL |
|
||||
| `gemstone.exe` | 558,080 | 0 | 2,737 | Primary Zune HD shell — Metro scene graph, gesture routing, view lifecycle |
|
||||
| `hostloader.dll` | 33,792 | 1 | 193 | System DLL |
|
||||
| `ietheme.dll` | 46,592 | 9 | 57 | System DLL |
|
||||
| `imaging.dll` | 306,688 | 2 | 916 | System DLL |
|
||||
| `imgutil.dll` | 52,224 | 12 | 281 | System DLL |
|
||||
| `iphlpapi.dll` | 51,200 | 62 | 270 | System DLL |
|
||||
| `jscript.dll` | 709,120 | 4 | 2,010 | JScript engine (zie.exe) |
|
||||
| `jsproxy.dll` | 18,432 | 92 | 85 | System DLL |
|
||||
| `libEGL.dll` | 74,240 | 36 | 320 | OpenGL ES / EGL stack |
|
||||
| `libGLESv2.dll` | 263,680 | 143 | 931 | OpenGL ES / EGL stack |
|
||||
| `libKD.dll` | 84,992 | 173 | 544 | Kernel/user debug + crypto helpers |
|
||||
| `libnvaudio_capture.dll` | 51,200 | 5 | 288 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvaudio_encoder.dll` | 50,176 | 5 | 251 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvcwm.dll` | 11,776 | 32 | 99 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvddk_audiomixer.dll` | 5,120 | 11 | 30 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_audio.dll` | 1,358,848 | 24 | 2,082 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_image.dll` | 176,128 | 4 | 504 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_manager.dll` | 5,632 | 9 | 27 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_misc.dll` | 29,696 | 4 | 150 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_service.dll` | 6,144 | 5 | 47 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_tracklist.dll` | 7,168 | 15 | 38 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_video.dll` | 107,520 | 8 | 330 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_videorenderer.dll` | 17,920 | 1 | 107 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmm_writer.dll` | 56,320 | 2 | 101 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvmux_file_writer.dll` | 44,544 | 5 | 242 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvodm_audiocodec.dll` | 17,408 | 12 | 86 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvodm_dtvtuner.dll` | 9,728 | 5 | 65 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvodm_imager.dll` | 160,768 | 10 | 184 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvomx.dll` | 169,472 | 22 | 561 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvsm.dll` | 49,664 | 47 | 176 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvvideo_capture.dll` | 83,968 | 5 | 427 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvvideo_encoder.dll` | 57,856 | 5 | 297 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvvideo_transform_filter.dll` | 58,368 | 3 | 342 | NVIDIA Tegra multimedia driver/library |
|
||||
| `libnvxdrm.dll` | 6,144 | 7 | 39 | NVIDIA Tegra multimedia driver/library |
|
||||
| `mlang.dll` | 133,632 | 14 | 392 | System DLL |
|
||||
| `msacmce.dll` | 18,944 | 13 | 83 | System DLL |
|
||||
| `msasn1.dll` | 42,496 | 139 | 213 | System DLL |
|
||||
| `msdmo.dll` | 26,112 | 19 | 91 | System DLL |
|
||||
| `mshtml.dll` | 4,115,456 | 6 | 13,036 | Trident HTML engine (zie.exe) |
|
||||
| `msls31.dll` | 199,168 | 80 | 558 | System DLL |
|
||||
| `msxml3.dll` | 547,328 | 10 | 2,039 | MSXML3 (feed/web parsing) |
|
||||
| `netmui.dll` | 4,096 | 0 | 10 | System DLL |
|
||||
| `nspm.dll` | 10,240 | 1 | 51 | System DLL |
|
||||
| `ole32.dll` | 166,912 | 40 | 915 | COM runtime |
|
||||
| `oleaut32.dll` | 176,640 | 231 | 654 | OLE Automation |
|
||||
| `pngfilt.dll` | 45,056 | 4 | 226 | System DLL |
|
||||
| `schannel.dll` | 92,160 | 37 | 319 | TLS/SSL provider |
|
||||
| `secur32.dll` | 12,288 | 25 | 74 | SSPI security provider |
|
||||
| `services.exe` | 11,264 | 0 | 66 | Native executable |
|
||||
| `shcore.dll` | 10,752 | 31 | 76 | System DLL |
|
||||
| `shdoclc.dll` | 127,488 | 0 | 0 | IE shell docs (resource-only; 0 code functions) |
|
||||
| `shdocvw.dll` | 420,352 | 123 | 1,712 | Shell doc/view (IE browser control) |
|
||||
| `shellcelog.dll` | 9,728 | 5 | 65 | System DLL |
|
||||
| `shlwapi.dll` | 130,048 | 480 | 460 | System DLL |
|
||||
| `ssllsp.dll` | 30,208 | 3 | 186 | System DLL |
|
||||
| `udp2tcp.exe` | 8,704 | 0 | 66 | Native executable |
|
||||
| `urlmon.dll` | 301,056 | 53 | 1,161 | URL moniker / download |
|
||||
| `wininet.dll` | 453,632 | 166 | 1,723 | WinINet (HTTP/FTP) |
|
||||
| `winsock.dll` | 6,144 | 33 | 75 | Winsock 1.1 |
|
||||
| `wmadmod.dll` | 531,968 | 5 | 952 | System DLL |
|
||||
| `wmdrm.dll` | 419,840 | 63 | 1,068 | System DLL |
|
||||
| `wmvdmod.dll` | 881,152 | 3 | 1,063 | System DLL |
|
||||
| `ws2serv.dll` | 30,720 | 9 | 294 | Winsock 2 service |
|
||||
| `wspm.dll` | 8,704 | 1 | 53 | System DLL |
|
||||
| `wzcsapi.dll` | 8,192 | 14 | 61 | Wireless Zero Config (Wi-Fi) API |
|
||||
| `xhttpdll.dll` | 80,896 | 31 | 266 | HTTP client library |
|
||||
| `xmllite.dll` | 154,112 | 6 | 563 | System DLL |
|
||||
| `xnalauncher.exe` | 157,696 | 0 | 696 | XNA app launcher/host |
|
||||
| `xuidll.dll` | 465,408 | 529 | 2,288 | Xbox UI (XUI) runtime ported to WinCE — layout, animation, touch dispatch |
|
||||
| `zam_serv.dll` | 93,184 | 5 | 532 | Zune Application Manager (XNA mini-app lifecycles) |
|
||||
| `zcab.dll` | 50,176 | 12 | 252 | CAB decompression helper |
|
||||
| `zconfig_serv.dll` | 35,328 | 5 | 207 | Device configuration service |
|
||||
| `zcontent_serv.dll` | 56,832 | 5 | 228 | Media library indexing + metadata/search service |
|
||||
| `zcredentials_serv.dll` | 15,360 | 5 | 118 | Credential/DRM key service |
|
||||
| `zd3d.dll` | 185,344 | 62 | 237 | Direct3D-Mobile interface (D3DX helpers) over Tegra APX |
|
||||
| `zdknet.dll` | 68,096 | 38 | 382 | ZDK networking API |
|
||||
| `zdksystem.dll` | 135,168 | 359 | 996 | Zune Development Kit system API (app-facing framework) |
|
||||
| `zhud_serv.dll` | 189,440 | 5 | 1,167 | Playback/volume Heads-Up-Display and lock-shade service |
|
||||
| `zie.exe` | 197,632 | 0 | 1,121 | Zune Internet Explorer shell |
|
||||
| `ziehooks.dll` | 5,632 | 6 | 28 | Internet Explorer shell hooks |
|
||||
| `zlib.dll` | 37,376 | 18 | 74 | zlib decompression (WinCE port) |
|
||||
| `zmassive.dll` | 87,552 | 19 | 572 | Massive (audio/marketplace) client library |
|
||||
| `zmedia_serv.dll` | 273,920 | 5 | 1,221 | Core media playback engine (pipeline, queue, EQ) |
|
||||
| `znet_serv.dll` | 492,544 | 5 | 2,046 | Networking service (download, marketplace transport) |
|
||||
| `zrender.dll` | 211,968 | 3 | 614 | High-level scene/texture rendering engine |
|
||||
| `zserial.dll` | 19,456 | 18 | 114 | Serial/device identity helper |
|
||||
| `zsplash.exe` | 18,944 | 0 | 107 | Animated startup bootsplash |
|
||||
| `zuncab.exe` | 145,408 | 0 | 90 | Zune CAB extractor/installer |
|
||||
|
||||
## Notable modules
|
||||
|
||||
- **gemstone.exe** (2,737 fns) — the shell; scene classes `GemStartScene`, `GemPivotScene`,
|
||||
`GemNowPlayingScene`, `GemLibraryAlbumGridContent`, `GemLibraryLetterPickerScene`.
|
||||
- **xuidll.dll** (2,288 fns, 529 exports) — the XUI engine: `XuiElement*`, `XuiText*`,
|
||||
`XuiTouch*`, `XuiScroll*`, `XuiProcessMultiTouchMessage`, `XuiSetTouchSettings`.
|
||||
- **zdksystem.dll** (996 fns, 359 exports) — the app-facing ZDK framework API.
|
||||
- **zd3d.dll** (237 fns, 62 exports) — D3DX/`D3DXMatrix*` helpers over Direct3D-Mobile.
|
||||
- **zmedia_serv.dll** (1,221 fns) / **zhud_serv.dll** (1,167 fns) — playback + HUD.
|
||||
- **znet_serv.dll** (2,046 fns) — networking/marketplace transport.
|
||||
- **shdoclc.dll** — resource-only (0 code functions).
|
||||
|
||||
> Raw per-function decompiled C and the symbol/string dumps live only in the external
|
||||
> corpus (`ghidra/decompiled/`, `ghidra/symbols/`) and are never committed.
|
||||
@@ -293,49 +293,18 @@ def extract_modules(raw, img_start, romhdr_off, nummods):
|
||||
return cur
|
||||
|
||||
|
||||
def run_ghidra_analysis(module_names=None):
|
||||
ghidra_headless = "/home/john/.local/ghidra/support/analyzeHeadless"
|
||||
if not os.path.exists(ghidra_headless):
|
||||
log("Ghidra analyzeHeadless not found. Skipping decompilation.")
|
||||
def run_ghidra_analysis(module_names=None, tier="all"):
|
||||
"""Delegate to the corpus runner: recover symbols, decompile every function
|
||||
for the selected modules, and write the manifest. Raw output stays external."""
|
||||
runner = os.path.join(os.path.dirname(os.path.abspath(__file__)), "ghidra_corpus.py")
|
||||
if not os.path.exists(runner):
|
||||
log("ghidra_corpus.py not found; skipping decompilation.")
|
||||
return
|
||||
|
||||
if module_names is None:
|
||||
module_names = ["gemstone.exe", "xuidll.dll"]
|
||||
|
||||
os.makedirs(GHIDRA_DIR, exist_ok=True)
|
||||
os.makedirs(DECOMPILED_DIR, exist_ok=True)
|
||||
project_dir = os.path.join(GHIDRA_DIR, "project")
|
||||
project_name = "ZuneHD_Project"
|
||||
os.makedirs(project_dir, exist_ok=True)
|
||||
|
||||
# The post-script exports per-function decompiled C plus a JSON function index
|
||||
# (name/entry/size) so the corpus is greppable without re-running Ghidra.
|
||||
script_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "ghidra")
|
||||
post_script = os.path.join(script_dir, "ExportDecompiled.java")
|
||||
|
||||
log(f"Running headless Ghidra analysis on {module_names}...")
|
||||
for mod in module_names:
|
||||
mod_path = os.path.join(MODULES_DIR, mod)
|
||||
if not os.path.exists(mod_path):
|
||||
log(f"Module {mod_path} not found, skipping.")
|
||||
continue
|
||||
|
||||
cmd = [
|
||||
ghidra_headless,
|
||||
project_dir,
|
||||
project_name,
|
||||
"-import", mod_path,
|
||||
"-overwrite",
|
||||
"-scriptPath", script_dir,
|
||||
]
|
||||
if os.path.exists(post_script):
|
||||
cmd += ["-postScript", "ExportDecompiled.java", DECOMPILED_DIR]
|
||||
log(f"Invoking Ghidra for {mod}...")
|
||||
res = run_cmd(cmd)
|
||||
if res.returncode == 0:
|
||||
log(f"Successfully analyzed {mod} in Ghidra project {project_name}.")
|
||||
else:
|
||||
log(f"Ghidra analysis finished with code {res.returncode}.")
|
||||
cmd = [sys.executable, runner, "--tier", tier]
|
||||
if module_names:
|
||||
cmd = [sys.executable, runner, "--modules", ",".join(module_names)]
|
||||
log(f"Running corpus builder: {' '.join(cmd)}")
|
||||
subprocess.run(cmd)
|
||||
|
||||
|
||||
def write_readme():
|
||||
@@ -363,11 +332,20 @@ never be committed, packaged, or shipped.
|
||||
- `zmedia_serv.dll`: Core playback engine.
|
||||
- `zd3d.dll` & `zrender.dll`: Tegra APX 2600 Direct3D-Mobile hardware renderer.
|
||||
- `zam_serv.dll`: Zune Application Manager.
|
||||
- `ghidra/`: Headless Ghidra project and decompiled function analysis.
|
||||
- `ghidra/`: Headless Ghidra corpus (symbols, decompiled functions, string indexes,
|
||||
`corpus-manifest.json`) produced by `scripts/ghidra_corpus.py`.
|
||||
|
||||
## Regenerating
|
||||
|
||||
Run `python3 scripts/disassemble_zune_hd.py` from the `dorado-hd` repository root.
|
||||
```bash
|
||||
python3 scripts/disassemble_zune_hd.py --extract-only # assets + 109 ARM32 PE modules
|
||||
python3 scripts/ghidra_corpus.py --tier all --jobs 8 # symbols + full decompilation
|
||||
```
|
||||
|
||||
Coverage: 109/109 modules, ~66.6k functions decompiled, 4,169 named exports,
|
||||
22,479 strings indexed. Synthesized metadata is documented in the dorado-hd
|
||||
repository (`docs/zune-hd-module-inventory.md`, `docs/zune-hd-api-reference.md`,
|
||||
`docs/zune-hd-assets.md`).
|
||||
"""
|
||||
with open(readme_path, "w") as f:
|
||||
f.write(content)
|
||||
@@ -399,7 +377,7 @@ def main():
|
||||
write_readme()
|
||||
|
||||
if args.ghidra:
|
||||
run_ghidra_analysis(["gemstone.exe", "xuidll.dll"])
|
||||
run_ghidra_analysis(tier="all")
|
||||
|
||||
log("Pipeline execution complete.")
|
||||
|
||||
|
||||
@@ -1,21 +1,37 @@
|
||||
// Export decompiled C functions and a JSON function index for the current program.
|
||||
// Export decompiled C functions, a JSON function index, and a string index for the
|
||||
// current program. Optionally applies recovered export/import symbol names first.
|
||||
//
|
||||
// Run by the pipeline as a Ghidra post-script:
|
||||
// analyzeHeadless <proj dir> <proj name> -process <module> -noanalysis \
|
||||
// -scriptPath <dir> -postScript ExportDecompiled.java <output root> [maxFunctions]
|
||||
// analyzeHeadless <proj dir> <proj name> -import <module> \
|
||||
// -scriptPath <dir> \
|
||||
// -postScript ExportDecompiled.java <outputRoot> <maxFunctions|0=all> [symbolsJson]
|
||||
//
|
||||
//@category Zune
|
||||
|
||||
import com.google.gson.GsonBuilder;
|
||||
import com.google.gson.JsonArray;
|
||||
import com.google.gson.JsonElement;
|
||||
import com.google.gson.JsonObject;
|
||||
import com.google.gson.JsonParser;
|
||||
|
||||
import ghidra.app.decompiler.DecompileResults;
|
||||
import ghidra.app.decompiler.DecompInterface;
|
||||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.Address;
|
||||
import ghidra.program.model.data.StringDataInstance;
|
||||
import ghidra.program.model.listing.Data;
|
||||
import ghidra.program.model.listing.Function;
|
||||
import ghidra.program.model.listing.FunctionIterator;
|
||||
import ghidra.program.model.symbol.SourceType;
|
||||
import ghidra.util.task.ConsoleTaskMonitor;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.FileOutputStream;
|
||||
import java.io.FileReader;
|
||||
import java.io.OutputStreamWriter;
|
||||
import java.io.Writer;
|
||||
import java.util.HashSet;
|
||||
import java.util.Set;
|
||||
|
||||
public class ExportDecompiled extends GhidraScript {
|
||||
|
||||
@@ -23,42 +39,53 @@ public class ExportDecompiled extends GhidraScript {
|
||||
public void run() throws Exception {
|
||||
String[] args = getScriptArgs();
|
||||
String outRoot = args.length > 0 ? args[0] : ".";
|
||||
int maxDecompile = args.length > 1 ? Integer.parseInt(args[1]) : 3000;
|
||||
int maxDecompile = args.length > 1 ? Integer.parseInt(args[1]) : 0;
|
||||
String symbolsPath = args.length > 2 ? args[2] : null;
|
||||
|
||||
String module = currentProgram.getName().replaceAll("[^A-Za-z0-9._-]", "_");
|
||||
File moduleDir = new File(outRoot, module);
|
||||
moduleDir.mkdirs();
|
||||
|
||||
// ---- apply recovered export/import names (best effort) -----------------
|
||||
int renamed = 0;
|
||||
Set<Long> exportAddrs = new HashSet<Long>();
|
||||
if (symbolsPath != null && new File(symbolsPath).isFile()) {
|
||||
renamed = applySymbols(symbolsPath, exportAddrs);
|
||||
println("[ExportDecompiled] applied " + renamed + " symbols from " + symbolsPath);
|
||||
}
|
||||
|
||||
DecompInterface decomp = new DecompInterface();
|
||||
decomp.openProgram(currentProgram);
|
||||
ConsoleTaskMonitor monitor = new ConsoleTaskMonitor();
|
||||
|
||||
StringBuilder index = new StringBuilder("[\n");
|
||||
JsonArray index = new JsonArray();
|
||||
int total = 0;
|
||||
int done = 0;
|
||||
boolean first = true;
|
||||
|
||||
FunctionIterator functions = currentProgram.getFunctionManager().getFunctions(true);
|
||||
while (functions.hasNext()) {
|
||||
Function function = functions.next();
|
||||
total++;
|
||||
|
||||
String entry = function.getEntryPoint().toString();
|
||||
Address entryAddr = function.getEntryPoint();
|
||||
String entry = entryAddr.toString();
|
||||
String name = function.getName();
|
||||
long size = function.getBody().getNumAddresses();
|
||||
|
||||
if (!first) {
|
||||
index.append(",\n");
|
||||
JsonObject rec = new JsonObject();
|
||||
rec.addProperty("name", name);
|
||||
rec.addProperty("entry", entry);
|
||||
rec.addProperty("size", size);
|
||||
rec.addProperty("isThunk", function.isThunk());
|
||||
rec.addProperty("isExport", exportAddrs.contains(entryAddr.getUnsignedOffset()));
|
||||
try {
|
||||
rec.addProperty("signature", function.getPrototypeString(false, false));
|
||||
} catch (Exception ignore) {
|
||||
// signature not always available
|
||||
}
|
||||
first = false;
|
||||
|
||||
index.append(" {")
|
||||
.append("\"name\":\"").append(jsonEsc(name)).append("\",")
|
||||
.append("\"entry\":\"").append(entry).append("\",")
|
||||
.append("\"size\":").append(size).append(",")
|
||||
.append("\"isThunk\":").append(function.isThunk());
|
||||
|
||||
if (done < maxDecompile) {
|
||||
boolean shouldDecompile = (maxDecompile <= 0) || (done < maxDecompile);
|
||||
if (shouldDecompile) {
|
||||
DecompileResults result = decomp.decompileFunction(function, 60, monitor);
|
||||
if (result != null && result.decompileCompleted()) {
|
||||
String c = result.getDecompiledFunction().getC();
|
||||
@@ -68,23 +95,103 @@ public class ExportDecompiled extends GhidraScript {
|
||||
writer.write("// " + name + " @ " + entry + "\n");
|
||||
writer.write(c);
|
||||
}
|
||||
index.append(",\"lines\":").append(c.split("\n", -1).length);
|
||||
rec.addProperty("lines", c.split("\n", -1).length);
|
||||
done++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
index.append("}");
|
||||
index.add(rec);
|
||||
}
|
||||
|
||||
index.append("\n]\n");
|
||||
File indexPath = new File(outRoot, module + ".functions.json");
|
||||
try (Writer writer = new OutputStreamWriter(new FileOutputStream(indexPath), "UTF-8")) {
|
||||
writer.write(index.toString());
|
||||
try (Writer writer = new OutputStreamWriter(
|
||||
new FileOutputStream(new File(outRoot, module + ".functions.json")), "UTF-8")) {
|
||||
writer.write(new GsonBuilder().setPrettyPrinting().create().toJson(index));
|
||||
}
|
||||
|
||||
// ---- string index ------------------------------------------------------
|
||||
JsonArray strings = new JsonArray();
|
||||
for (Data data : currentProgram.getListing().getDefinedData(true)) {
|
||||
if (data == null) {
|
||||
continue;
|
||||
}
|
||||
Object value = null;
|
||||
try {
|
||||
value = data.getValue();
|
||||
} catch (Exception ignore) {
|
||||
continue;
|
||||
}
|
||||
if (!(value instanceof String)) {
|
||||
continue;
|
||||
}
|
||||
String text = (String) value;
|
||||
if (text == null || text.isEmpty() || text.length() > 4096) {
|
||||
continue;
|
||||
}
|
||||
JsonObject s = new JsonObject();
|
||||
s.addProperty("address", data.getAddress().toString());
|
||||
s.addProperty("value", text);
|
||||
Function owner = getFunctionContaining(data.getAddress());
|
||||
if (owner != null) {
|
||||
s.addProperty("function", owner.getName());
|
||||
s.addProperty("functionEntry", owner.getEntryPoint().toString());
|
||||
}
|
||||
strings.add(s);
|
||||
}
|
||||
try (Writer writer = new OutputStreamWriter(
|
||||
new FileOutputStream(new File(outRoot, module + ".strings.json")), "UTF-8")) {
|
||||
writer.write(new GsonBuilder().setPrettyPrinting().create().toJson(strings));
|
||||
}
|
||||
|
||||
println("[ExportDecompiled] " + module + ": " + total + " functions indexed, "
|
||||
+ done + " decompiled -> " + moduleDir.getAbsolutePath());
|
||||
+ done + " decompiled, " + strings.size() + " strings, "
|
||||
+ renamed + " symbols applied -> " + moduleDir.getAbsolutePath());
|
||||
}
|
||||
|
||||
private int applySymbols(String path, Set<Long> exportAddrs) {
|
||||
int renamed = 0;
|
||||
try (FileReader reader = new FileReader(path)) {
|
||||
JsonObject root = JsonParser.parseReader(reader).getAsJsonObject();
|
||||
renamed += applyList(root.getAsJsonArray("exports"), exportAddrs, true);
|
||||
renamed += applyList(root.getAsJsonArray("imports"), exportAddrs, false);
|
||||
} catch (Exception ex) {
|
||||
println("[ExportDecompiled] symbol apply failed: " + ex.getMessage());
|
||||
}
|
||||
return renamed;
|
||||
}
|
||||
|
||||
private int applyList(JsonArray list, Set<Long> exportAddrs, boolean isExport) {
|
||||
if (list == null) {
|
||||
return 0;
|
||||
}
|
||||
int n = 0;
|
||||
for (JsonElement element : list) {
|
||||
JsonObject o = element.getAsJsonObject();
|
||||
if (!o.has("name") || !o.has("vaddr")) {
|
||||
continue;
|
||||
}
|
||||
String name = o.get("name").getAsString();
|
||||
long vaddr = o.get("vaddr").getAsLong();
|
||||
if (name == null || name.isEmpty() || vaddr <= 0) {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
Address addr = toAddr(vaddr);
|
||||
Function function = getFunctionAt(addr);
|
||||
if (function != null) {
|
||||
function.setName(name, SourceType.IMPORTED);
|
||||
n++;
|
||||
} else {
|
||||
createLabel(addr, name, true);
|
||||
n++;
|
||||
}
|
||||
if (isExport) {
|
||||
exportAddrs.add(vaddr);
|
||||
}
|
||||
} catch (Exception ignore) {
|
||||
// duplicate/invalid symbol; skip
|
||||
}
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
private static String safeName(String name) {
|
||||
@@ -96,20 +203,4 @@ public class ExportDecompiled extends GhidraScript {
|
||||
}
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
private static String jsonEsc(String value) {
|
||||
StringBuilder sb = new StringBuilder(value.length());
|
||||
for (int i = 0; i < value.length(); i++) {
|
||||
char ch = value.charAt(i);
|
||||
switch (ch) {
|
||||
case '\\': sb.append("\\\\"); break;
|
||||
case '"': sb.append("\\\""); break;
|
||||
case '\n': sb.append("\\n"); break;
|
||||
case '\r': sb.append("\\r"); break;
|
||||
case '\t': sb.append("\\t"); break;
|
||||
default: sb.append(ch); break;
|
||||
}
|
||||
}
|
||||
return sb.toString();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Zune HD reverse-engineering corpus builder.
|
||||
|
||||
Takes the reconstructed ARM32 PE modules produced by ``disassemble_zune_hd.py``
|
||||
and, for each module:
|
||||
|
||||
1. recovers named exports/imports with rizin -> ``ghidra/symbols/<mod>.symbols.json``;
|
||||
2. imports the module into its own headless-Ghidra project, applies the recovered
|
||||
symbol names, decompiles every function, and exports
|
||||
``<mod>.functions.json`` / ``<mod>.strings.json`` / per-function ``.c`` files;
|
||||
3. records the result in ``ghidra/corpus-manifest.json``.
|
||||
|
||||
Raw output (decompiled C, symbol/string dumps) is Microsoft-derived and stays in
|
||||
this out-of-tree corpus; it is never committed. Only synthesized analysis is.
|
||||
|
||||
Usage:
|
||||
python3 scripts/ghidra_corpus.py --tier 1 # Zune-native modules
|
||||
python3 scripts/ghidra_corpus.py --modules gemstone.exe,xuidll.dll
|
||||
python3 scripts/ghidra_corpus.py --tier all --jobs 8 --resume
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
BASE_DIR = os.environ.get(
|
||||
"ZUNE_HD_DISASSEMBLY_DIR",
|
||||
os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "zune-hd-disassembly")),
|
||||
)
|
||||
MODULES_DIR = os.path.join(BASE_DIR, "modules")
|
||||
GHIDRA_DIR = os.path.join(BASE_DIR, "ghidra")
|
||||
DECOMPILED_DIR = os.path.join(GHIDRA_DIR, "decompiled")
|
||||
SYMBOLS_DIR = os.path.join(GHIDRA_DIR, "symbols")
|
||||
PROJECTS_DIR = os.path.join(GHIDRA_DIR, "projects")
|
||||
LOGS_DIR = os.path.join(GHIDRA_DIR, "logs")
|
||||
MANIFEST = os.path.join(GHIDRA_DIR, "corpus-manifest.json")
|
||||
|
||||
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
GHIDRA_SCRIPT_DIR = os.path.join(SCRIPT_DIR, "ghidra")
|
||||
ANALYZE_HEADLESS = "/home/john/.local/ghidra/support/analyzeHeadless"
|
||||
|
||||
# Zune shell + first-party services (highest signal for Dorado-HD parity).
|
||||
TIER_1 = [
|
||||
"gemstone.exe", "xuidll.dll", "zdksystem.dll", "zd3d.dll", "zrender.dll",
|
||||
"zhud_serv.dll", "zcontent_serv.dll", "zmedia_serv.dll", "znet_serv.dll",
|
||||
"zconfig_serv.dll", "zcredentials_serv.dll", "zam_serv.dll", "ziehooks.dll",
|
||||
"zie.exe", "compositor.exe", "zsplash.exe", "xnalauncher.exe", "zuncab.exe",
|
||||
"zcab.dll", "zdknet.dll", "zserial.dll", "zmassive.dll", "zlib.dll",
|
||||
"ZWmtStreamer.dll",
|
||||
]
|
||||
|
||||
|
||||
def log(msg):
|
||||
print(f"[*] {msg}", flush=True)
|
||||
|
||||
|
||||
def run(cmd, timeout):
|
||||
return subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
|
||||
|
||||
|
||||
def sha256(path):
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(1 << 20), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def extract_symbols(mod, mod_path, timeout):
|
||||
"""Return (exports, imports) lists recovered by rizin, and write the symbols file."""
|
||||
exports, imports = [], []
|
||||
try:
|
||||
res = run(["rizin", "-q", "-e", "scr.color=0", "-c", "iEj", mod_path], timeout)
|
||||
if res.returncode == 0 and res.stdout.strip():
|
||||
exports = json.loads(res.stdout)
|
||||
except Exception as ex:
|
||||
log(f" [{mod}] export scan failed: {ex}")
|
||||
try:
|
||||
res = run(["rizin", "-q", "-e", "scr.color=0", "-c", "iij", mod_path], timeout)
|
||||
if res.returncode == 0 and res.stdout.strip():
|
||||
for item in json.loads(res.stdout):
|
||||
imports.append({
|
||||
"name": item.get("name"),
|
||||
"vaddr": item.get("plt"),
|
||||
"lib": item.get("libname"),
|
||||
"type": item.get("type"),
|
||||
})
|
||||
except Exception as ex:
|
||||
log(f" [{mod}] import scan failed: {ex}")
|
||||
|
||||
clean_exports = [
|
||||
{
|
||||
"name": e.get("name") or e.get("realname"),
|
||||
"vaddr": e.get("vaddr"),
|
||||
"type": e.get("type"),
|
||||
"size": e.get("size", 0),
|
||||
"ordinal": e.get("ordinal"),
|
||||
}
|
||||
for e in exports
|
||||
if e.get("vaddr") and (e.get("name") or e.get("realname"))
|
||||
]
|
||||
|
||||
os.makedirs(SYMBOLS_DIR, exist_ok=True)
|
||||
sym_path = os.path.join(SYMBOLS_DIR, f"{mod}.symbols.json")
|
||||
with open(sym_path, "w") as f:
|
||||
json.dump({"module": mod, "exports": clean_exports, "imports": imports}, f)
|
||||
return clean_exports, imports
|
||||
|
||||
|
||||
def build_module(mod, max_functions, use_symbols, timeout):
|
||||
mod_path = os.path.join(MODULES_DIR, mod)
|
||||
if not os.path.isfile(mod_path):
|
||||
return {"module": mod, "status": "missing"}
|
||||
|
||||
started = time.time()
|
||||
exports, imports = extract_symbols(mod, mod_path, 120) if use_symbols else ([], [])
|
||||
sym_path = os.path.join(SYMBOLS_DIR, f"{mod}.symbols.json")
|
||||
|
||||
project_dir = os.path.join(PROJECTS_DIR, mod)
|
||||
os.makedirs(project_dir, exist_ok=True)
|
||||
os.makedirs(LOGS_DIR, exist_ok=True)
|
||||
os.makedirs(DECOMPILED_DIR, exist_ok=True)
|
||||
|
||||
cmd = [
|
||||
ANALYZE_HEADLESS,
|
||||
project_dir,
|
||||
"p_" + mod.replace(".", "_"),
|
||||
"-import", mod_path,
|
||||
"-overwrite",
|
||||
"-analysisTimeoutPerFile", str(timeout),
|
||||
"-scriptPath", GHIDRA_SCRIPT_DIR,
|
||||
"-postScript", "ExportDecompiled.java", DECOMPILED_DIR, str(max_functions),
|
||||
sym_path if use_symbols else "",
|
||||
"-deleteProject",
|
||||
]
|
||||
|
||||
log_path = os.path.join(LOGS_DIR, f"{mod}.log")
|
||||
status = "ok"
|
||||
returncode = 0
|
||||
try:
|
||||
res = run(cmd, timeout + 120)
|
||||
returncode = res.returncode
|
||||
with open(log_path, "w") as f:
|
||||
f.write(res.stdout or "")
|
||||
f.write("\n--- STDERR ---\n")
|
||||
f.write(res.stderr or "")
|
||||
if res.returncode != 0:
|
||||
status = "error"
|
||||
except subprocess.TimeoutExpired:
|
||||
status = "timeout"
|
||||
returncode = -1
|
||||
with open(log_path, "a") as f:
|
||||
f.write("\nTIMEOUT\n")
|
||||
except Exception as ex:
|
||||
status = "error"
|
||||
with open(log_path, "a") as f:
|
||||
f.write(f"\nEXCEPTION: {ex}\n")
|
||||
|
||||
fn_index = os.path.join(DECOMPILED_DIR, f"{mod}.functions.json")
|
||||
str_index = os.path.join(DECOMPILED_DIR, f"{mod}.strings.json")
|
||||
functions = decompiled = strings = 0
|
||||
if os.path.isfile(fn_index):
|
||||
try:
|
||||
data = json.load(open(fn_index))
|
||||
functions = len(data)
|
||||
decompiled = sum(1 for d in data if "lines" in d)
|
||||
except Exception:
|
||||
status = "error" if status == "ok" else status
|
||||
if os.path.isfile(str_index):
|
||||
try:
|
||||
strings = len(json.load(open(str_index)))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"module": mod,
|
||||
"size": os.path.getsize(mod_path),
|
||||
"sha256": sha256(mod_path),
|
||||
"status": status,
|
||||
"returncode": returncode,
|
||||
"duration_s": round(time.time() - started, 1),
|
||||
"exports": len(exports),
|
||||
"imports": len(imports),
|
||||
"functions": functions,
|
||||
"decompiled": decompiled,
|
||||
"strings": strings,
|
||||
}
|
||||
|
||||
|
||||
def select_modules(args):
|
||||
if args.modules:
|
||||
return [m.strip() for m in args.modules.split(",") if m.strip()]
|
||||
all_modules = sorted(os.listdir(MODULES_DIR))
|
||||
if args.tier == "1":
|
||||
return [m for m in TIER_1 if m in all_modules]
|
||||
if args.tier == "2":
|
||||
return [m for m in all_modules if m not in TIER_1]
|
||||
return all_modules
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Build the Zune HD RE corpus (Ghidra).")
|
||||
parser.add_argument("--tier", choices=["1", "2", "all"], default="all")
|
||||
parser.add_argument("--modules", help="Comma-separated module names (overrides --tier).")
|
||||
parser.add_argument("--jobs", type=int, default=max(1, min(8, (os.cpu_count() or 4) - 2)))
|
||||
parser.add_argument("--max-functions", type=int, default=0, help="0 = decompile every function.")
|
||||
parser.add_argument("--no-symbols", action="store_true", help="Skip rizin symbol recovery.")
|
||||
parser.add_argument("--resume", action="store_true", help="Skip modules already in the manifest as ok.")
|
||||
parser.add_argument("--timeout", type=int, default=3600, help="Per-module analysis timeout (s).")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not os.path.exists(ANALYZE_HEADLESS):
|
||||
log(f"Ghidra analyzeHeadless not found at {ANALYZE_HEADLESS}.")
|
||||
sys.exit(1)
|
||||
os.makedirs(PROJECTS_DIR, exist_ok=True)
|
||||
|
||||
modules = select_modules(args)
|
||||
existing = {}
|
||||
if os.path.isfile(MANIFEST):
|
||||
try:
|
||||
for entry in json.load(open(MANIFEST)):
|
||||
existing[entry["module"]] = entry
|
||||
except Exception:
|
||||
existing = {}
|
||||
if args.resume:
|
||||
modules = [m for m in modules if existing.get(m, {}).get("status") != "ok"]
|
||||
|
||||
log(f"Building corpus for {len(modules)} module(s) with {args.jobs} worker(s).")
|
||||
|
||||
results = dict(existing)
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=args.jobs) as pool:
|
||||
futures = {
|
||||
pool.submit(build_module, m, args.max_functions, not args.no_symbols, args.timeout): m
|
||||
for m in modules
|
||||
}
|
||||
for future in concurrent.futures.as_completed(futures):
|
||||
mod = futures[future]
|
||||
try:
|
||||
entry = future.result()
|
||||
except Exception as ex:
|
||||
entry = {"module": mod, "status": "error", "error": str(ex)}
|
||||
results[mod] = entry
|
||||
log(
|
||||
f" [{entry.get('status'):>7}] {mod:<24} "
|
||||
f"fns={entry.get('functions', 0):>6} dec={entry.get('decompiled', 0):>6} "
|
||||
f"exp={entry.get('exports', 0):>4} {entry.get('duration_s', 0)}s"
|
||||
)
|
||||
with open(MANIFEST, "w") as f:
|
||||
json.dump(sorted(results.values(), key=lambda r: r["module"]), f, indent=1)
|
||||
|
||||
ok = sum(1 for r in results.values() if r.get("status") == "ok")
|
||||
log(f"Manifest written: {MANIFEST} ({ok}/{len(results)} ok).")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user