Heretek-AI/RE-AI
1
0
Fork 0
mirror of https://github.com/Heretek-AI/RE-AI.git synced 2026-08-24 15:46:39 -04:00
Code Issues 1 Packages Projects Releases Wiki Activity
114 Commits 2 Branches 0 Tags
main
Commit Graph

10 Commits

Author SHA1 Message Date
John Smith fdba4063f7 fix(ksy): unityfs.ksy + unity_addressables.ksy file-header layout + endianness 
Post-run follow-up to the 2026-06-06-r01 stress test
(Output/2026-06-06-r01/gap-analysis.md).

The two KSY files had three classes of bugs (K1, K2 in the
gap-analysis):

- Wrong file-header layout in unityfs.ksy: parsed ONE
  unity_version strz + a phantom platform u4 +
  has_directory_info u1 + reserved u1. The actual file has
  TWO strz (bundle_format_version + unity_version) followed
  by file_size s8 — no platform, has_directory_info, or
  reserved. unityfs.ksy now matches the unity_addressables.ksy
  layout.
- Wrong endianness in both files: shipped as
  endian: le. The actual Addressables bundle on disk has
  00 00 00 07 at offset 8 (version=7 BE), not 07 00 00 00
  (version=117M LE). The 2026-06-06-r01 plan instructed
  flipping unity_addressables.ksy to LE based on
  speculation; the live file is the source of truth and it
  is big-endian. Reverted. Both files now correctly
  declare endian: be.
- compressed_block_info.uncompressed_size was s8 (8 bytes)
  but the upstream AssetStudio / UABE references declare
  it as u4 (4 bytes). The 8-byte read was walking into
  flags and num_blocks and producing nonsense values like
  279 GB for a 19 KB file. Fixed to u4 in BOTH files.

The inner bundle_header sub-type in unityfs.ksy lost the
has_directory_info param coercion (the flag is gone).
The inner directory block in both files is parsed
unconditionally. Deeper parse of the inner bundle_header
and directory blocks is a follow-up format-decode iteration.

tests/test_re_kaitai_formats.py: test_unityfs_ksy_parses_synthetic_header
docstring updated to document the deferred synthetic-buffer
rewrite (the buffer is still the pre-Cycle-2 layout; the
test soft-skips because the kaitai runtime isn't installed on
this host, so the buffer is dormant).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-06 15:56:38 -04:00
John Smith 4003eee7f4 feat(catalog): 6 bucket refactors + new publisher-internal-diagnostic-hostname detector 
Post-run follow-up to the 2026-06-06-r01 stress test
(Output/2026-06-06-r01/gap-analysis.md). The catalog changes:

- C1: split 'activation' into 'ue-component-activation'
  (Unity component-lifecycle noise) and 'license-activation'
  (the real license-gate vocabulary). ANTI-TAMPER-TAXONOMY.md
  Pattern B now references license-activation.count. Eliminates
  the 615 FPs in P3R.exe's UE component vocabulary.
- C2: split 'fingerprint' into 'custom-fingerprint' (high-signal
  HW-fingerprint literals) and 'windows-com-api-name' (standard
  COM/typelib property names). Eliminates the 48 FPs in
  P3R.exe.
- C3: 'telemetry_leak' gets exclude_keywords for asian / Asian /
  Asia / albanian / Albanian / width / Width /
  East_Asian_Width / Caucasian_Albanian / stasianwidth /
  sesasianwidth. Eliminates the 13 Unicode-UCD FPs.
- C4: 'hwid' (seeded from hwid_apis.high_signal) gets
  exclude_keywords for cl /Zi /Fd, ossl_static.pdb,
  /Fdopenssl. Eliminates the OpenSSL-static-link FPs.
- C5: 'obfuscation' gets exclude_keywords for __TBB_, tbb::,
  C:\ci\builds\, C:/ci/builds/, C:\BuildBot\,
  /ci/builds/. Eliminates the 41 TBB / CI-build FPs in
  tbb12.dll.
- C6: anti_debug_indicators.checks[].confirmation: field
  added; enum 'string_only' / 'import_only' / 'requires_disasm' /
  'requires_xref'. The 4 byte-pattern checks (RDTSC, INT 2D,
  INT 3, exception-hooking) are now 'requires_disasm'.
  Catalog has the metadata; consumer-side wiring in
  re-drm-fingerprint is deferred.

- L1: new 'publisher-internal-diagnostic-hostname' leak
  detector in servers/re-leak-scan/src/re_leak_scan/patterns.py.
  Matches internal-TLD anchor (.internal, .corp, .lan, .local,
  .intra, .private, .home.arpa) + a diagnostic-product stem
  (jenkins, jira, grafana, prometheus, kibana, splunk, sentry,
  bitbucket, gerrit, artifactory, nexus, sonarqube, vault,
  consul, etcd, datadog, newrelic, pagerduty) so public
  hostnames like jenkins.io are correctly rejected. Risk: HIGH.
  Discovered in target-B's
  pers.exe::PASystemInfoScanner.SenderInfomation (a .NET WPF
  class that does a DNS lookup of a publisher-internal .io TLD
  staging relay and conditionally sends the un-hashed machine
  fingerprint to it).

- servers/re-lief/src/re_lief/categorizers.py: added
  load_excludes() (returns {category_name: [exclude, ...]}) +
  categorize() now honors the exclude list. Backward-compatible:
  existing call sites that don't add exclude_keywords: to their
  YAML entries see no behavior change. New YAML schema fields:
  exclude_keywords: (per category, optional) and
  confirmation: (per anti_debug_indicators.checks[] entry,
  optional).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-06 15:56:15 -04:00
John Smith 1f9a626127 feat(data): extend drm-indicators catalog (C1, C2, C3) 
C1 - Sample-B encrypted-VM family pattern indicator
  New section_indicators rule: .arch / .link / .xcode / .xtext /
  .sbss section set with .rodata as encrypted body (high entropy,
  100x+ .text size). Distinct from the IL2CPP-target variant
  because there's no GameAssembly.dll + global-metadata.dat
  pairing — the proprietary engine is the target.

  New pattern_indicators mapping: "encrypted-VM bytecode,
  proprietary-engine target" with confidence: Medium-High.

  New heuristic rule: large (>100 MB) high-entropy (>7.5)
  read-only section is the encrypted body.

C2 - publisher telemetry pipeline leak attack-surface category
  New string_categories category: telemetry_leak. Catches the
  Sentry / Logstash / Confluence / Google Drive / AWS / Slack
  patterns as a single bucket. Pairs with the more detailed
  re-leak-scan MCP server.

  New pattern_indicators mapping: "publisher telemetry pipeline
  leak" with confidence: High. This is an attack-surface
  category, not a protection category.

C3 - HWID-vector API catalog extensions
  - IWmiProvider (WMI-based fingerprinting)
  - BCryptGenRandom + BCryptOpenAlgorithmProvider (Win10+ crypto)
  Promoted from medium_signal to high_signal where applicable.

last_updated: 2026-06-04 -> 2026-06-05.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-05 23:27:11 -04:00
John Smith f5e5e9e72c feat(re-lief): categorize_strings tool + catalog-driven string bucketing 
Adds a keyword-bucketed strings dump to the re-lief MCP server, turning
the manual-grep step that today lives in the LLM's head into a
catalog-driven, deterministic lookup. Superset of extract_strings
(same {ascii, utf16le, totals, truncated} shape for backward compat)
plus a by_category block with 11 semantic categories (anti_debug,
hwid, crypto, network, registry, process, file, fingerprint,
activation, obfuscation, misc).

The categorization vocabulary lives in a new
data/drm-indicators.yaml::string_categories section. Two seed
categories (anti_debug, hwid) inherit their keyword lists from
existing catalog sections via a seed_from / seed_field YAML pointer
— when a future agent adds a new HWID API to hwid_apis.high_signal,
the categorizer picks it up on next MCP-server reload with zero
Python change. The YAML is the single source of truth for both the
indicator set that re-drm-fingerprint reads and the keyword set
that the categorizer reads.

Five skills (re-static-triage, re-malware-triage, re-drm-fingerprint,
re-vm-reverse, re-format-decode) had their manual-grep step replaced
with a call to re-lief.categorize_strings. No new workflow steps
were added — the categorizer IS the string scan.

ANTI-TAMPER-TAXONOMY.md gains a "Recognizing the patterns in
arbitrary binaries" section that documents Pattern A (encrypted-VM
bytecode interpreter: 7 section-name co-occurrence + W^X .idata +
.text virt>>raw + .ecode lazy-decrypt stub + vendor-tagged PDB +
late-bound export tail + 8+ HWID APIs) and Pattern B
(hardware-fingerprinting routine in a third-party launcher
activation library: ordinal-only exports + WinHTTP + OpenSSL +
HWID-vector APIs + split anti-debug surface) in vendor-neutral
category terms. No vendor / publisher / game / PDB-path literals
appear in any shipped file.

Tests: 7 new soft-skip tests in test_re_lief_categorize_strings.py
covering the result shape, the seed_from inheritance, the bundled
Activation64.dll high-signal hits, the legacy extract_strings
wrapper, and the GameAssembly full-section vs skip_sections paths.
All always-on tests (leakage, frontmatter, server registration,
smoke) continue to pass. ./verify.sh is green.
 2026-06-05 16:01:49 -04:00
John Smith 5caa6b8def chore: vendor-neutral sanitization + leakage test + taxonomy doc 
Sweep the working tree to remove commercial anti-tamper / DRM product
names, publisher names, and game titles from every shipped file, and
add the regression test + taxonomy doc that lock the policy in.

- ANTI-TAMPER-TAXONOMY.md (new): documents the five vendor-neutral
  categories (encrypted-VM bytecode interpreter, MBA-obfuscated
  arithmetic, legacy disc-based protection, hardware-fingerprinting
  routine, anti-debug / anti-emulation), the inference chain from a
  binary's observable features to a category, and the negative space
  (what RE-AI explicitly will not do).
- tests/test_no_vendor_leakage.py (new): greps every shipped file for
  banned vendor / publisher / game / PDB-tag strings; parametrized
  per-needle so failures point at the exact term to fix. Excludes
  data/ksy/ (byte-pattern specs), data/capa-rules/ (vendored upstream
  rule bundle), the gitignored Input/Output/docs/.venv*, and self
  (CLAUDE.md + this test, which must quote the banned strings to
  communicate the rule).
- data/drm-indicators.yaml: rewrite the header to attribute sources
  to the public literature on the categories, not to specific
  commercial products; add the vendor-neutral-by-policy paragraph.
- skills/re-drm-fingerprint, re-vm-reverse, re-mba-deobfuscate,
  re-il2cpp-decompile, re-il2cpp-static-triage: replace inline
  vendor / publisher / game names with category descriptors and
  generic file roles ("the IL2CPP game binary", "the publisher's
  Assembly-CSharp.dll").
- servers/re-il2cpp/src/re_il2cpp/server.py + README: replace example
  FQNs (e.g. publisher.Player.TakeDamage) with generic placeholders
  (MyGame.PlayerController.TakeDamage).
- servers/re-il2cpp/pyproject.toml, servers/re-triton/pyproject.toml,
  servers/re-triton/src/re_triton/{server,symbolic}.py: structured-error
  defenses + dep-string corrections surfaced by the same sweep
  (Quarkslab Triton vs PyPI triton GPU compiler collision).
- servers/re-gdb/src/re_gdb/gdb_mi.py, re-kaitai/src/re_kaitai/kaitai_runner.py:
  matching code-side cleanups.
- install.sh: capa.sig deployment into all three venvs; capa-rules
  clone; kaitai-struct-compiler GitHub fallback; Z3/Capstone CMake
  env-var workaround for Fedora.
- README, CHANGELOG: scrub product / publisher / game names; reword
  v2.0.0 entry to match the new vocabulary.
- tests/test_re_il2cpp_check.py, test_re_lief_imports.py,
  test_re_triton_smoke.py: matching sanitization + new coverage.
- .gitignore: add CLAUDE.md (per-user guidance file) and the two
  installer-generated dirs data/capa-rules/ and data/ksy/_compiled/
  so neither bloats the tree nor races with install.sh.

Verified: pytest tests/test_no_vendor_leakage.py
tests/test_skills_frontmatter.py tests/test_plugin_manifest.py
tests/test_smoke.py — 26 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-05 11:08:39 -04:00
John Smith 8f66836e7a release: v2.3.0 - post-smoke-test fixes + kaitai 0.10 + unityfs directory 
- unityfs.ksy: endian be -> le (real UnityFS is LE)
- unityfs.ksy: add block, directory, directory_entry types;
  parse per-block records and the directory block (v0.2 of spec)
- tests/test_re_kaitai_formats.py: synthetic LE buffer covers
  full file -> bundle_header -> compressed_block_info -> blocks ->
  directory -> entries chain; new unity_raw.ksy parse test
- pyproject.toml + .claude-plugin/plugin.json: bump to 2.3.0
- CHANGELOG.md: replace [Unreleased] with [2.3.0] - 2026-06-04
  and document the new tools, the kaitai 0.10 fixes, the
  install.sh extras, and the test coverage additions

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 19:40:12 -04:00
John Smith b1832b50bf fix(kaitai): unityfs.ksy bundle_header generic-type + magic NUL escape 
Address the 'deeper kaitai 0.10 compatibility (generic types in
bundle_header require a different syntax)' issue that the previous
big-bang plan explicitly deferred.

Two changes to data/ksy/unityfs.ksy:

1. The bundle_header call site (seq/6) was passing a u1 (the
   has_directory_info field) into a parameter declared as
   'type: bool' (CalcBooleanType), which kaitai 0.10 rejects with
   'cannot pass argument #0 of type Int1Type(false) into parameter
   p_has_directory_info of type CalcBooleanType'. The kaitai 0.10
   call-site syntax for passing a single typed argument is
   positional, so the fix is to change the call to
   'bundle_header(has_directory_info != 0)'. The generated Python
   now correctly constructs Unityfs.BundleHeader with the
   coerced boolean.

2. The magic field's contents string was being read literally
   as 11 bytes (backslash-x-0-0 plus the 7 ASCII chars) instead
   of 8 bytes including a NUL. Switched to a YAML double-quoted
   string with the canonical \0 escape so the generated Python
   reads 8 bytes. Fixes the runtime 'validation failed: not
   equal, expected b"UnityFS\x00"' error on every real
   UnityFS file.

Test changes (tests/test_re_kaitai_formats.py):

- Replace the old encoding-substring check with
  test_unityfs_ksy_compiles that actually runs
  kaitai-struct-compiler and asserts returncode 0.

- Add test_unityfs_ksy_parses_synthetic_header that builds a
  minimal valid UnityFS file in memory (BE-encoded to match the
  spec's 'endian: be'), compiles the .ksy, loads the generated
  Python module, parses the file, and asserts the recovered
  fields (version, unity_version, platform, has_directory_info,
  bundle_header.unity_revision, bundle_header.size,
  bundle_header.compressed_block_info.blocks_count) are correct.

Verification:
- pytest tests/ -> 36 passed, 10 skipped
- unityfs.ksy now compiles cleanly against kaitai 0.10
- Synthetic UnityFS file parses end-to-end and recovers the
  expected header fields

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 19:25:27 -04:00
John Smith 1d72681a92 fix(install/spec/api): address bugs surfaced by v2.x smoke test 
P0 — install (unblocks every other tool)
- install.sh: add post-install steps for LIEF, re-il2cpp[rva] extra,
  Quarkslab Triton source install, capa-rules clone, capa.sig into
  both venv site-packages, and ksy compilation
- servers/re-il2cpp/README.md: add top-level Install block
- servers/re-triton/pyproject.toml: switch pin from triton>=0.6
  (which resolved to the GPU compiler) to
  triton @ git+https://github.com/JonathanSalwan/Triton
- servers/re-triton/README.md: fix install instructions

P1 — bug fixes (code is broken, P0 install won't help)
- data/ksy/unityfs.ksy: add explicit encoding: UTF-8 to both strz
  fields, drop redundant size-eos, use '0.10' string ks-version,
  remove broken compression_flags enum
- servers/re-capa/capa_runner.py + server.py: add rules= param to
  extract_mbc and find_interesting (was silently dropped)
- servers/re-lief/parsers.py: fix get_imports_exports imports loop
  to walk imp.entries (was one level too shallow — every function
  name came back blank)
- servers/re-kaitai/kaitai_runner.py: replace hard-coded
  list_known_formats tuple with directory glob (tuple drifted out
  of sync with kaitaistruct)

P2 — new features
- data/ksy/unity_raw.ksy: new spec for the Unity raw asset-bundle
  format (the level/sharedassets files actually use this, not
  UnityFS)
- servers/re-il2cpp/tables.py: add get_assembly_types(path,
  image_name) that walks the typeDef range owned by a single
  IL2CPP image
- servers/re-il2cpp/server.py: expose get_assembly_types as an MCP
  tool
- servers/re-il2cpp/metadata.py: re-export get_assembly_types
- skills/re-il2cpp-static-triage/SKILL.md: new condensed 30-second
  triage skill, focused on the recoverable class graph and
  explicitly NOT promising to read function bodies

P3 — test coverage (regression guard for P0/P1 fixes)
- tests/test_re_capa_rules.py: rules-arg forwarding (3 tests)
- tests/test_re_triton_smoke.py: triton pin + emulate smoke
- tests/test_re_lief_imports.py: import-name recovery + encrypted-VM bytecode interpreter
  HWID import visibility
- tests/test_re_kaitai_formats.py: list_known_formats + encoding
  fix verification + new unity_raw.ksy compiles
- tests/test_servers_import.py: extend re-lief skip to also
  tolerate missing capstone (optional dep)

Verification
- pytest tests/ → 36 passed, 9 skipped
- All 12 existing re-il2cpp tests still pass
- New get_assembly_types returns all 2,697 Assembly-CSharp types
- unityfs.ksy and unity_raw.ksy now compile against kaitai 0.10

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 19:11:33 -04:00
John Smith 83fe9aa353 feat(data): ship unityfs.ksy starter + extend re-format-decode 
A 130-line starter Kaitai Struct spec for the UnityFS asset bundle
format, plus four new rows in the re-format-decode skill's
'Common file types' magic-bytes table.

The .ksy covers the file header (magic + version + unity_version
string + platform + has_directory_info) and the bundle header
(signature + size + compressed-block info) up to the boundary
where LZ4/LZMA decompression becomes necessary. Per the
re-format-decode skill's 'Iteration 2..N' loop, the user extends
from here: add a block-bytes repeat, plug in a decompression
library, parse the directory entries, and finally the assets.

Format reverse-engineered from SeriousCache/UABE and the
AssetStudio wiki. The 'blocks' sub-record is intentionally
commented out as the natural next iteration; the .ksy compiles
and parses the header so the user has a working starting point.

Magic-bytes table additions:
  AF 1B B1 FA          -> IL2CPP global-metadata.dat (use re-il2cpp-decompile)
  55 6E 69 74 79 46 53 00  -> UnityFS (modern asset bundle)
  55 6E 69 74 79 57 65 62  -> UnityWeb (legacy)
  55 6E 69 74 79 52 61 77  -> UnityRaw (legacy)

The mitmproxy addendum is already part of the re-il2cpp-decompile
skill (added in the previous commit) as a 'Network capture'
section that ties Unity game analysis to re-mitm2swagger.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 14:29:36 -04:00
John Smith 355b33651a feat: add re-vm-reverse, re-mba-deobfuscate, re-drm-fingerprint + drm-indicators.yaml 
Three new skills and one companion data file, derived from the
analysis of https://connorjaydunn.github.io/blog/posts/encrypted-VM bytecode interpreter-analysis/
and Connor Dunn's broader DRM-reverse-engineering methodology.

Skills (all use existing MCP servers, no new infrastructure):

- re-vm-reverse: custom VM bytecode analysis. Workflow combines
  re-lief section triage, re-rizin dispatcher pattern matching,
  re-gdb frequency analysis, and re-llm-decompile handler lifting.
  Applies to encrypted-VM bytecode interpreter, encrypted-VM bytecode interpreter, encrypted-VM bytecode interpreter, encrypted-VM bytecode interpreter.

- re-mba-deobfuscate: Mixed-Boolean-Arithmetic simplification via
  Triton symbolic execution and Z3 equivalence queries. Workflow
  is LLM-driven (reads disassembly, translates to Python exprs) and
  Z3-verified (proves equivalence to a candidate simpler form).

- re-drm-fingerprint: hardware-fingerprinting routine detection.
  Combines section, import, string, and anti-debug static signals
  to produce a confidence score (Low/Medium/High) and a vendor guess.

Companion data:

- data/drm-indicators.yaml: 384 lines covering KUSER_SHARED_DATA
  offsets, PEB fields, HWID-vector API catalog, section-name
  heuristics, VM dispatcher patterns, MBA identity catalog,
  anti-debug catalog, and vendor fingerprints. LLM-readable prose
  where the LLM needs to query it.

Decision recorded in CHANGELOG: sogen is NOT added as an MCP
server (GPL-2.0 license + setup cost). encrypted-VM bytecode interpreter-specific skill NOT
added (vendor-specific framing goes stale; the three generic skills
above are the right packaging).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 11:53:39 -04:00