mirror of
https://github.com/mandiant/speakeasy.git
synced 2026-10-12 02:18:47 +00:00
Willi Ballenthin
c5543a6fa1
* winenv: emulate unhooked imports from win32metadata signatures
Imports without a hand-written @apihook used to be fatal (or, with
functions_always_exist, a blind 4-argument stdcall stub). Speakeasy now
ships a signature database generated from Microsoft's win32metadata and,
when a handler is missing, emulates the call from its declaration:
- consume the right number of argument slots (64-bit scalars and
by-value
structs occupy several slots on x86) and clean up the stack according
to the calling convention
- trace the call with parameter names and decoded PSTR/PWSTR/BOOL/float
arguments, e.g.
kernel32.MoveFileExW(lpExistingFileName: "C:\old.txt", lpNewFileName:
"C:\new.txt", dwFlags: 0x1)
- return a type-appropriate success value (TRUE, S_OK/0, a fresh handle)
The data comes from the marlersoft/win32json export of win32metadata,
vendored as the deps/win32json submodule (MIT).
scripts/gen_win32_signatures.py
flattens its ~17k declarations into
speakeasy/resources/win32/signatures.json.gz
(321 KB), resolving typedefs, enums and struct layouts to a small set of
type
codes. The file is a build artifact: setup.py regenerates it for every
wheel
and sdist, `just gen-signatures` does so for a source checkout, and it
is
not committed. win32metadata records neither calling conventions nor
varargs, so scripts/win32_overrides.json supplies cdecl DLLs/functions,
variadic functions, and DLL/name aliases for forwarders such as
psapi!EnumProcesses -> kernel32!K32EnumProcesses.
Runtime support lives in speakeasy.winenv.api.sigdb as a pluggable
SignatureDatabase so further sources (e.g. undocumented ntdll natives)
can
be added later. The fallback is enabled by default via the new
modules.signature_fallback option and takes precedence over
functions_always_exist when a declaration exists; it is skipped entirely
when the database has not been generated.
Also:
- fix argc of gdi32.LineTo (1 -> 3), gdi32.MoveToEx (1 -> 4) and
kernel32.GetProcessHandleCount (1 -> 2), found by cross-checking every
hook against the metadata; tests/test_apihook_signatures.py now keeps
hooks and metadata in agreement
- GetProcAddress resolves functions known only through their signature
- ApiEvent/log_api accept a pre-rendered argument list ("name: value")
- restrict setuptools package discovery to speakeasy* (scripts/ and
tests/
were being packaged) and exclude tests/capa-testfiles from ruff
- release workflow checks out submodules so the build can regenerate the
database
* sigdb: decode enum and flag parameters symbolically
Integer parameters whose win32metadata type is an enum now carry the
enum name as a type code qualifier (u32:FILE_CREATION_DISPOSITION); the
kind, and therefore all slot arithmetic, is unchanged. The generator
emits a new top-level "enums" table holding only the ~1,000 enums some
function references, with member values masked to the enum's size so
negative members compare equal to the unsigned slots the emulator
reads. Database format bumps to 2.
At runtime, signature-emulated calls render an exact member match by
name and decompose [Flags] enums greedily, widest member first, so
composite names (FILE_GENERIC_READ) win over their constituent bits and
unaccounted bits are appended as hex:
CreateFileW(..., dwDesiredAccess: GENERIC_READ|GENERIC_WRITE,
dwShareMode: FILE_SHARE_NONE, ...,
dwCreationDisposition: CREATE_ALWAYS,
dwFlagsAndAttributes: FILE_ATTRIBUTE_NORMAL, ...)
win32metadata keeps GENERIC_READ and friends as plain constants rather
than members of FILE_ACCESS_FLAGS; a new "enum_extra" override appends
named constants to an enum so those decode too.
Claude-Session: https://claude.ai/code/session_01T713tGk7vbsvYUnD4dBnBQ
* sigdb: zero-fill Out parameters of signature-emulated calls
A call we only know the signature of reports success without producing
data, so anything the caller reads back through an Out pointer was
uninitialized stack. Out-only pointer parameters are now zeroed for as
many bytes as the prototype declares: the pointed-to scalar or struct,
a buffer sized by a sibling count/size parameter (including counts
passed by pointer), or just the terminator of an unsized string. Capped
at 64 KiB per parameter; In/Out parameters, NULL pointers, void* without
a size and arrays without a count are left alone.
To make that possible the generator now records what pointers point at
(p:u32, p:h, p:p, a:u16, ps:NAME), carries declared buffer lengths as a
4th parameter element ({"n": i} bytes / {"c": i} elements / {"k": n}
fixed), and emits a "structs" table with sizes for both pointer sizes
of every struct some parameter points at (1,898 structs).
Fixing the lookup of anonymous nested unions (_Anonymous_e__Union is
referenced with an empty Parents chain) lets every struct lay out,
which brings skipped declarations from 62 to 0. That exposed
kernel32!SetFilePointerEx: LARGE_INTEGER by value is two x86 slots, so
the hook's argc of 4 miscleaned the stack; it is now 5 with arch-aware
unpacking (the extra slot is unused and harmless on x64).
* sigdb: render pointer-to-struct arguments as JSON-like field listings
The struct table now carries every field of every struct some parameter
points at, transitively: [name, type code, offset32, offset64], with
anonymous nested types keyed OUTER.INNER and inline arrays as arr:N:CODE
(3,291 structs). Enums referenced only from struct fields join the enum
table, so structs are emitted before enums.
A new speakeasy.winenv.api.sigfmt.ArgFormatter owns all argument
rendering for signature-emulated calls, behind memory callbacks so it is
unit-tested against a plain byte buffer. In-pointers to known structs
expand to {field: value, ...} using the same decoding as parameters
(strings, BOOLs, enums/flags, GUIDs); WCHAR arrays render as text, byte
arrays as text or hex, other arrays as [..] capped at 8 items; pointers
inside structs are followed one level so OBJECT_ATTRIBUTES.ObjectName
shows its UNICODE_STRING text; LARGE_INTEGER shows as one number. Output
is capped at 1 KiB per argument and depth-bounded. By-value aggregates
(st:, GUID) render from the slots on x86 and through the reference on
x64. Reading a string that ends at an unmapped page now shrinks the read
instead of giving up.
CreateDirectoryExW(lpTemplateDirectory: "C:\tmpl", lpNewDirectory: "C:\new",
lpSecurityAttributes: {nLength: 0xc, lpSecurityDescriptor: 0x0, bInheritHandle: TRUE}) -> 0x1
* sigdb: add phnt as a second signature source for native APIs
win32metadata declares only 89 ntdll functions and none of the Ldr*,
Csr*, Dbg* or most Rtl*/Nt*/Zw* families, which is exactly what the
samples that reach the fallback tend to import. phnt (winsiderss/phnt,
MIT) is vendored as deps/phnt, pinned by commit since upstream has no
tags, and scripts/gen_phnt_signatures.py turns its prototypes into the
same table format the win32metadata generator emits.
The parser is line-oriented, relying on phnt's uniform layout
(NTSYSCALLAPI / return type / NTAPI / Name( / params / );), strips
comments and CRLF, skips FORCEINLINE bodies and callback typedefs, and
reads SAL annotations for direction (_In_/_Out_/_Inout_/_opt_/
_Reserved_) and buffer sizes (_Out_writes_bytes_(N) -> {"n": i},
_In_reads_(N) -> {"c": i}). Types resolve through the headers' own
typedefs (struct tags, *P<NAME> aliases, scalar aliases, enums) plus a
hand-written map of Windows SDK scalars and a P<NAME>/ALL_CAPS heuristic
for SDK structs phnt does not define. Enum bodies are evaluated with a
small constant-expression evaluator so information-class enums decode
(SystemInformationClass: SystemProcessInformation). Struct layouts are
not parsed from C; ps: codes resolve across every loaded source, so
OBJECT_ATTRIBUTES, UNICODE_STRING, LARGE_INTEGER and CLIENT_ID still
render from the win32metadata table.
Result: 2,520 prototypes, 2,477 supported, 87 enums, 67 KB gzipped.
PhntSource is loaded after Win32MetadataSource in the default database;
setup.py, the justfile recipe and the docs cover both generators, and
the hook argc lint now also checks the ntdll handlers (all agree).
* sigdb: always emulate unhooked imports from signatures; drop STATUS.md
Remove the modules.signature_fallback config flag: imports without a
handler that have a known signature are now always emulated from it.
functions_always_exist keeps its meaning for imports that are in neither
the handlers nor the signature databases.
Delete the STATUS.md phase tracker; the design notes live in
doc/api-handlers.md and doc/limitations.md.
* sigdb: complete type hints on the signature-emulation code and tests
Every function added by the signature work now has full parameter and
return annotations: the sigdb/sigfmt modules, the winemu fallback
methods, the two generators, the setup.py build hook and all new tests.
Callable parameters of buffer_count/out_buffer_size get the ReadUInt and
LookupStruct aliases; record_api_event/log_api take a Run and typed
argv/display.
Also clear the mypy findings the annotations surfaced in these files:
enum sizes asserted non-None in the win32 generator, list annotations in
the phnt generator, bytes|bytearray for _find_terminator, a renamed
array-count temporary in sigfmt, and Optional narrowing in the tests
(spec/loader, se.emu, lookup results via _sig/_enum/_struct helpers).
* sigdb: cleanup
41 lines
1.1 KiB
YAML
41 lines
1.1 KiB
YAML
name: Upload Python Package
|
|
|
|
on:
|
|
release:
|
|
types: [published]
|
|
|
|
permissions: {}
|
|
|
|
jobs:
|
|
deploy:
|
|
runs-on: ubuntu-latest
|
|
environment:
|
|
name: release
|
|
permissions:
|
|
contents: read
|
|
id-token: write
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
submodules: recursive
|
|
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
|
|
with:
|
|
python-version: "3.13"
|
|
- name: Install dependencies
|
|
run: |
|
|
python -m pip install --upgrade pip
|
|
pip install build
|
|
- name: Build package
|
|
run: |
|
|
python -m build
|
|
- name: Upload package artifacts
|
|
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
|
|
with:
|
|
path: dist/*
|
|
- name: Publish package
|
|
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # release/v1.13.0
|
|
with:
|
|
skip-existing: true
|
|
verbose: true
|
|
print-hash: true
|