mirror of
https://github.com/qilingframework/qiling.git
synced 2026-10-12 02:21:40 +00:00
Rebuilt ARCHITECTURE.md and ARCHITECTURE/ with eatmycode 2.0.0
This commit is contained in:
xwings
+73
-531
@@ -1,541 +1,83 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Qiling Framework — Architecture
|
||||
# Qiling Architecture
|
||||
|
||||
This is the control center for agent-readable architecture docs.
|
||||
Cross-cutting facts live here; each subsystem is documented once in
|
||||
`ARCHITECTURE/<module>.md` (see the [Index](#index)). `CLAUDE.md` and
|
||||
`AGENT.md` are symlinks to this file.
|
||||
## Read First
|
||||
|
||||
## Mission and Constraints
|
||||
Before planning code changes or reviewing code, read
|
||||
[Agent Rules](ARCHITECTURE/AGENT_RULES.md). Follow the Task Index to the
|
||||
owning module and read pages whose **Read when** trigger matches the task.
|
||||
Load partner modules only for affected boundaries; never load the entire
|
||||
ARCHITECTURE directory. Reuse unchanged pages already read in this session.
|
||||
Check claims against source, configuration, and tests; they remain
|
||||
authoritative. If a route or fact is missing or stale, inspect source and
|
||||
repair the affected docs. For broad changes, work through owners in batches
|
||||
and retain cross-owner constraints and verification evidence.
|
||||
|
||||
Qiling is a binary emulation framework: it emulates and sandboxes code in
|
||||
an isolated environment across platforms and architectures. Built on
|
||||
Unicorn Engine, it adds what raw CPU emulation lacks: operating-system
|
||||
context (syscalls, APIs, filesystems, registries), executable-format
|
||||
loading, and dynamic linking (`README.md:13-27`).
|
||||
## Project Snapshot
|
||||
|
||||
Supported combinations are defined in code:
|
||||
|
||||
- Architectures — `QL_ARCH` (`qiling/const.py:15`): 8086, x86, x86-64,
|
||||
ARM, ARM64, Cortex-M, MIPS, RISC-V 32/64, PowerPC.
|
||||
- Operating systems — `QL_OS` (`qiling/const.py:28`): Linux, FreeBSD,
|
||||
macOS, Windows, UEFI, DOS, QNX, MCU (bare-metal), BLOB.
|
||||
- Formats: ELF, PE, Mach-O, COM/EXE/MBR, Intel HEX / raw firmware; kernel
|
||||
modules for Windows `.sys`, Linux `.ko`, macOS `.kext`.
|
||||
|
||||
Observable behavior: `Qiling(argv, rootfs, …).run()` executes the target
|
||||
with hooks at instruction, basic-block, memory, interrupt, syscall, and
|
||||
API level; VM state save/restore; hot patching; GDB-server and built-in
|
||||
Qdb debugging (with record/replay); AFL++ fuzzing; and an opt-in kernel
|
||||
proxy that forwards chosen Linux syscalls to a real kernel.
|
||||
|
||||
Constraints and non-goals:
|
||||
|
||||
- Emulation is single-Unicorn, cooperative-thread; there is no preemptive
|
||||
threading, no signal delivery, and networking uses host sockets
|
||||
(`TODO.md:9-21`). Real threading and signals are proposals only.
|
||||
- Windows and macOS emulation need host-collected, non-redistributable
|
||||
system libraries; those suites are host-gated.
|
||||
- Coverage of syscalls, Win32 APIs, and peripheral registers is
|
||||
demand-driven; unrequested surface is out of scope (see deviations).
|
||||
- `unicorn` is hard-pinned; upgrading it is a project-wide event.
|
||||
|
||||
## Languages and Toolchain
|
||||
|
||||
| Area | Language / tool | Evidence |
|
||||
| ---- | --------------- | -------- |
|
||||
| `qiling/`, `tests/`, `examples/*.py`, `qltool`, `qltui.py` | Python, declared `^3.10` | `pyproject.toml:37` |
|
||||
| CI matrix | Python 3.11 and 3.13 on `windows-latest` and `ubuntu-latest` (four jobs); the ubuntu/3.13 job carries a `container: Docker` marker | `.github/workflows/build-ci.yml:13-18` |
|
||||
| Packaging | Poetry 2 (`poetry-core>=2.0,<3.0`), version `1.4.12.dev0`, GPL-2.0-or-later; `qltool` console script | `pyproject.toml:4`, `:7`, `:33-34`, `:61-63` |
|
||||
| Wheel/sdist build check | `poetry check --lock`, `python -m build`, `twine check --strict` | `.github/workflows/pythonpublish.yml:20-44` |
|
||||
| Container | `python:3.13-slim-trixie` multi-stage Poetry wheel build | `Dockerfile:1`, `:18-23` |
|
||||
| Editor config | 4-space indent, LF, UTF-8, final newline for `*.py` | `.editorconfig:6-12` |
|
||||
| Fixture sources | C/asm under `examples/src/`, `examples/shellcodes/`, `examples/fuzzing/*/fuzz.c` (built out-of-tree; binaries live in the rootfs submodule) | `examples/src/linux/hello.c` |
|
||||
| Test scripts | Bash (`tests/test_onlinux.sh`, `tests/test_macho.sh`), batch (`tests/test_pe.bat`) | `tests/test_onlinux.sh:7-20` |
|
||||
|
||||
Runtime dependencies (`pyproject.toml:36-50`): `unicorn == 2.1.3`
|
||||
(hard-pinned), `capstone ^5`, `keystone-engine ^0.9.2`, `pefile`,
|
||||
`pyelftools`, `python-registry`, `gevent >=24.10`, `multiprocess`,
|
||||
`pyyaml ^6`, `windows-curses` (Windows only), and the TUI trio
|
||||
`python-fx`/`questionary`/`termcolor`. Extras: `fuzz` → `unicornafl`,
|
||||
`fuzzercorn`; `RE` → `r2libr` (`pyproject.toml:57-59`).
|
||||
|
||||
No formatter, linter, or type checker is configured in the tree
|
||||
(inspected: `pyproject.toml`, no `setup.cfg`/`tox.ini`/`.flake8`/
|
||||
`.pre-commit-config.yaml`). Declared support is the `^3.10` range above;
|
||||
the versions this refresh was verified with (Python 3.13.5, Poetry 2.1.2,
|
||||
unicorn 2.1.3) are a local observation, not a repository constraint.
|
||||
| Fact | Value and evidence |
|
||||
| --- | --- |
|
||||
| Purpose | Binary emulation over Unicorn with images, OS APIs, hooks and firmware; entry is `Qiling(...).run()` ([core](qiling/core.py)). Complete OS/device fidelity is outside implemented scope. |
|
||||
| Language / package | Python `^3.10`; package `1.4.12.dev0`, GPL-2.0-or-later; Poetry backend `poetry-core>=2.0,<3.0` ([manifest](pyproject.toml)). Fixture C/C++/assembly toolchains are local. |
|
||||
| Engines | Unicorn **2.1.3**, Capstone `^5`, Keystone `^0.9.2`; other dependencies/platform extras in the manifest and [lock](poetry.lock). |
|
||||
| Compatibility | Arch/OS enums in [const.py](qiling/const.py), composition in [utils.py](qiling/utils.py); enums do not promise every combination. [CI](.github/workflows/build-ci.yml) declares Python 3.11/3.13 on Ubuntu/Windows; macOS job is commented out. |
|
||||
|
||||
## System Design
|
||||
|
||||
Layers, bottom to top by import direction; a layer imports only layers
|
||||
below it (plus the `Qiling` facade type from `qiling/core.py`, which every
|
||||
module imports for annotations):
|
||||
|
||||
| Layer | Owner doc | Responsibility and state |
|
||||
| ----- | --------- | ------------------------ |
|
||||
| Arch | [arch.md](ARCHITECTURE/arch.md) | The single Unicorn instance, registers, stack, disassembler/assembler, calling conventions |
|
||||
| Core | [core.md](ARCHITECTURE/core.md) | `Qiling` facade, composition order, hook engine, component factories, profiles, logging, exceptions |
|
||||
| OS base | [os-base.md](ARCHITECTURE/os-base.md) | Memory manager/heap, fcall marshalling, rootfs-confined paths, fs mapper, fd objects, green-thread base |
|
||||
| OS personalities | [os-posix.md](ARCHITECTURE/os-posix.md), [os-windows.md](ARCHITECTURE/os-windows.md), [os-baremetal.md](ARCHITECTURE/os-baremetal.md) | Syscall/API/interrupt dispatch and run loops; own guest process state |
|
||||
| Loader | [loader.md](ARCHITECTURE/loader.md) | Parses the untrusted image, maps it, builds initial state, records image tables; imports the OS personality's structs, hooks, and API tables (`qiling/loader/elf.py:24-27`, `pe.py:22-26`) |
|
||||
| Hardware | [hw.md](ARCHITECTURE/hw.md) | MMIO peripherals and chip maps for MCU mode |
|
||||
| Optional | [debugger.md](ARCHITECTURE/debugger.md), [extensions.md](ARCHITECTURE/extensions.md), [kernel-proxy.md](ARCHITECTURE/kernel-proxy.md), [cli.md](ARCHITECTURE/cli.md) | Consume only the public `Qiling`/`QlOs` API |
|
||||
|
||||
Rules a change must respect:
|
||||
|
||||
- **Name-based composition.** Core resolves arch/loader/OS/component/
|
||||
debugger classes by deriving module and class names from enum names in
|
||||
`qiling/utils.py:297-417`; concrete classes are never imported by core.
|
||||
- **Downward imports only.** Documented exceptions: `qiling/arch/cortex_m.py:22`
|
||||
and `qiling/os/mcu/mcu.py:11` import `qiling/extensions/multitask.py`,
|
||||
`qiling/arch/utils.py:94` lazily imports r2, `qiling/arch/x86_utils.py:10`
|
||||
imports `QlMemoryManager` from OS base for a type annotation, and
|
||||
`qiling/cli.py:22-23` imports coverage/report. Do not add more. Optional
|
||||
modules mostly use the public API but also reach into concrete types
|
||||
(`qiling/debugger/gdb/gdb.py:38`, `qiling/extensions/tracing/formats/registers.py:2`);
|
||||
that is downward and allowed.
|
||||
- **Dispatch models.** POSIX intercepts syscalls by number
|
||||
(`qiling/os/posix/posix.py:170`); Windows/UEFI intercept API calls by
|
||||
IAT address (`qiling/os/windows/windows.py:172`); DOS intercepts
|
||||
interrupts by `(intno, AH)` (`qiling/os/dos/dos.py:83`); MCU steps
|
||||
instruction-by-instruction with peripherals (`qiling/os/mcu/mcu.py:53`).
|
||||
- **User override protocol.** `set_syscall`/`set_api` with
|
||||
`QL_INTERCEPT.CALL|ENTER|EXIT` (`qiling/const.py:55`) is the only
|
||||
supported way to replace or wrap emulated behavior; the kernel proxy is
|
||||
built entirely on it.
|
||||
- **Trust boundaries.** The guest is untrusted. Host exposure exists at:
|
||||
rootfs path resolution (`qiling/os/path.py:239`), loader parsing of
|
||||
header-derived sizes (`qiling/loader/`), host sockets and `os.fork` in
|
||||
POSIX (`qiling/os/posix/syscall/sched.py:50-59`), and explicitly
|
||||
forwarded syscalls in the kernel proxy.
|
||||
- **Fidelity over abstraction.** Per-OS and per-chip code is deliberately
|
||||
repetitive to mirror real platform behavior (see deviations).
|
||||
|
||||
## Runtime and Data Flow
|
||||
|
||||
1. Entry: `qltool` (checkout) or the installed console script call
|
||||
`qiling.cli.run`, which ends in `Qiling(**ql_args)` (`qiling/cli.py:276`);
|
||||
library users construct `Qiling` directly.
|
||||
2. `Qiling.__init__` (`qiling/core.py:36`) guesses arch/OS from the file if
|
||||
not given (`qiling/utils.py:278`), then composes in fixed order: arch →
|
||||
struct/hook mixins → logger → profile → loader → memory manager → OS →
|
||||
hardware manager (bare-metal only) → `loader.run()` → stop guard
|
||||
(`qiling/core.py:154-197`). The target is fully mapped when the
|
||||
constructor returns.
|
||||
3. `Qiling.run()` (`qiling/core.py:561`) instantiates the debugger if set,
|
||||
applies queued patches, writes the exit trap, and delegates to
|
||||
`os.run()`; the debugger's `run()` follows.
|
||||
4. The OS run loop drives `Qiling.emu_start` (`qiling/core.py:743`), the
|
||||
thin wrapper over `uc.emu_start` that manages the thumb bit, `QL_STATE`,
|
||||
and re-raises exceptions captured inside hooks.
|
||||
5. Guest traps re-enter Python through `QlCoreHooks` dispatchers
|
||||
(`qiling/core_hooks.py:167-276`) into the OS layer's syscall/API
|
||||
handler, which reads arguments via the arch/ABI, executes, logs through
|
||||
`QlOsUtils.print_function`, records stats, and writes the return value.
|
||||
6. Exit: the guest reaches the OS exit point or exit trap, `emu_stop` is
|
||||
called, `run()` returns; `ql.os.exit_code` carries the status. Unicorn
|
||||
errors surface as `UcError` after `QlOs.emu_error` dumps context
|
||||
(`qiling/os/os.py:249`).
|
||||
|
||||
Configuration contract: INI profiles per OS in `qiling/profiles/<os>.ql`
|
||||
(sections such as `[OS32]/[OS64]`, `[CODE]`, `[KERNEL]`, `[MISC]`,
|
||||
`[NETWORK]`; Windows adds `[PATH]`, `[USER]`, `[REGISTRY]`, …) merged with
|
||||
a user path or dict (`qiling/utils.py:419-449`); MCU uses YAML merged into
|
||||
`ql.env`. The `env` kwarg is the guest environment for POSIX/Windows and
|
||||
the chip map for MCU.
|
||||
|
||||
Persistence: none by default. `Qiling.save/restore` snapshot per-component
|
||||
state (optional pickle file); the Windows registry writes hives back at
|
||||
run end; `QlPeCache` caches parsed DLLs when `libcache=True`.
|
||||
|
||||
Concurrency: one Unicorn per `Qiling`; POSIX/Windows threads are gevent
|
||||
greenlets; MCU fast mode uses the cooperative `MultiTaskUnicorn`; `clone`
|
||||
without `CLONE_VM` forks the host process; the kernel proxy is a child
|
||||
process over a Unix socketpair.
|
||||
|
||||
## Workspace Map
|
||||
|
||||
| Path | Holds |
|
||||
| ---- | ----- |
|
||||
| `qiling/` | The framework package (owners in the Index) |
|
||||
| `qiling/profiles/*.ql` | Default per-OS INI profiles |
|
||||
| `qiling/os/posix/kernel_proxy/` | Kernel proxy (Linux-host optional feature) |
|
||||
| `qiling/extensions/mcu/` | Chip `env` maps (owned by [hw.md](ARCHITECTURE/hw.md)) |
|
||||
| `tests/` | Standalone `unittest` files run from `tests/`; CI drivers `test_onlinux.sh`, `test_pe.bat`, `test_macho.sh`; Qdb scripts in `qdb_scripts/`; test-only INI profiles in `profiles/`; scratch output in `log_test/` |
|
||||
| `examples/` | Demo scripts, `fuzzing/`, `mcu/`, `shellcodes/`, `extensions/`, `scripts/` (DLL/dylib collectors), `src/` (fixture sources), and the `rootfs/` submodule (fixture binaries; do not edit in place, update the submodule) |
|
||||
| `jexamples/` | Legacy examples, not covered by CI |
|
||||
| `docs/` | One-line pointers to https://docs.qiling.io, images, and two Windows DLL inventory lists (`DLLX86.txt`, `DLLX8664.txt`); not a documentation source |
|
||||
| `qltool`, `qiling/cli.py`, `qltui.py` | CLI launcher, implementation, TUI ([cli.md](ARCHITECTURE/cli.md)) |
|
||||
| `pyproject.toml`, `poetry.lock` | The only manifest and lock file |
|
||||
| `Dockerfile` | Container build of the wheel (build tooling; no runtime code) |
|
||||
| `.github/workflows/` | `build-ci.yml` (tests), `pythonpublish.yml` (build checks + publish), `dockerimage.yml`, `giteesync.yml` |
|
||||
| `TODO.md` | Hybrid-kernel design and the "Existing Issues" list; `ChangeLog` (stale at 1.4.6); `CREDITS.md`; `COPYING` |
|
||||
|
||||
Generated or vendored: `examples/rootfs/` (git submodule, regenerate with
|
||||
`git submodule update --init`); `qiling/os/uefi/Uefi*.py`,
|
||||
`PiMultiPhase.py`, `ProcessorBind.py` are EDK2-derived type tables (edit
|
||||
with care, no generator in tree); `qiling/debugger/gdb/xml/` mirrors GDB's
|
||||
target descriptions.
|
||||
|
||||
## Coding Style and Code Design
|
||||
|
||||
No enforced formatter/linter/type checker exists; the rules below are
|
||||
observed conventions with canonical implementations.
|
||||
|
||||
- **Indentation**: 4 spaces (`.editorconfig:9-10`); every package file
|
||||
indents with spaces except two that still contain tab-indented lines
|
||||
(`qiling/os/posix/syscall/epoll.py:140-141`,
|
||||
`qiling/debugger/qdb/branch_predictor/__init__.py:13-17`); do not add
|
||||
more.
|
||||
- **File header**: shebang plus the three-line framework banner
|
||||
(`qiling/core.py:1-4`).
|
||||
- **Typing**: public APIs are annotated; circular imports are avoided with
|
||||
`from __future__ import annotations` and `TYPE_CHECKING` guards
|
||||
(`qiling/log.py:6`, `:19-20`; used in 42/49 modules respectively).
|
||||
Callback signatures are `Protocol`s (`qiling/core_hooks.py:55-134`).
|
||||
- **Naming**: classes `Ql<Thing>`; enums `QL_*` (`qiling/const.py`);
|
||||
syscall handlers `ql_syscall_<name>` (`qiling/os/posix/posix.py:19`);
|
||||
Win32 hooks `hook_<Name>` with `@winsdkapi`
|
||||
(`qiling/os/windows/dlls/kernel32/fibersapi.py:13-16`); UEFI `@dxeapi`;
|
||||
peripherals `<Chip><Periph>` with an inner `Type` struct
|
||||
(`qiling/hw/char/stm32f4xx_usart.py:12`).
|
||||
- **Errors**: raise `QlErrorBase` subclasses from `qiling/exception.py`;
|
||||
unknown syscalls/APIs log a warning and raise only under `debug_stop`
|
||||
(`qiling/os/posix/posix.py:255-258`, `qiling/os/windows/windows.py:196-199`).
|
||||
`TODO.md:628-644` lists bare `except:` sites and `assert`-based
|
||||
validation as known debt (18 bare `except:` remained at this refresh);
|
||||
do not add new ones.
|
||||
- **Logging**: always `ql.log`; `print` is reserved for Qdb/TUI/IDA and
|
||||
guest console output. Stray diagnostic prints remain in
|
||||
`qiling/extensions/multitask.py:328`, `:365`, `qiling/exception.py:90`,
|
||||
`qiling/loader/macho.py:313`, `qiling/os/macos/kernel_api/kernel_api.py:1397`,
|
||||
and `qiling/extensions/tracing/formats/tenet.py:64`; do not add more.
|
||||
Syscall/API lines go through `QlOsUtils.print_function`
|
||||
(`qiling/os/utils.py:106`).
|
||||
- **Guest structs**: `ctypes` via `qiling/os/struct.py` factories
|
||||
(`get_packed_struct`/`get_aligned_struct`) so endian/pointer width follow
|
||||
the target (`qiling/os/posix/syscall/epoll.py:31-45`).
|
||||
- **Component resolution**: never import concrete arch/OS/loader classes
|
||||
in core; rely on the `select_*` factories (`qiling/utils.py:297-417`).
|
||||
- **Tests**: one standalone `unittest` module per subsystem in `tests/`,
|
||||
run from that directory with relative rootfs paths
|
||||
(`tests/test_elf.py:146`); host-gated cases use `unittest.skipUnless`
|
||||
(`tests/test_kernel_proxy.py:17`, `tests/test_pathutils.py:31`).
|
||||
- **Docstrings**: Google-style `Args:`/`Returns:` on public methods
|
||||
(`qiling/core.py:561-570`); comments explain workarounds with issue links
|
||||
(`qiling/os/os.py:215-221`).
|
||||
|
||||
## Verification and Review Map
|
||||
|
||||
Setup (Linux, from a clone with the submodule):
|
||||
|
||||
```sh
|
||||
git submodule update --init # examples/rootfs fixtures
|
||||
python3 -m pip install -e . # or: poetry install
|
||||
cd examples/rootfs/x86_linux/kernel && unzip -P infected m0hamed_rootkit.ko.zip # only for test_elf_ko.py
|
||||
```
|
||||
|
||||
All test commands run from `tests/`; a pass is unittest `OK` and exit 0.
|
||||
Run suites one at a time: several bind fixed localhost ports
|
||||
(`tests/test_elf.py:926`, `tests/test_tendaac15_httpd.py:98`,
|
||||
`tests/test_debugger.py:112`); during this refresh a `test_posix.py` run
|
||||
started alongside `test_elf.py` blocked in `epoll_wait` until killed.
|
||||
|
||||
| Change area | Run | Owner doc |
|
||||
| ----------- | --- | --------- |
|
||||
| Core facade, hooks, factories | `python3 test_shellcode.py` | [core.md](ARCHITECTURE/core.md) |
|
||||
| Arch/registers/CPU models | `python3 test_cpu_models.py`; `python3 test_riscv.py` | [arch.md](ARCHITECTURE/arch.md) |
|
||||
| Loaders | `python3 -m unittest test_elf.ELFTest.test_elf_linux_x8664`; `python3 test_uefi.py`; `python3 test_dos.py`; `python3 test_mcu.py` | [loader.md](ARCHITECTURE/loader.md) |
|
||||
| Memory/paths/structs | `python3 test_pathutils.py && python3 test_struct.py` | [os-base.md](ARCHITECTURE/os-base.md) |
|
||||
| POSIX syscalls | `python3 test_posix.py` (adds `test_elf.py`, `test_riscv.py`, `test_qltool.py`); then `./test_onlinux.sh` for the CI set | [os-posix.md](ARCHITECTURE/os-posix.md) |
|
||||
| Kernel proxy | `python3 test_kernel_proxy.py` (Linux host) | [kernel-proxy.md](ARCHITECTURE/kernel-proxy.md) |
|
||||
| Windows/UEFI/DOS | `python3 test_uefi.py && python3 test_dos.py`; PE suites via `test_pe.bat` on Windows | [os-windows.md](ARCHITECTURE/os-windows.md) |
|
||||
| MCU/BLOB run loops, peripherals | `python3 test_mcu.py`; `python3 -m unittest test_blob.BlobTest.test_uboot_arm` | [os-baremetal.md](ARCHITECTURE/os-baremetal.md), [hw.md](ARCHITECTURE/hw.md) |
|
||||
| Debuggers | `python3 test_qdb.py`; `python3 test_debugger.py` | [debugger.md](ARCHITECTURE/debugger.md) |
|
||||
| Extensions | `python3 test_history.py`; `python3 test_r2.py` with `[RE]` | [extensions.md](ARCHITECTURE/extensions.md) |
|
||||
| CLI/packaging | `python3 test_qltool.py` (needs the package installed); `python -I tests/test_qltool.py InstalledQltool_Test -v` from the repo root against an installed wheel | [cli.md](ARCHITECTURE/cli.md) |
|
||||
|
||||
CI code checks: `build-ci.yml` runs `tests/test_onlinux.sh` on Ubuntu
|
||||
(the container branch is dead code, see Roadmap) and
|
||||
`tests/test_pe.bat` on Windows after `examples/scripts/dllscollector.bat`
|
||||
(`.github/workflows/build-ci.yml:44-81`); `pythonpublish.yml` runs
|
||||
`poetry check --lock`, `python -m build`, `twine check --strict`, and the
|
||||
installed-wheel test on every push and pull request
|
||||
(`.github/workflows/pythonpublish.yml:3`, `:20-44`).
|
||||
|
||||
Known failing or gated cases on a clean Linux checkout (evidence from
|
||||
this refresh): `test_blob.BlobTest.test_blob_raw` (missing fixture in the
|
||||
pinned submodule), `test_elf_ko.ELF_KO_Test.test_demigod_m0hamed_x86`
|
||||
(needs the unzip step), `test_kernel_proxy…test_ptr_out_writes_back_to_guest_memory`
|
||||
(test bug, see owner), and `InstalledQltool_Test` when the package is not
|
||||
installed. Coverage gaps: no unit tests for the hook engine, memory
|
||||
manager, heap, or calling conventions; Windows and macOS suites need
|
||||
their hosts.
|
||||
|
||||
Review constraints beyond the shared checks: keep dependency direction
|
||||
downward; keep guest-derived sizes bounded before host allocation; keep
|
||||
`unicorn` pinned; new runtime dependencies are blockers (see deviations).
|
||||
|
||||
## Roadmap
|
||||
|
||||
Status: released project in maintenance (1.4.12.dev0). All modules are
|
||||
`done` except the kernel proxy, which is `in progress (Phase 0)`.
|
||||
Established implementation milestones exist only for the hybrid kernel
|
||||
work in `TODO.md` (Phases 0–5, `TODO.md:81-622`); no other milestone IDs
|
||||
are used in the tree.
|
||||
|
||||
Accepted coding work (evidence-backed, not yet done):
|
||||
|
||||
- Kernel proxy Phase 0 test fix (`tests/test_kernel_proxy.py:449-451`);
|
||||
Phases 1–5 remain designs (`TODO.md:271-622`).
|
||||
- Bump `examples/rootfs` so `test_blob_raw` has its fixture
|
||||
([os-baremetal.md](ARCHITECTURE/os-baremetal.md)).
|
||||
|
||||
Proposals (from `TODO.md:624-693`, each with an owner and check in the
|
||||
module docs): remove bare `except:` blocks and `assert` validation; bound
|
||||
`read_cstring`; structured `map_info` lookup; move the thumb fixup into
|
||||
the arch layer; implement Windows/UEFI `save/restore`; hook-engine
|
||||
`isinstance` cleanup; profile-driven guard page; un-skip the ARM and
|
||||
wchar tests. Cross-cutting gaps: `ChangeLog` stops at 1.4.6
|
||||
(`ChangeLog:4`); macOS CI is commented out
|
||||
(`.github/workflows/build-ci.yml:83-93`); `jexamples/` is unexercised;
|
||||
`.github/workflows/build-ci.yml:77` reads `matrix.contrainer` (typo) so
|
||||
the Docker job always takes the native branch. The feature wishlist is
|
||||
GitHub issue [#333](https://github.com/qilingframework/qiling/issues/333).
|
||||
|
||||
## Development Loop
|
||||
|
||||
Frame → Write → Prove → Review → Gate. Findings return to Write;
|
||||
uncertainty that changes the plan returns to Frame.
|
||||
|
||||
Use one subagent per role when available, otherwise distinct labeled
|
||||
passes. Tester and Verifier report findings and never edit; Coder repairs.
|
||||
|
||||
| Role | Stages | Handoff |
|
||||
| ---- | ------ | ------- |
|
||||
| Planner | Frame | Goal, observable checks, assumptions, affected files/owners, and plan. |
|
||||
| Coder | Write | Planned changes or repairs to named findings. |
|
||||
| Tester | Prove | Commands, results, and behavioral/structural evidence. |
|
||||
| Verifier | Review + Gate | Evidence-backed findings or verified completion. |
|
||||
|
||||
### The loop
|
||||
|
||||
1. **Frame:** Inspect the request, code, docs, and conventions before
|
||||
planning. Give the goal and each plan step an observable check. When
|
||||
using eatmycode, run its Version and Freshness Gate before trusting
|
||||
architecture; include versions, migration scope, Index/agent-file
|
||||
changes, and verification commands in architecture plans. Resolve
|
||||
uncertainty from evidence and record the narrowest supported assumptions.
|
||||
Only Planner may ask one focused question, when a required decision
|
||||
cannot be discovered or safely inferred and guessing changes the result.
|
||||
2. **Write:** Apply Coding Discipline. Make the planned change; for a
|
||||
repair, address only named findings. Update affected architecture with
|
||||
changes to its documented contracts.
|
||||
3. **Prove:** Run relevant tests and structural checks, retaining observable
|
||||
evidence. For architecture work under eatmycode, apply its Architecture
|
||||
Verification. Failures and missing, duplicate, or obsolete coverage
|
||||
become Coder findings. Re-run affected checks after repairs; never send
|
||||
a red result to Review.
|
||||
4. **Review:** Apply every Review Check as a separate pass over full affected
|
||||
files. Use an independent agent or isolated pass for Fit, Dependencies,
|
||||
and Security when available. Return findings to Coder, then re-prove
|
||||
and re-review the repairs.
|
||||
5. **Gate:** Confirm completion only when the Definition of Done passes.
|
||||
Return unmet criteria to the responsible stage; continue until resolved.
|
||||
If an external constraint prevents verification, state the missing
|
||||
evidence and remaining work without claiming completion or readiness.
|
||||
|
||||
Handoffs are automatic. Continue without pauses for plan approval,
|
||||
permission to continue, or review/reporting ceremonies. Finish with the
|
||||
harness's normal concise completion handoff.
|
||||
|
||||
### Definition of Done
|
||||
|
||||
- **Correctness:** The goal and named checks pass. Tests cover claimed
|
||||
behavior; bug fixes have a reproducing regression test. The project
|
||||
builds and tests from a fresh clone without local-only dependencies.
|
||||
Owning modules' **How to Test** commands pass with evidence.
|
||||
- **Review:** Every Review Check ran and its completion threshold passes.
|
||||
- **Contract:** Docs reflect source and let an agent locate owners,
|
||||
constraints, and verification commands. When using eatmycode, architecture
|
||||
satisfies its Output Contract, verification, and version rules. Public
|
||||
names, signatures, errors, and recovery are intelligible. Breaking
|
||||
changes, deprecations, dependencies, licenses, and attribution are handled;
|
||||
commit or PR text, when present, explains why.
|
||||
- **Scope:** Changed lines serve the goal and follow Coding Discipline;
|
||||
no debugging remnants, commented-out code, secrets, tokens, or local paths
|
||||
remain. Test edits follow the inventory and coverage rules below.
|
||||
|
||||
### Iterating without thrashing
|
||||
|
||||
- Each repair pass targets a named finding; nits alone do not trigger one.
|
||||
- Two no-change passes force Gate re-evaluation. If Done still fails,
|
||||
return the surviving evidence to Frame.
|
||||
- Three passes against the same finding return to Frame for a new approach.
|
||||
- Never widen scope to satisfy a finding. Record coding follow-ups under
|
||||
**Open Gaps / Roadmap** and keep non-coding work outside architecture.
|
||||
|
||||
## Coding Discipline
|
||||
|
||||
- Implement only the goal. Prefer the simplest approach that passes its
|
||||
checks; simplify code materially larger than the problem.
|
||||
- Match local style. Avoid speculative features, flexibility, single-use
|
||||
abstractions, and checks for impossible conditions.
|
||||
- Keep edits surgical: no unrelated refactoring, reformatting, or cleanup.
|
||||
Remove imports, variables, and functions made unused by this change;
|
||||
leave pre-existing dead code alone unless requested.
|
||||
- Make success concrete: validation rejects invalid input in a named test;
|
||||
a regression test fails before a bug fix and passes after; behavior tests
|
||||
pass before and after a refactor.
|
||||
|
||||
### Before editing tests
|
||||
|
||||
Before any test edit, including during Write, inventory the whole suite:
|
||||
enumerate every test file and case name, then read in full tests whose
|
||||
subject, fixtures, or assertions touch the change. Use a subagent for broad
|
||||
inventory when supported. Plan all additions, changes, merges, and removals
|
||||
from that evidence, citing `file:line`, before executing the test edits.
|
||||
|
||||
- **Reuse first:** Extend the test owning the behavior or sharing its
|
||||
setup, fixtures, and subject. Add a function/file only if no existing
|
||||
owner fits or merging would obscure which case failed.
|
||||
- **Add only required coverage:** A bug fix needs its regression test;
|
||||
a capability needs a test of its claimed behavior. Avoid duplicates.
|
||||
- **Retire only what changed:** Remove tests of deleted behavior and merge
|
||||
new duplicates, citing surviving coverage. Record unrelated suspected
|
||||
redundancy under **Open Gaps / Roadmap**.
|
||||
- **Preserve coverage:** Never delete or weaken tests to turn red green.
|
||||
Removal needs evidence that behavior is gone or covered elsewhere;
|
||||
coverage of claimed behavior must not decrease.
|
||||
|
||||
### Project-Specific Deviations
|
||||
|
||||
- Emulation fidelity beats abstraction: syscall, API, and peripheral
|
||||
implementations mirror the real platform's observable behavior even
|
||||
when that means repetitive per-OS or per-chip code. Cross-OS
|
||||
"unification" is a scope increase, not a simplification.
|
||||
- Coverage is demand-driven by design (see the OS and HW module docs).
|
||||
Adding an unrequested syscall, Win32 API, or peripheral register is
|
||||
out of scope; record it under the owning module's **Open Gaps /
|
||||
Roadmap**.
|
||||
- `unicorn` is hard-pinned (`pyproject.toml:39`). Changing it, or any
|
||||
behavior that depends on its version, is a project-wide event and
|
||||
never an incidental part of another change.
|
||||
- The kernel proxy must integrate only through `set_syscall` and the fd
|
||||
table (`TODO.md:65-79`); changes to `load_syscall` or existing handlers
|
||||
on its behalf are out of scope.
|
||||
|
||||
## Review Checks
|
||||
|
||||
Run every check against every change before confirming a code edit is
|
||||
complete, even when no commit or merge is requested. Keep checks separate.
|
||||
|
||||
- **Evidence or no finding:** Cite `file:line` for every finding.
|
||||
- **Repository authority:** Demand only conventions supported by the tree.
|
||||
- **Full context:** Read affected files, not only hunks; context can expose
|
||||
unreachable code, unused parameters, or hidden duplication.
|
||||
- **Code and impact:** Review the change, never the author or how it was made.
|
||||
|
||||
### 1. Style and Naming
|
||||
|
||||
Check indentation and local conventions; leave machine-checkable formatting
|
||||
to existing formatters/linters and never demand unrelated reformatting.
|
||||
Mixed indentation is `major`; a consistent new file with the wrong local
|
||||
indent is `nit`. Compare names with nearby precedents. If the repository
|
||||
is inconsistent, demand nothing. A local naming mismatch is `nit`; an
|
||||
inconsistent public name is `major`.
|
||||
|
||||
### 2. Duplication
|
||||
|
||||
Search distinctive constants, errors, fields, and call sequences, beyond
|
||||
symbol names, for the same job. Cite both sites and a remedy. Cross-layer
|
||||
duplication is `major`; small local repetition is `nit`. Similar code with
|
||||
meaningfully different branches is not duplication.
|
||||
|
||||
### 3. Quality
|
||||
|
||||
Require followable control flow, errors handled where they occur, and
|
||||
proportionate abstractions. Swallowed errors, inappropriate prints,
|
||||
unexplained magic values, and dead branches are `major`. Remove unrequested
|
||||
configurability, one-caller wrappers, filler comments, debugging remnants,
|
||||
and unrelated formatting. Missing tests belong to Prove.
|
||||
|
||||
### 4. Fit
|
||||
|
||||
Read the root architecture and owning module before the diff. Check
|
||||
language/toolchain constraints, conventions, scope, layering, ownership,
|
||||
invariants, public-API growth, compatibility, and performance claims against
|
||||
source. A layering violation or unjustified public API is `major`.
|
||||
Architectural/public-behavior changes need matching docs in the same change.
|
||||
|
||||
### 5. Dependencies
|
||||
|
||||
Check manifests/imports, maintenance, supply-chain risk, advisories,
|
||||
install-time behavior, license, transitive cost, and standard-library
|
||||
alternatives. An unjustified top-level dependency is `major`; a live
|
||||
advisory or abandoned upstream is `blocker`. Incomplete evidence does not pass.
|
||||
|
||||
### 6. Security
|
||||
|
||||
Check defects and widened exposure: unsafe memory access, unchecked sizes
|
||||
or offsets, integer overflow, traversal, unsafe deserialization, command
|
||||
construction, committed secrets, and unbounded untrusted input. Trace input
|
||||
to impact; without a reachable path there is no finding. A real defect is
|
||||
`major`; a trust-boundary break is `blocker`. Describe fixes without exploit
|
||||
steps.
|
||||
|
||||
### Severity and the completion threshold
|
||||
|
||||
| Severity | Effect |
|
||||
| -------- | ------ |
|
||||
| `blocker` | Must not confirm completion or merge. |
|
||||
| `major` | Must be resolved before confirming completion or merging. |
|
||||
| `nit` | Apply or consciously decline. |
|
||||
| `info` | Context or a question; no action implied. |
|
||||
|
||||
Confirm completion or merge only with no `blocker` or unresolved `major`.
|
||||
A check that did not run does not pass; explain evidence-backed
|
||||
inapplicability. Findings feed Write and Gate directly.
|
||||
|
||||
### Project-Specific Deviations
|
||||
|
||||
- **Security, scope.** Qiling *emulates* untrusted binaries; guest code
|
||||
doing something hostile inside the sandbox is the product working, not
|
||||
a finding. Findings target the host boundary: rootfs escape via path
|
||||
handling (`qiling/os/path.py:239`), unchecked guest-controlled sizes or
|
||||
offsets reaching host allocations or `struct` unpacking, parser input in
|
||||
`qiling/loader/` reachable from an untrusted image, and syscalls
|
||||
forwarded to the host by the kernel proxy
|
||||
(`qiling/os/posix/kernel_proxy/__init__.py:168-211`; guest-buffer
|
||||
marshalling at `:237-265`).
|
||||
- **Dependencies.** `pyproject.toml` is the only manifest. A new
|
||||
top-level runtime dependency is `blocker` absent an explicit request;
|
||||
optional integrations belong in an extra (`fuzz`, `RE`).
|
||||
- **Style and Naming.** No formatter or linter runs in CI; the check enforces
|
||||
only `.editorconfig` (4-space indent, LF, final newline) and the local
|
||||
conventions in *Coding Style and Code Design*.
|
||||
|
||||
## Index
|
||||
|
||||
Reading path: run the freshness gate, read the sections above, pick the
|
||||
owner below from the paths you are touching, then read that module doc
|
||||
and its listed partners before the code.
|
||||
|
||||
| Module doc | Source paths | Responsibility | Read it when you… |
|
||||
| ---------- | ------------ | -------------- | ------------------ |
|
||||
| [core.md](ARCHITECTURE/core.md) | `qiling/__init__.py`, `core.py`, `core_hooks*.py`, `core_struct.py`, `utils.py`, `const.py`, `exception.py`, `log.py`, `host.py`, `profiles/` | `Qiling` facade, composition order, hook engine, component factories, profiles, logging | add a constructor option or hook type, register a new arch/OS/loader name, change save/restore or logging. Partners: every other module |
|
||||
| [arch.md](ARCHITECTURE/arch.md) | `qiling/arch/`, `qiling/cc/` | CPU layer: Unicorn instance, registers, stack, disassembler, CPU models, calling conventions | add an architecture or CPU model, touch registers/thumb handling, change argument marshalling. Partners: os-base, debugger, os-posix (syscall ABI) |
|
||||
| [loader.md](ARCHITECTURE/loader.md) | `qiling/loader/` | ELF, PE, PE/UEFI, Mach-O, DOS, MCU firmware, raw blob loading and initial state | change image parsing, entry/exit points, DLL/ld.so resolution, PE cache, MCU peripheral wiring. Partners: os-base, os-posix, os-windows, hw |
|
||||
| [os-base.md](ARCHITECTURE/os-base.md) | `qiling/os/*.py` | Shared OS services: memory manager/heap, fcall, rootfs paths, fs mapper, fd objects, threads, stats, structs | change memory mapping, path virtualization, API-call protocol, stdio, struct helpers. Partners: all OS personalities, extensions |
|
||||
| [os-posix.md](ARCHITECTURE/os-posix.md) | `qiling/os/posix/` (except `kernel_proxy/`), `qiling/os/linux/`, `freebsd/`, `macos/`, `qnx/` | Syscall dispatch, ABIs, syscall implementations, Linux threads/futex/procfs, kernel modules, macOS/QNX personalities | add or fix a syscall, change dispatch or fd handling, touch multithread emulation. Partners: os-base, arch, loader, kernel-proxy |
|
||||
| [kernel-proxy.md](ARCHITECTURE/kernel-proxy.md) | `qiling/os/posix/kernel_proxy/`, `TODO.md` | Opt-in forwarding of chosen syscalls to a real Linux kernel via a helper process | work on the hybrid kernel roadmap, the IPC protocol, proxy fds, or `forward_syscall`. Partners: os-posix, os-base |
|
||||
| [os-windows.md](ARCHITECTURE/os-windows.md) | `qiling/os/windows/`, `qiling/os/uefi/`, `qiling/os/dos/` | Win32/NT API emulation and kernel objects, UEFI services/protocols, DOS interrupts | add a Win32 API or UEFI protocol, change handles/registry/fibers/threads, touch DOS interrupts. Partners: loader, os-base, arch |
|
||||
| [os-baremetal.md](ARCHITECTURE/os-baremetal.md) | `qiling/os/mcu/`, `qiling/os/blob/`, `qiling/extensions/multitask.py` | MCU and raw-blob run loops, cooperative multitasking over Unicorn | change stepping/fast mode, interrupt delivery timing, blob execution. Partners: hw, arch, loader |
|
||||
| [hw.md](ARCHITECTURE/hw.md) | `qiling/hw/`, `qiling/extensions/mcu/` | MMIO peripherals, hardware manager, chip `env` maps | add a peripheral or chip, change MMIO routing or peripheral hooks. Partners: os-baremetal, loader, os-base |
|
||||
| [debugger.md](ARCHITECTURE/debugger.md) | `qiling/debugger/` | GDB remote-serial server and Qdb (stepping, branch prediction, record/replay) | change RSP handling, target XML, Qdb commands or per-arch support. Partners: core, arch |
|
||||
| [extensions.md](ARCHITECTURE/extensions.md) | `qiling/extensions/` (except `multitask.py`, `mcu/`) | AFL fuzzing, coverage/trace writers, heap sanitizer, r2/IDA integration, pipes, reports, SDK stub generator | add a coverage/trace format, fuzzing harness support, sanitizer, or analysis integration. Partners: core, os-base, cli |
|
||||
| [cli.md](ARCHITECTURE/cli.md) | `qltool`, `qiling/cli.py`, `qltui.py` | `qltool` subcommands/flags and the TUI; packaging smoke test | add or change a CLI flag, TUI prompt, or console-script behavior. Partners: core, debugger, extensions |
|
||||
`Qiling.__init__` composes arch → hooks/packing → logging/profile → loader →
|
||||
memory → OS → optional hardware, then runs the loader. `run()` applies
|
||||
patches and delegates to the OS loop; Unicorn hooks enter Python dispatch.
|
||||
Arch owns CPU/register state, memory owns mappings, loaders own image state,
|
||||
and OS personalities own process/API state. Factory names are compatibility
|
||||
contracts ([core](qiling/core.py), [factories](qiling/utils.py)).
|
||||
|
||||
Guest bytes/pointers cross host boundaries through loaders, memory, files
|
||||
and sockets. Rootfs handling is not complete
|
||||
host isolation; explicit mappings expose host objects. Snapshot files use
|
||||
pickle and require trusted producers. The opt-in kernel proxy executes real
|
||||
host syscalls. Follow the owners below for these contracts. Cooperative
|
||||
scheduling and partial snapshots do not provide complete machine recovery.
|
||||
|
||||
## Code Conventions
|
||||
|
||||
- **Required:** [.editorconfig](.editorconfig) sets Python UTF-8, LF, four
|
||||
spaces, final newline and trailing-space removal. No formatter, linter or
|
||||
type-checker gate is configured in the manifest/CI.
|
||||
- **Observed:** `Ql*` classes, snake_case helpers, uppercase constants;
|
||||
legacy camelCase APIs and wildcard imports remain. Follow nearby code,
|
||||
preserve public names, use `ql.log` and `QlError*` where established;
|
||||
annotations are mixed ([core](qiling/core.py), [errors](qiling/exception.py)).
|
||||
- **Preserved project rules:** favor platform fidelity, keep API/peripheral
|
||||
additions demand-driven, and treat Unicorn upgrades as project-wide work.
|
||||
New top-level runtime dependencies require an explicit request; optional
|
||||
integrations use extras. These strengthen shared review (prior project
|
||||
architecture; current [manifest](pyproject.toml)). Keep dependencies toward
|
||||
underlying services; read [core](ARCHITECTURE/modules/core.md) before
|
||||
changing cross-layer imports for the existing exceptions.
|
||||
- `AGENT.md` and `CLAUDE.md` alias this page; asset/fixture edit constraints
|
||||
live with owners.
|
||||
|
||||
## Verification
|
||||
|
||||
| Change/check | Command and working directory | Prerequisites / pass evidence |
|
||||
| --- | --- | --- |
|
||||
| Setup | `python3 -m venv .venv`; activate it; `python -m pip install -e .` — root | Python in declared range; engine wheels/build prerequisites. |
|
||||
| Local CLI | `python qltool --help` — root | Installed dependencies; help exits zero. |
|
||||
| Shared primitives | `python -m unittest test_pathutils test_struct` — `tests/` | Passing assertions; no full emulation claim. |
|
||||
| Runtime changes | Owner commands below; Linux aggregate `./test_onlinux.sh` — `tests/` | [Rootfs submodule](.gitmodules), test-specific libraries/hosts; aggregate stops on first failure. |
|
||||
| Build / metadata | [CLI and build checks](ARCHITECTURE/modules/cli-build.md#verification) for packaging/config changes | Mirrors [package CI](.github/workflows/pythonpublish.yml); lint/type checks unavailable. |
|
||||
|
||||
## Task Index
|
||||
|
||||
| Source paths / task trigger | Responsibility | Read next |
|
||||
| --- | --- | --- |
|
||||
| `qiling/{core*,utils,const,exception,host,log,__init__}.py`, `arch/`, `cc/`, `loader/`, `profiles/`; composition, CPU, images | Runtime and loading | [Runtime routes](ARCHITECTURE/indexes/runtime.md) |
|
||||
| `qiling/os/` shared files, `posix/` except `kernel_proxy/`, `linux/`, `freebsd/`, `macos/`, `qnx/`, `windows/`, `uefi/`, `dos/`; OS behavior | Services and personalities | [OS routes](ARCHITECTURE/indexes/operating-systems.md) |
|
||||
| `qiling/os/{mcu,blob}/`, `hw/`, `extensions/{mcu/,multitask.py}`; firmware/devices | Firmware and MMIO | [Firmware routes](ARCHITECTURE/indexes/firmware.md) |
|
||||
| `qiling/os/posix/kernel_proxy/`, hybrid-kernel sections of `TODO.md`, `tests/test_kernel_proxy.py`; forwarding | Host-kernel integration | [Kernel proxy](ARCHITECTURE/modules/kernel-proxy.md) |
|
||||
| `qiling/debugger/`, `tests/test_*debugger.py`, `test_qdb.py`, `qdb_scripts/`; debugging | GDB and Qdb | [Debugger](ARCHITECTURE/modules/debugger.md) |
|
||||
| Other `qiling/extensions/`, `examples/fuzzing/`, `tests/test_{history,r2}.py`; integrations | Instrumentation | [Extensions](ARCHITECTURE/modules/extensions.md) |
|
||||
| `qiling/cli.py`, `qltool`, `qltui.py`, manifests/lock, `Dockerfile`, `.github/`, test drivers, `.gitmodules`; tooling | CLI and development checks | [CLI and build](ARCHITECTURE/modules/cli-build.md) |
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Agent Rules
|
||||
|
||||
Owner: [Project architecture](../ARCHITECTURE.md)
|
||||
|
||||
Read when: before planning code changes or reviewing code.
|
||||
|
||||
## Development Loop
|
||||
|
||||
Frame → Write → Prove → Review → Gate. Findings return to Write;
|
||||
uncertainty that changes the plan returns to Frame.
|
||||
|
||||
Use one subagent per role when available, otherwise distinct labeled
|
||||
passes. Tester and Verifier report findings and never edit; Coder repairs.
|
||||
|
||||
| Role | Stages | Handoff |
|
||||
| ---- | ------ | ------- |
|
||||
| Planner | Frame | Goal, observable checks, assumptions, affected files/owners, and plan. |
|
||||
| Coder | Write | Planned changes or repairs to named findings. |
|
||||
| Tester | Prove | Commands, results, and behavioral/structural evidence. |
|
||||
| Verifier | Review + Gate | Evidence-backed findings or verified completion. |
|
||||
|
||||
### The loop
|
||||
|
||||
1. **Frame:** Inspect the request, code, docs, and conventions before
|
||||
planning. Give the goal and each plan step an observable check. When
|
||||
using eatmycode, run its Version and Freshness Gate before trusting
|
||||
architecture; include versions, migration scope, routing/agent-file
|
||||
changes, and verification commands in architecture plans. Resolve
|
||||
uncertainty from evidence and record the narrowest supported assumptions.
|
||||
Only Planner may ask one focused question, when a required decision
|
||||
cannot be discovered or safely inferred and guessing changes the result.
|
||||
2. **Write:** Apply Coding Discipline. Make the planned change; for a
|
||||
repair, address only named findings. Update affected architecture with
|
||||
changes to its documented contracts.
|
||||
3. **Prove:** Run relevant tests and structural checks, retaining observable
|
||||
evidence. For architecture work under eatmycode, apply its Architecture
|
||||
Verification. Failures and missing, duplicate, or obsolete coverage
|
||||
become Coder findings. Re-run affected checks after repairs; never send
|
||||
a red result to Review.
|
||||
4. **Review:** Apply every Review Check as a separate pass over full affected
|
||||
files. Use an independent agent or isolated pass for Fit, Dependencies,
|
||||
and Security when available. Return findings to Coder, then re-prove
|
||||
and re-review the repairs.
|
||||
5. **Gate:** Confirm completion only when the Definition of Done passes.
|
||||
Return unmet criteria to the responsible stage; continue until resolved.
|
||||
If an external constraint prevents verification, state the missing
|
||||
evidence and remaining work without claiming completion or readiness.
|
||||
|
||||
Handoffs are automatic. Continue without pauses for plan approval,
|
||||
permission to continue, or review/reporting ceremonies. Finish with the
|
||||
harness's normal concise completion handoff.
|
||||
|
||||
### Definition of Done
|
||||
|
||||
- **Correctness:** The goal and named checks pass. Tests cover claimed
|
||||
behavior; bug fixes have a reproducing regression test. The project
|
||||
builds and tests from a fresh clone without local-only dependencies.
|
||||
Owning modules' **Verification** commands pass with evidence.
|
||||
- **Review:** Every Review Check ran and its completion threshold passes.
|
||||
- **Contract:** Docs reflect source and let an agent locate owners,
|
||||
constraints, and verification commands. When using eatmycode, architecture
|
||||
satisfies its Output Contract, verification, and version rules. Public
|
||||
names, signatures, errors, and recovery are intelligible. Breaking
|
||||
changes, deprecations, dependencies, licenses, and attribution are handled;
|
||||
commit or PR text, when present, explains why.
|
||||
- **Scope:** Changed lines serve the goal and follow Coding Discipline;
|
||||
no debugging remnants, commented-out code, secrets, tokens, or local paths
|
||||
remain. Test edits follow the inventory and coverage rules below.
|
||||
|
||||
### Iterating without thrashing
|
||||
|
||||
- Each repair pass targets a named finding; nits alone do not trigger one.
|
||||
- Two no-change passes force Gate re-evaluation. If Done still fails,
|
||||
return the surviving evidence to Frame.
|
||||
- Three passes against the same finding return to Frame for a new approach.
|
||||
- Never widen scope to satisfy a finding. Record coding follow-ups under
|
||||
**Known Gaps** and keep non-coding work outside architecture.
|
||||
|
||||
## Coding Discipline
|
||||
|
||||
- Implement only the goal. Prefer the simplest approach that passes its
|
||||
checks; simplify code materially larger than the problem.
|
||||
- Match local style. Avoid speculative features, flexibility, single-use
|
||||
abstractions, and checks for impossible conditions.
|
||||
- Keep edits surgical: no unrelated refactoring, reformatting, or cleanup.
|
||||
Remove imports, variables, and functions made unused by this change;
|
||||
leave pre-existing dead code alone unless requested.
|
||||
- Make success concrete: validation rejects invalid input in a named test;
|
||||
a regression test fails before a bug fix and passes after; behavior tests
|
||||
pass before and after a refactor.
|
||||
|
||||
### Before editing tests
|
||||
|
||||
Before any test edit, including during Write, inventory the whole suite
|
||||
with discovery tools; keep the full file/case listing outside model context.
|
||||
Load matching inventory entries and read in full tests whose subject,
|
||||
fixtures, or assertions touch the change. Use a subagent for broad inventory
|
||||
when supported. Plan all additions, changes, merges, and removals from that
|
||||
evidence, citing `file:line`, before executing the test edits.
|
||||
|
||||
- **Reuse first:** Extend the test owning the behavior or sharing its
|
||||
setup, fixtures, and subject. Add a function/file only if no existing
|
||||
owner fits or merging would obscure which case failed.
|
||||
- **Add only required coverage:** A bug fix needs its regression test;
|
||||
a capability needs a test of its claimed behavior. Avoid duplicates.
|
||||
- **Retire only what changed:** Remove tests of deleted behavior and merge
|
||||
new duplicates, citing surviving coverage. Record unrelated suspected
|
||||
redundancy under **Known Gaps**.
|
||||
- **Preserve coverage:** Never delete or weaken tests to turn red green.
|
||||
Removal needs evidence that behavior is gone or covered elsewhere;
|
||||
coverage of claimed behavior must not decrease.
|
||||
|
||||
## Review Checks
|
||||
|
||||
Run every check against every change before confirming a code edit is
|
||||
complete, even when no commit or merge is requested. Keep checks separate.
|
||||
|
||||
- **Evidence or no finding:** Cite `file:line` for every finding.
|
||||
- **Repository authority:** Demand only conventions supported by the tree.
|
||||
- **Full context:** Read affected files, not only hunks; context can expose
|
||||
unreachable code, unused parameters, or hidden duplication.
|
||||
- **Code and impact:** Review the change, never the author or how it was made.
|
||||
|
||||
### 1. Style and Naming
|
||||
|
||||
Check indentation and local conventions; leave machine-checkable formatting
|
||||
to existing formatters/linters and never demand unrelated reformatting.
|
||||
Mixed indentation is `major`; a consistent new file with the wrong local
|
||||
indent is `nit`. Compare names with nearby precedents. If the repository
|
||||
is inconsistent, demand nothing. A local naming mismatch is `nit`; an
|
||||
inconsistent public name is `major`.
|
||||
|
||||
### 2. Duplication
|
||||
|
||||
Search distinctive constants, errors, fields, and call sequences, beyond
|
||||
symbol names, for the same job. Cite both sites and a remedy. Cross-layer
|
||||
duplication is `major`; small local repetition is `nit`. Similar code with
|
||||
meaningfully different branches is not duplication.
|
||||
|
||||
### 3. Quality
|
||||
|
||||
Require followable control flow, errors handled where they occur, and
|
||||
proportionate abstractions. Swallowed errors, inappropriate prints,
|
||||
unexplained magic values, and dead branches are `major`. Remove unrequested
|
||||
configurability, one-caller wrappers, filler comments, debugging remnants,
|
||||
and unrelated formatting. Missing tests belong to Prove.
|
||||
|
||||
### 4. Fit
|
||||
|
||||
Follow the root Read First instructions and read the owning module before
|
||||
the diff. Check language/toolchain constraints, conventions, scope, layering,
|
||||
ownership, invariants, public-API growth, compatibility, and performance
|
||||
claims against source. A layering violation or unjustified public API is `major`.
|
||||
Architectural/public-behavior changes need matching docs in the same change.
|
||||
|
||||
### 5. Dependencies
|
||||
|
||||
Check manifests/imports, maintenance, supply-chain risk, advisories,
|
||||
install-time behavior, license, transitive cost, and standard-library
|
||||
alternatives. An unjustified top-level dependency is `major`; a live
|
||||
advisory or abandoned upstream is `blocker`. Incomplete evidence does not pass.
|
||||
|
||||
### 6. Security
|
||||
|
||||
Check defects and widened exposure: unsafe memory access, unchecked sizes
|
||||
or offsets, integer overflow, traversal, unsafe deserialization, command
|
||||
construction, committed secrets, and unbounded untrusted input. Trace input
|
||||
to impact; without a reachable path there is no finding. A real defect is
|
||||
`major`; a trust-boundary break is `blocker`. Describe fixes without exploit
|
||||
steps.
|
||||
|
||||
### Severity and the completion threshold
|
||||
|
||||
| Severity | Effect |
|
||||
| -------- | ------ |
|
||||
| `blocker` | Must not confirm completion or merge. |
|
||||
| `major` | Must be resolved before confirming completion or merging. |
|
||||
| `nit` | Apply or consciously decline. |
|
||||
| `info` | Context or a question; no action implied. |
|
||||
|
||||
Confirm completion or merge only with no `blocker` or unresolved `major`.
|
||||
A check that did not run does not pass; explain evidence-backed
|
||||
inapplicability. Findings feed Write and Gate directly.
|
||||
@@ -1,149 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# Arch — CPU architecture layer
|
||||
|
||||
## Goal
|
||||
|
||||
Own everything CPU-specific: the Unicorn `Uc` instance, register access,
|
||||
stack primitives, disassembler/assembler, CPU models, and per-arch calling
|
||||
conventions. This is the bottom layer: every other module reads `ql.arch`;
|
||||
arch depends only on Unicorn/Capstone/Keystone (with the documented
|
||||
exceptions below). No roadmap milestone applies; maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — all ten architectures are exercised by the CI suites; the CPU
|
||||
model enums are checked against Unicorn's constants by
|
||||
`tests/test_cpu_models.py` (observed: `Ran 7 tests … OK`).
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/arch/arch.py` | Abstract base `QlArch`: owns `uc`, `regs`, stack push/pop, save/restore, disassembler/assembler |
|
||||
| `qiling/arch/x86.py`, `x86_utils.py`, `x86_const.py`, `msr.py` | `QlArchIntel` base + `QlArchA8086`/`QlArchX86`/`QlArchX8664`; GDT/segment setup; MSRs |
|
||||
| `qiling/arch/arm.py`, `arm_utils.py`, `arm_const.py`, `cpr.py` | ARM: thumb handling, coprocessor registers |
|
||||
| `qiling/arch/arm64.py`, `arm64_const.py`, `cpr64.py` | AArch64 |
|
||||
| `qiling/arch/cortex_m.py`, `cortex_m_const.py` | Cortex-M on top of ARM: `QlInterruptContext`, NVIC-style exception entry/exit for MCU mode; uses `MultiTaskUnicorn` |
|
||||
| `qiling/arch/mips.py`, `riscv.py`, `riscv64.py`, `ppc.py` (+ `*_const.py`) | Remaining architectures |
|
||||
| `qiling/arch/register.py` | `QlRegisterManager`: attribute-style register read/write |
|
||||
| `qiling/arch/models.py` | CPU model enums (`X86_CPU_MODEL` … `RISCV64_CPU_MODEL`) |
|
||||
| `qiling/arch/utils.py` | `QlArchUtils` (disassembly output for verbose modes) and the `assembler()` factory |
|
||||
| `qiling/cc/__init__.py`, `intel.py`, `arm.py`, `mips.py`, `ppc.py`, `riscv.py` | Calling conventions (argument/return marshalling) consumed by `qiling/os/fcall.py` |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Local patterns:
|
||||
|
||||
- One class per architecture named `QlArch<ENUMNAME>` so `select_arch`
|
||||
can derive it (`qiling/utils.py:376-406`).
|
||||
- Register tables live in `*_const.py` as name→Unicorn-constant maps and are
|
||||
handed to `QlRegisterManager` (`qiling/arch/register.py:16`).
|
||||
- Calling conventions are small classes named after the ABI (`cdecl`,
|
||||
`stdcall`, `ms64`, `amd64`, `macosx64`, `aarch64`, `aarch32`, `mipso32`;
|
||||
`qiling/cc/intel.py:61-95`, `qiling/cc/arm.py:35-40`,
|
||||
`qiling/cc/mips.py:9`), all deriving from `QlCommonBaseCC`
|
||||
(`qiling/cc/__init__.py:110`).
|
||||
- `TODO.md:638-644` records that GDT/segment validation in
|
||||
`qiling/arch/x86_utils.py` uses `assert`; treat that as observed, not a
|
||||
convention to copy.
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- `QlArch` declares `uc` as an abstract property that each concrete arch
|
||||
builds lazily as a `cached_property` (`qiling/arch/x86.py:27`); the base
|
||||
(`qiling/arch/arch.py:32-34`) exposes `regs` (`:42`),
|
||||
`stack_push/stack_pop` (`:52`/`:66`), `save/restore` via `UcContext`
|
||||
(`:108`/`:112`), `disassembler` (`:117`), and `assembler` (`:125`).
|
||||
Everything above arch must go through these.
|
||||
- `ql.uc` is a proxy to `arch.uc` (`qiling/core.py:479`); there is exactly
|
||||
one Unicorn instance per `Qiling` (the multi-Unicorn threading idea in
|
||||
`TODO.md:462-488` is a proposal only).
|
||||
- **Layering exceptions**: `qiling/arch/cortex_m.py:22` imports
|
||||
`MultiTaskUnicorn` from `qiling/extensions/multitask.py`;
|
||||
`qiling/arch/utils.py:94` lazily imports the r2 extension for symbol
|
||||
names; `qiling/arch/x86_utils.py:10` imports `QlMemoryManager` from OS
|
||||
base for the GDT manager's constructor annotation. Do not add further
|
||||
upward imports; see [os-baremetal.md](os-baremetal.md) for the multitask
|
||||
contract.
|
||||
- CPU models come in through the `cputype` kwarg; `select_arch` forwards
|
||||
the value unvalidated (`qiling/utils.py:377`) and the arch class applies
|
||||
it with `ctl_set_cpu_model` (`qiling/arch/arm.py:47`); a model belongs
|
||||
to exactly one enum in `qiling/arch/models.py`.
|
||||
- Endianness and thumb are constructor inputs for ARM/MIPS only
|
||||
(`qiling/utils.py:379-386`).
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/arch/arch.py:22` - `QlArch(ABC)` - base class; see properties
|
||||
above.
|
||||
- `qiling/arch/register.py:11` - `QlRegisterManager` - `ql.arch.regs.rax`
|
||||
style access (`__getattr__` `:35`, `__setattr__` `:44`), plus
|
||||
`read/write` by name or Unicorn id (`:53`/`:62`).
|
||||
- `qiling/arch/x86.py:22,53,79,111` - `QlArchIntel` / `QlArchA8086` /
|
||||
`QlArchX86` / `QlArchX8664`.
|
||||
- `qiling/arch/cortex_m.py:67` - `QlArchCORTEX_M(QlArchARM)` -
|
||||
`interrupt_handler` (`:146`) consults `ql.hw.nvic` and enters the handler
|
||||
inside `QlInterruptContext` (`:25`).
|
||||
- `qiling/arch/utils.py:106` - `assembler(arch, endianness, is_thumb)` -
|
||||
Keystone factory used by `qltool code --format asm`.
|
||||
- `qiling/cc/__init__.py:9` - `QlCC` - abstract calling convention
|
||||
(`getRawParam`, `setReturnValue`, …); `QlCommonBaseCC` (`:110`).
|
||||
- `qiling/utils.py:376` - `select_arch(archtype, cputype, endian, thumb)` -
|
||||
the only construction path (`qiling/core.py:154`).
|
||||
|
||||
## Interactions
|
||||
|
||||
- Constructed first by [core.md](core.md); `QlCoreStructs`/`QlCoreHooks`
|
||||
are initialized from `arch.endian`/`arch.bits`/`arch.uc`
|
||||
(`qiling/core.py:157-158`).
|
||||
- [loader.md](loader.md) and the OS layers set initial register/stack state
|
||||
through `arch.regs` and the stack primitives.
|
||||
- `qiling/cc/` is consumed by `QlFunctionCall` ([os-base.md](os-base.md))
|
||||
and by the Windows fcall selector (`qiling/os/windows/windows.py:41-65`).
|
||||
- POSIX syscall ABIs are a separate table in
|
||||
`qiling/os/posix/syscall/abi/` ([os-posix.md](os-posix.md)), not here.
|
||||
- [debugger.md](debugger.md) reads/writes registers through this layer and
|
||||
ships per-arch GDB target XML.
|
||||
- [hw.md](hw.md) NVIC peripherals call `arch.interrupt_handler`
|
||||
(`qiling/hw/intc/cm_nvic.py:55`, `:127`).
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_cpu_models.py # pass = "Ran 7 tests … OK", exit 0
|
||||
```
|
||||
|
||||
- Broader coverage comes from `tests/test_shellcode.py` (five archs) and the
|
||||
per-OS suites; RISC-V has `tests/test_riscv.py` (`Ran 4 tests … OK`).
|
||||
- There is no unit test for `qiling/cc/`; it is proven through Windows API
|
||||
calls (Windows host) and UEFI (`tests/test_uefi.py`).
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **New architecture**: add `qiling/arch/<name>.py` with `QlArch<ENUM>`,
|
||||
a `*_const.py` register table, a `qiling/cc/` convention, a
|
||||
`qiling/os/posix/syscall/abi/` ABI, a GDB XML directory, and the enum in
|
||||
`qiling/const.py:15`; then extend `select_arch`.
|
||||
- **Register changes** affect `qiling/debugger/gdb/xml/<arch>/` and
|
||||
`qiling/debugger/qdb/arch/`; run `tests/test_debugger.py` and
|
||||
`tests/test_qdb.py`.
|
||||
- **Do not** put OS-specific state (segment selectors for Windows, TLS) in
|
||||
arch classes; the OS layer owns that (`qiling/os/windows/windows.py:132`).
|
||||
- Improvement candidate (proposal): move the thumb-bit fixup out of
|
||||
`Qiling.emu_start` (`qiling/core.py:759`) into `QlArchARM` once Unicorn
|
||||
reflects thumb mode in `cpsr` at init. Success check:
|
||||
`tests/test_shellcode.py::test_linux_arm_thumb` and `tests/test_qdb.py`
|
||||
stay green.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- PowerPC has no OS-level test suite beyond CPU model selection; RISC-V has
|
||||
four Linux tests.
|
||||
- 8086 GDB stop replies use the wrong register names (FIXME at
|
||||
`qiling/debugger/gdb/gdb.py:242`).
|
||||
- `TODO.md:655-657` notes the 32-bit GDT data segment is built with
|
||||
`QL_X86_A_PRIV_0` (`qiling/arch/x86_utils.py:167`); the 64-bit manager
|
||||
already uses ring 3 (`:200`, `:215`).
|
||||
@@ -1,113 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# CLI — qltool and qltui
|
||||
|
||||
## Goal
|
||||
|
||||
Give users a no-code way to run emulations: `qltool run` executes a binary
|
||||
against a rootfs, `qltool code` runs shellcode (hex/asm/bin), `qltool
|
||||
examples` prints usage samples, and `qltool qltui` launches an interactive
|
||||
TUI that gathers the same options. Owns argument parsing and the mapping
|
||||
from flags to `Qiling` kwargs; it must not implement emulation behavior.
|
||||
No roadmap milestone applies; maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — `tests/test_qltool.py` (observed: `Ran 8 tests … OK` once the
|
||||
package is installed) shells out to `qltool` for run/code subcommands,
|
||||
coverage, JSON, and log filtering; `InstalledQltool_Test` checks the
|
||||
installed console script, shellcode exit status, bundled profiles, and TUI
|
||||
import outside the checkout.
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qltool` | Checkout launcher: `from qiling.cli import run` |
|
||||
| `qiling/cli.py` | Argparse CLI; installed as the `qltool` console script (`pyproject.toml:33-34`); builds kwargs and drives `Qiling` |
|
||||
| `qltui.py` | questionary/pyfx/termcolor TUI; collects options and returns them to `qltool` |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Enum-valued flags use `__make_enum_arg`
|
||||
argparse actions mapping lowercase names to `QL_ARCH`/`QL_OS`/`QL_ENDIAN`/
|
||||
`QL_VERBOSE` (`qiling/cli.py:59-75`). Errors surface as argparse errors or
|
||||
`Qiling` exceptions; the process exits with `ql.os.exit_code`
|
||||
(`qiling/cli.py:321`). `qltui.py` is the only module that imports
|
||||
`questionary`, `pyfx`, and `termcolor` (`pyproject.toml:48-50`).
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- Subcommands: `run` (`-f`, `--rootfs`, `--args …`), `code` (`-f`/`-i`,
|
||||
`--arch`, `--os`, `--endian`, `--thumb`, `--format asm|hex|bin`),
|
||||
`examples`, `qltui` (`qiling/cli.py:196-217`); common flags cover
|
||||
verbosity, `--env` (pickled dict), `--gdb`, `--qdb`, `--rr`,
|
||||
`--profile`, `--filter`, `--log-file`, `--log-plain`, `--no-console`,
|
||||
`--root`, `--debug-stop`, `--multithread`, `--timeout`, `--coverage-file`,
|
||||
`--coverage-format`, `--json`, `--libcache` (`:225-242`).
|
||||
- `handle_run`/`handle_code` return the kwargs dict; `Qiling(**ql_args)`
|
||||
at `qiling/cli.py:276` is the single construction point, followed by
|
||||
optional Qdb (`:279`), gdbserver (`:285`), coverage-wrapped `ql.run()`
|
||||
(`:306-310`), JSON report (`:312`), and exit (`:321`).
|
||||
- `code --format asm` assembles with Keystone via
|
||||
`qiling.arch.utils.assembler` (`:104`).
|
||||
- The installed script and the checkout launcher must behave identically;
|
||||
`InstalledQltool_Test` runs with an empty `PYTHONPATH` from a temp
|
||||
directory to prove profiles ship in the wheel (`tests/test_qltool.py:55-72`).
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/cli.py:189` - `run()` - argparse setup and dispatch.
|
||||
- `qiling/cli.py:129` - `handle_run(options)` - builds
|
||||
`{'argv': [file]+args, 'rootfs': …}`.
|
||||
- `qiling/cli.py:78` - `handle_code(options)` - reads hex/asm/bin shellcode.
|
||||
- `qiling/cli.py:59` - `__make_enum_arg(enum_rmap, aliases)` - argparse
|
||||
action factory.
|
||||
- `qltui.py` - TUI entry invoked by the `qltui` subcommand.
|
||||
|
||||
## Interactions
|
||||
|
||||
- Thin client of [core.md](core.md): constructs `Qiling` and calls `run()`.
|
||||
- Attaches [debugger.md](debugger.md) via `--gdb [HOST:PORT]` /
|
||||
`--qdb [--rr]`.
|
||||
- Uses [extensions.md](extensions.md) for coverage (`cov_utils.factory`)
|
||||
and the JSON report.
|
||||
- Packaging: `pyproject.toml:33-34` defines the console script; the PyPI
|
||||
workflow runs `InstalledQltool_Test` against the built wheel
|
||||
(`.github/workflows/pythonpublish.yml:41-44`).
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
python3 -m pip install -e .
|
||||
cd tests && python3 test_qltool.py # pass = "Ran 8 tests … OK", exit 0
|
||||
```
|
||||
|
||||
- `Qltool_Test` uses the checkout launcher `../qltool`;
|
||||
`InstalledQltool_Test` needs the console script on the interpreter's
|
||||
scripts path (hence the install step). Without it the two installed
|
||||
cases error with `FileNotFoundError: …/bin/qltool`.
|
||||
- Manual smoke test from the repository root (pass = prints
|
||||
`Hello, World!`):
|
||||
|
||||
```sh
|
||||
./qltool run -f examples/rootfs/x8664_linux/bin/x8664_hello --rootfs examples/rootfs/x8664_linux
|
||||
```
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **New flag**: add it to the common or subcommand parser, thread it into
|
||||
`ql_args` or the post-construction block, mirror it in `qltui.py`, and
|
||||
add a `Qltool_Test` case.
|
||||
- **Do not** add emulation logic here; new behavior belongs to the owning
|
||||
module and is only surfaced by a flag.
|
||||
- Keep `qltool` (checkout) a two-line launcher so installed and checkout
|
||||
behavior cannot diverge.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- `qltui.py` has only an import smoke test; its interactive flows are not
|
||||
covered.
|
||||
- Complex setups (fs mappers, custom hooks) still require the Python API.
|
||||
@@ -1,176 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# Core — the Qiling facade and plumbing
|
||||
|
||||
## Goal
|
||||
|
||||
Own the public API and object lifecycle of an emulation: the `Qiling` class
|
||||
composes arch, loader, memory, OS, and (bare-metal only) hardware components,
|
||||
and exposes hooks, memory/register access, patching, save/restore, and the
|
||||
component factories that resolve names to classes. It must not implement any
|
||||
CPU, format, OS, or peripheral behavior itself. No roadmap milestone applies;
|
||||
this is released infrastructure with maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — exercised by every suite; the **How to Test** command boots
|
||||
shellcode end-to-end through `Qiling.__init__` → `run()` → `emu_start`
|
||||
(observed: `Ran 6 tests … OK`).
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/__init__.py` | Exports `Qiling`; `__version__` from package metadata |
|
||||
| `qiling/core.py` | `Qiling`: constructor/composition root, `run`, `emu_start/stop`, `save/restore`, `patch`, properties, stop guard |
|
||||
| `qiling/core_hooks.py` | `QlCoreHooks` mixin: wraps Unicorn hooks, dispatches to `Hook` lists, honors `QL_HOOK_BLOCK` |
|
||||
| `qiling/core_hooks_types.py` | `Hook`, `HookAddr`, `HookIntr`, `HookRet` records |
|
||||
| `qiling/core_struct.py` | `QlCoreStructs` mixin: endian/bit-width-aware `pack*/unpack*` helpers |
|
||||
| `qiling/utils.py` | Name→class factories (`select_arch/loader/os/component/debugger`), binary sniffing, profile loading |
|
||||
| `qiling/const.py` | Enums `QL_ARCH`, `QL_OS`, `QL_VERBOSE`, `QL_INTERCEPT`, `QL_STOP`, `QL_STATE`; groupings `QL_OS_POSIX`, `QL_OS_BAREMETAL`; hook flags |
|
||||
| `qiling/exception.py` | `QlErrorBase` and its subclasses |
|
||||
| `qiling/host.py` | `QlHost`: describes the *hosting* platform for pass-through decisions |
|
||||
| `qiling/log.py` | Logger setup, colored/plain formatters, regex filter behind `Qiling.filter` |
|
||||
| `qiling/profiles/*.ql` | Default per-OS INI profiles merged with user overrides |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python only; root toolchain and style rules apply (see
|
||||
[ARCHITECTURE.md](../ARCHITECTURE.md#coding-style-and-code-design)). Local
|
||||
patterns to follow:
|
||||
|
||||
- Circular-import avoidance with `TYPE_CHECKING` guards and string
|
||||
annotations (`qiling/core.py:1-33`, `qiling/log.py:19-20`).
|
||||
- Components are reached through read-only properties (`mem`, `arch`,
|
||||
`loader`, `os`, `hw`, `log`; `qiling/core.py:203-244`); setters exist only
|
||||
for `verbose`, `debugger`, `filter`, `debug_stop`.
|
||||
- Public hook methods are fully typed with callback `Protocol`s
|
||||
(`qiling/core_hooks.py:55-131`).
|
||||
- Errors are `QlErrorBase` subclasses (`qiling/exception.py:9`); constructor
|
||||
failures raise `QlErrorFileNotFound`, `QlErrorArch`, `QlErrorOsType`
|
||||
(`qiling/core.py:104`, `:143-149`).
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- **Composition order is fixed** (`qiling/core.py:154-197`): arch →
|
||||
`QlCoreStructs`/`QlCoreHooks` init → logger → profile → loader → memory →
|
||||
OS → hw (bare-metal only) → `loader.run()` → stop guard. Later components
|
||||
read earlier ones during their own `__init__` (e.g. the OS reads
|
||||
`ql.profile` and `ql.mem`), so reordering breaks them.
|
||||
- **Core never imports concrete subclasses.** All resolution goes through
|
||||
the dynamic-import factories in `qiling/utils.py` (`select_arch` `:376`,
|
||||
`select_loader` `:297`, `select_os` `:409`, `select_component` `:323`,
|
||||
`select_debugger` `:332`), which derive module and class names from the
|
||||
enum names. Adding an arch/OS means adding a module whose class name
|
||||
matches that derivation.
|
||||
- **Hook dispatch protocol**: `QlCoreHooks` registers one Unicorn hook per
|
||||
type and fans out to Python `Hook` lists; a callback returning an int with
|
||||
`QL_HOOK_BLOCK` set (`qiling/const.py:77`) stops remaining hooks
|
||||
(`qiling/core_hooks.py:186`, `:209`). Address hooks are keyed per address
|
||||
(`hook_address`, `:550`). `begin=1, end=0` means "whole address space".
|
||||
- **Exceptions raised inside hooks** are captured by the hook wrapper
|
||||
(`qiling/core_hooks.py:141-144`) into `ql._internal_exception`, which
|
||||
`emu_start` resets beforehand and re-raises after `uc.emu_start` returns
|
||||
(`qiling/core.py:763`, `:773-774`), because Unicorn cannot propagate
|
||||
Python exceptions through its C callbacks.
|
||||
- **Emulation state** is tracked in `QL_STATE` (`qiling/const.py:67`) around
|
||||
`emu_start` (`qiling/core.py:765-770`); `QlOs.call` refuses to move `pc`
|
||||
once stopped to work around a Unicorn bug (`qiling/os/os.py:215-221`).
|
||||
- **Stop guard**: when `stop=QL_STOP.*` is requested, a trap page is mapped
|
||||
at or above `0x9000000` (`qiling/core.py:525`) and the loader's
|
||||
`skip_exit_check` decides whether the trap is written (`:546-558`).
|
||||
- **Thumb workaround**: `emu_start` forces the low bit of `begin` when the
|
||||
arch was initialized in thumb mode (`qiling/core.py:759`); the FIXME there
|
||||
explains why this cannot live in the arch layer today.
|
||||
- **Save/restore** snapshots are per-component dicts (`qiling/core.py:609`,
|
||||
`:658`); each component implements its own `save()/restore()`.
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/core.py:35` - `Qiling(QlCoreHooks, QlCoreStructs)` - facade;
|
||||
constructor kwargs at `:36-58` are the public construction contract
|
||||
(`argv`, `rootfs`, `env`, `code`, `ostype`, `archtype`, `cputype`,
|
||||
`verbose`, `profile`, `multithread`, `stop`, `endian`, `thumb`,
|
||||
`libcache`, …).
|
||||
- `qiling/core.py:561` - `Qiling.run(begin, end, timeout, count)` -
|
||||
instantiates the debugger if configured, applies binary patches, writes
|
||||
the exit trap, calls `os.run()`, then `debugger.run()`.
|
||||
- `qiling/core.py:743` - `Qiling.emu_start(begin, end, timeout, count)` -
|
||||
thin wrapper over `uc.emu_start`; manages thumb bit, `QL_STATE`, and
|
||||
exception re-raise.
|
||||
- `qiling/core.py:609` / `:658` - `save()` / `restore()` - snapshot
|
||||
regs/mem/hw/fd/os/loader per component; optional pickle via `snapshot=`.
|
||||
- `qiling/core.py:594` - `patch(offset, data, target=None)` - queue a binary
|
||||
or library patch applied by `do_bin_patch`/`do_lib_patch` (`:503`, `:509`).
|
||||
- `qiling/core_hooks.py:150` - `QlCoreHooks` - `hook_code` (`:400`),
|
||||
`hook_block` (`:422`), `hook_address` (`:550`), `hook_intno` (`:575`),
|
||||
`hook_mem_read/write` (`:592`/`:610`), `hook_insn` (`:646`),
|
||||
`hook_del` (`:686`).
|
||||
- `qiling/utils.py:278` - `ql_guess_emu_env(path)` - sniffs arch/OS/endian
|
||||
from path name, ELF, Mach-O, or PE headers when not given.
|
||||
- `qiling/utils.py:419` - `profile_setup(ostype, user_config)` - YAML for
|
||||
MCU, else `ConfigParser` over `qiling/profiles/<os>.ql` plus user overrides
|
||||
(path or dict); `getint` accepts any base.
|
||||
- `qiling/log.py:164` - `setup_logger(ql, logdevs, plain, override)` - builds
|
||||
the per-instance logger; `RegexFilter` (`:96`) implements `ql.filter`.
|
||||
|
||||
## Interactions
|
||||
|
||||
- Instantiates [arch.md](arch.md), [loader.md](loader.md),
|
||||
[os-base.md](os-base.md) (memory then OS), and [hw.md](hw.md) (bare-metal
|
||||
only) in the order above.
|
||||
- Lazily instantiates [debugger.md](debugger.md) inside `run()` via
|
||||
`select_debugger`.
|
||||
- [extensions.md](extensions.md), [cli.md](cli.md), and
|
||||
[kernel-proxy.md](kernel-proxy.md) consume only this public API.
|
||||
- The OS layers register their syscall/API entry hooks through
|
||||
`QlCoreHooks` (`hook_intno`/`hook_insn`/`hook_code`/`hook_intr`, see
|
||||
[os-posix.md](os-posix.md), [os-windows.md](os-windows.md)).
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_shellcode.py # pass = "Ran 6 tests … OK", exit 0
|
||||
```
|
||||
|
||||
- Covers `Qiling(code=…)` construction and `run()` for x86/x86-64/ARM/
|
||||
ARM64/MIPS Linux shellcode (`tests/test_shellcode.py:90`).
|
||||
- Hook semantics and save/restore are exercised indirectly by
|
||||
`tests/test_elf.py` and `tests/test_mcu.py` (`test_mcu_snapshot_stm32f411`,
|
||||
`tests/test_mcu.py:29`). There is no dedicated unit test for
|
||||
`QlCoreHooks`; see Open Gaps.
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **Adding a constructor option**: extend `Qiling.__init__`
|
||||
(`qiling/core.py:36`) and mirror it in `qiling/cli.py` if user-facing
|
||||
([cli.md](cli.md)); keep the composition order intact.
|
||||
- **Adding a hook type**: add the `Protocol`, the `hook_*` method, and a
|
||||
`_hook_*_cb` dispatcher in `qiling/core_hooks.py`, preserving
|
||||
`QL_HOOK_BLOCK` handling.
|
||||
- **Adding an arch/OS/loader**: only the naming derivation in
|
||||
`qiling/utils.py` factories and the enums in `qiling/const.py` change here;
|
||||
the implementation belongs to the owning module.
|
||||
- **Do not** import from `qiling/os`, `qiling/loader`, `qiling/arch` at
|
||||
module level in `qiling/core.py` beyond `TYPE_CHECKING`; the factories exist to
|
||||
keep that direction one-way.
|
||||
- Improvement candidates (proposals, not accepted work): replace the
|
||||
`type()` checks in hook dispatch with `isinstance` and document return
|
||||
semantics (`qiling/core_hooks.py:186`, `:209`); replace the hard-coded
|
||||
guard-page floor `0x9000000` (`qiling/core.py:525`) with a profile value.
|
||||
Success check: existing suites stay green.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- No dedicated unit test for the hook engine or for `save/restore` outside
|
||||
MCU; coverage is indirect.
|
||||
- The thumb workaround in `emu_start` (`qiling/core.py:759`) depends on
|
||||
Unicorn behavior; revisit only as part of a Unicorn upgrade (project-wide
|
||||
event, see the root deviations).
|
||||
- `ChangeLog` stops at 1.4.6 (`ChangeLog:4`) while `pyproject.toml:4` is
|
||||
1.4.12.dev0.
|
||||
- Feature wishlist lives in GitHub issue
|
||||
[#333](https://github.com/qilingframework/qiling/issues/333); the hybrid
|
||||
kernel roadmap is in `TODO.md` (owned by [kernel-proxy.md](kernel-proxy.md)).
|
||||
@@ -1,146 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# Debugger — GDB server and Qdb
|
||||
|
||||
## Goal
|
||||
|
||||
Let users debug emulated targets: a GDB remote-serial-protocol server so
|
||||
any GDB/IDA/lldb front end can attach cross-architecture, and Qdb, a
|
||||
built-in interactive debugger with stepping, branch prediction, and
|
||||
record/replay reverse debugging. Owns both front ends and their per-arch
|
||||
tables; it must not own register or memory semantics. No roadmap
|
||||
milestone applies; maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — Qdb covered by `tests/test_qdb.py` (observed: `Ran 6 tests … OK`,
|
||||
including RISC-V 32/64); the GDB server by `tests/test_debugger.py`
|
||||
(observed: `Ran 7 tests … OK`, ~71 s, drives scripted clients on port 9999).
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/debugger/debugger.py` | Base `QlDebugger` |
|
||||
| `qiling/debugger/gdb/gdb.py` | `QlGdb`: RSP packet handlers (`handle_c/g/G/m/M/q/v/s/X/Z/z`), `GdbSerialConn` socket transport |
|
||||
| `qiling/debugger/gdb/utils.py` | `QlGdbUtils`: breakpoint table and the per-instruction `dbg_hook` servicing breakpoints, stepping, and async interrupts |
|
||||
| `qiling/debugger/gdb/xmlregs.py`, `gdb/xml/<arch>/` | `QlGdbFeatures`: target-description XML per arch (a8086, x86, x8664, arm, arm64, cortex_m, mips, ppc, riscv, riscv64) |
|
||||
| `qiling/debugger/qdb/qdb.py` | `QlQdb(Cmd, QlDebugger)`: command loop (`do_run/step_in/step_over/continue/backward/breakpoint/…`) |
|
||||
| `qiling/debugger/qdb/arch/` | Per-arch register naming/aliases (arm, cortex-m, intel, mips, riscv) |
|
||||
| `qiling/debugger/qdb/branch_predictor/` | Predicts branch targets for step/next per arch |
|
||||
| `qiling/debugger/qdb/render/` | Register/stack/disasm context rendering per arch |
|
||||
| `qiling/debugger/qdb/utils.py`, `helper.py`, `context.py`, `misc.py`, `const.py` | Factories (`setup_branch_predictor`, `setup_context_render`), `SnapshotManager` for rr, expression helper |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Local patterns:
|
||||
|
||||
- GDB packet handlers are closures named `handle_<letter>` inside
|
||||
`QlGdb.run` (`qiling/debugger/gdb/gdb.py:248-762`) returning a reply
|
||||
string; stop replies use `SIGTRAP` (`:49`) for breakpoints and steps and
|
||||
`SIGINT` for async interrupts (`:262-265`).
|
||||
- Qdb commands are `do_<name>` methods on `QlQdb`
|
||||
(`qiling/debugger/qdb/qdb.py:204-691`).
|
||||
- Per-arch Qdb support is three parallel class families selected by
|
||||
`QL_ARCH` maps in `qiling/debugger/qdb/utils.py:109-146`; RISC-V was
|
||||
added following that pattern (`qiling/debugger/qdb/arch/arch_riscv.py:11`,
|
||||
`branch_predictor_riscv.py`, `render_riscv.py`).
|
||||
- `qiling/debugger/qdb/branch_predictor/__init__.py:13-17` contains
|
||||
tab-indented lines; do not add more.
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- **Activation**: `ql.debugger = True | "gdb" | "gdb:HOST:PORT" | "qdb" |
|
||||
"qdb:rr" | "qdb:<script>"` (`qiling/core.py:437`); `select_debugger`
|
||||
parses the string (`qiling/utils.py:332-373`); the instance is created in
|
||||
`Qiling.run` before `os.run()` and `run()`s afterwards
|
||||
(`qiling/core.py:577-592`).
|
||||
- **GDB server** hooks the entry point to pause the guest, then serves
|
||||
packets over `GdbSerialConn`; `QlGdbUtils.dbg_hook` runs on every
|
||||
instruction (`hook_code`) to service software breakpoints, single-step,
|
||||
and a throttled poll for a client `\x03` interrupt
|
||||
(`qiling/debugger/gdb/utils.py:57-88`, `INTR_POLL_INTERVAL`).
|
||||
- **Target description**: `QlGdbFeatures` loads `gdb/xml/<arch>/target.xml`
|
||||
with includes and derives the register map served for `qXfer:features`
|
||||
(`qiling/debugger/gdb/xmlregs.py:56-137`, `qiling/debugger/gdb/gdb.py:488`).
|
||||
- **Qdb** installs a breakpoint hook and drives `ql.emu_start` per
|
||||
step; branch predictors compute the next pc for `step_over`; `rr` mode
|
||||
takes a full `ql.save()` before and after each stepping command and
|
||||
keeps only the diff as a `SnapshotManager` layer
|
||||
(`qiling/debugger/qdb/utils.py:260`, decorator `:275-297`) for
|
||||
`do_backward`.
|
||||
- **State access** goes through `ql.arch.regs`, `ql.mem`, and
|
||||
`hook_address`; the one concrete-type import is `QlProcFS`, used to
|
||||
serve `/proc/self/maps` to the client (`qiling/debugger/gdb/gdb.py:38`,
|
||||
`:616`).
|
||||
- Exit reporting: when pc reaches the exit point the server replies
|
||||
`W<exit_code>` (`qiling/debugger/gdb/gdb.py:266-269`, `:716-718`);
|
||||
bare-metal targets have no
|
||||
exit code (issue #1276 comment).
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/debugger/debugger.py:13` - `QlDebugger` - base; `run()` (`:17`).
|
||||
- `qiling/debugger/gdb/gdb.py:84` - `QlGdb(QlDebugger)` - `__init__(ql, ip,
|
||||
port)` (`:88`), `run()` (`:139`) serves the session.
|
||||
- `qiling/debugger/gdb/gdb.py:820` - `GdbSerialConn` - transport;
|
||||
`poll_interrupt()` (`:859`) wired into `QlGdbUtils.check_interrupt`
|
||||
(`:145`).
|
||||
- `qiling/debugger/gdb/utils.py:16` - `QlGdbUtils` - `dbg_hook` (`:57`),
|
||||
`bp_insert/bp_remove` (`:90`/`:105`), `resume_emu` (`:120`).
|
||||
- `qiling/debugger/gdb/xmlregs.py:56` - `QlGdbFeatures(archtype, ostype)`.
|
||||
- `qiling/debugger/qdb/qdb.py:59` - `QlQdb(Cmd, QlDebugger)` -
|
||||
`__init__(ql, init_hook, rr, script)` (`:64`), `run_qdb_script` (`:87`),
|
||||
`run` (`:197`).
|
||||
- `qiling/debugger/qdb/utils.py:109` / `:128` - `setup_branch_predictor` /
|
||||
`setup_context_render` - per-arch selection.
|
||||
- `qiling/cli.py:227-229` - `--gdb`, `--qdb`, `--rr` flags ([cli.md](cli.md)).
|
||||
|
||||
## Interactions
|
||||
|
||||
- Instantiated by [core.md](core.md) at `Qiling.run` time, not
|
||||
construction.
|
||||
- Reads/writes state through [arch.md](arch.md) registers and
|
||||
[os-base.md](os-base.md) memory; breakpoints use `hook_address`.
|
||||
- GDB XML register files must match the arch register tables
|
||||
([arch.md](arch.md)); adding an arch means adding `gdb/xml/<arch>/` and
|
||||
the three Qdb families.
|
||||
- The IDA plugin in [extensions.md](extensions.md) is an alternative front
|
||||
end over the same public API.
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_qdb.py # pass = "Ran 6 tests … OK", exit 0
|
||||
```
|
||||
|
||||
- GDB server: `cd tests && python3 test_debugger.py` — pass =
|
||||
`Ran 7 tests … OK`; spawns `QlGdb` on `127.0.0.1:9999` and drives
|
||||
`SimpleGdbClient`/`ReadingGdbClient` (`tests/test_debugger.py:18`, `:47`).
|
||||
- Qdb scripts live in `tests/qdb_scripts/*.qdb` (`arm`, `arm_static`,
|
||||
`mips32el`, `riscv32`, `riscv64`, `x86`).
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **New RSP packet**: add a `handle_<letter>` closure in `QlGdb.run` and a
|
||||
scripted exchange in `tests/test_debugger.py`.
|
||||
- **Register changes** require updating `gdb/xml/<arch>/*.xml` and the
|
||||
Qdb `arch_<name>.py` alias table together.
|
||||
- **Stop-reply semantics** (`SIGTRAP`/`SIGINT`/`W`) are a client-visible
|
||||
contract; changing them needs a test in `tests/test_debugger.py`.
|
||||
- **Do not** reach into Unicorn directly; use `ql.arch.regs`/`ql.mem` so
|
||||
hooks and thumb handling stay consistent.
|
||||
- Improvement candidate (proposal): fix the a8086 register naming in stop
|
||||
replies (FIXME at `qiling/debugger/gdb/gdb.py:242`). Success check: a
|
||||
DOS target attach case in `tests/test_debugger.py`.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- Qdb has no PowerPC support (`qiling/debugger/qdb/arch/`); GDB XML covers
|
||||
all ten arches.
|
||||
- Record/replay keeps one diff layer per step but performs two full
|
||||
`ql.save()` calls per step (`qiling/debugger/qdb/utils.py:273`); slow
|
||||
on long runs.
|
||||
- `TODO.md:628-636` lists bare `except:` blocks in `qiling/debugger/qdb/qdb.py`.
|
||||
@@ -1,138 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# Extensions — optional tooling on top of the core
|
||||
|
||||
## Goal
|
||||
|
||||
House everything that builds on the public `Qiling` API without being
|
||||
required by it: AFL fuzzing, code-coverage and execution-trace writers,
|
||||
disassembly tracing, a heap sanitizer, radare2 and IDA Pro integration,
|
||||
fake stdio pipes, JSON run reports, and the Windows SDK stub generator.
|
||||
Two files under this package are owned elsewhere: `multitask.py`
|
||||
([os-baremetal.md](os-baremetal.md)) and `mcu/` ([hw.md](hw.md)). The
|
||||
kernel proxy is not here; see [kernel-proxy.md](kernel-proxy.md). No
|
||||
roadmap milestone applies; maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — the history coverage tracker has its own suite, run manually
|
||||
rather than by CI (observed: `Ran 4 tests … OK`); pipes are exercised by
|
||||
many suites; AFL and r2 need optional extras (`fuzz`, `RE`) and are
|
||||
untested here; the IDA plugin needs IDA Pro and is untested in CI.
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/extensions/afl/afl.py`, `qlfuzzer.py` | `ql_afl_fuzz` / `ql_afl_fuzz_custom` (unicornafl bridge) and the `QlFuzzer` harness base |
|
||||
| `qiling/extensions/coverage/utils.py`, `formats/` | `collect_coverage` context manager; `CoverageFactory` over `QlBaseCoverage` subclasses (`drcov`, `drcov_exact`, `ezcov`, `history`) |
|
||||
| `qiling/extensions/tracing/utils.py`, `formats/` | `collect_trace`; Tenet-style trace writers (`tenet`, `registers`) |
|
||||
| `qiling/extensions/trace.py` | Disassembly tracing: full trace or ring-buffer history |
|
||||
| `qiling/extensions/sanitizers/heap.py` | `QlSanitizedMemoryHeap`: canary-based heap sanitizer (UAF/OOB) |
|
||||
| `qiling/extensions/r2/r2.py` | `R2(ql)`: r2libr-backed sections/symbols/functions/xrefs of the loaded target |
|
||||
| `qiling/extensions/idaplugin/qilingida.py` | IDA Pro plugin driving Qiling from IDA (`PLUGIN_ENTRY`) |
|
||||
| `qiling/extensions/pipe.py` | `SimpleInStream`/`SimpleOutStream`/`NullOutStream`/`InteractiveInStream` fake stdio |
|
||||
| `qiling/extensions/report/report.py` | `generate_report(ql)`: JSON summary of a run |
|
||||
| `qiling/extensions/winsdkapi.py` | CLI generator emitting `@winsdkapi` stubs from Windows SDK JSON |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Coverage and trace writers register by
|
||||
subclassing a base with a `FORMAT_NAME` class attribute
|
||||
(`qiling/extensions/coverage/formats/base.py:16-23`,
|
||||
`qiling/extensions/coverage/formats/drcov.py:29-38`) and are discovered by
|
||||
`get_all_subclasses` (`qiling/extensions/coverage/utils.py:34`). Optional
|
||||
dependencies are imported lazily inside the modules that need them
|
||||
(`qiling/extensions/afl/afl.py`, `qiling/extensions/r2/r2.py`) so the core never
|
||||
requires `unicornafl` or `r2libr`. The IDA plugin is a single 2.3k-line
|
||||
file with IDA-specific conventions; treat it as its own style domain.
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- Everything here consumes only public APIs of [core.md](core.md) (hooks,
|
||||
`mem`, `arch.regs`, `save/restore`) and [os-base.md](os-base.md) (heap,
|
||||
stdio, `set_syscall`/`set_api`). Nothing in `qiling/os`, `qiling/loader`,
|
||||
or `qiling/arch` may import from here except the exceptions listed under
|
||||
"Downward imports only" in the root System Design section.
|
||||
- `collect_coverage` activates hooks on enter and dumps on exit even on
|
||||
exceptions (`qiling/extensions/coverage/utils.py:48-63`); `qltool
|
||||
--coverage-format` lists `factory.formats` (`qiling/cli.py:240`).
|
||||
- The heap sanitizer replaces `ql.os.heap` with a compatible object that
|
||||
surrounds chunks with canaries and detects double-free/UAF; `alloc` and
|
||||
`free` install memory hooks on the canaries to report faults
|
||||
(`qiling/extensions/sanitizers/heap.py:97-107`, `:125`).
|
||||
- `pipe.py` streams implement the `ql_file` surface expected by the fd
|
||||
table so they can be assigned to `ql.os.stdin/stdout/stderr`.
|
||||
- AFL: `ql_afl_fuzz` wraps `unicornafl.uc_afl_fuzz` with input placement,
|
||||
crash validation, and exit points; `QlFuzzer` builds a harness with
|
||||
`stage_call_site`/`feed_input` (`qiling/extensions/afl/qlfuzzer.py:75`, `:86`).
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/extensions/afl/afl.py:21` / `:87` - `ql_afl_fuzz` /
|
||||
`ql_afl_fuzz_custom`; `qiling/extensions/afl/qlfuzzer.py:14` - `QlFuzzer(ABC)`.
|
||||
- `qiling/extensions/coverage/utils.py:48` - `collect_coverage(ql, name,
|
||||
coverage_file)`; `:32` - `CoverageFactory`.
|
||||
- `qiling/extensions/tracing/utils.py:31` - `collect_trace(ql, name, trace_file)`.
|
||||
- `qiling/extensions/trace.py:145` / `:180` - `enable_full_trace` /
|
||||
`enable_history_trace`.
|
||||
- `qiling/extensions/sanitizers/heap.py:16` - `QlSanitizedMemoryHeap(ql,
|
||||
heap, fault_rate, canary_byte)`.
|
||||
- `qiling/extensions/r2/r2.py:135` - `R2(ql)`.
|
||||
- `qiling/extensions/pipe.py:62` / `:69` - `SimpleInStream` / `SimpleOutStream`;
|
||||
`:98` - `InteractiveInStream`.
|
||||
- `qiling/extensions/report/report.py:56` - `generate_report(ql, pretty_print)`.
|
||||
- `qiling/extensions/coverage/formats/history.py:13` - `History` - tested
|
||||
tracker (`tests/test_history.py`).
|
||||
|
||||
## Interactions
|
||||
|
||||
- [cli.md](cli.md) wires `--coverage-file/--coverage-format` and `--json`
|
||||
(`qiling/cli.py:306-320`).
|
||||
- The heap sanitizer wraps `QlMemoryHeap` ([os-base.md](os-base.md));
|
||||
demo `examples/uefi_sanitized_heap.py` with [os-windows.md](os-windows.md)
|
||||
UEFI, proven by `tests/test_uefi.py:24`.
|
||||
- Fuzzing harnesses in `examples/fuzzing/{linux_x8664,qnx_arm,stm32f429,
|
||||
tenda_ac15,dlink_dir815,rt_n12_b1}` pair AFL with `pipe.py` and
|
||||
`set_syscall` ([os-posix.md](os-posix.md)) or MCU mode
|
||||
([os-baremetal.md](os-baremetal.md)).
|
||||
- `pipe.py` is used by `tests/test_elf.py`, `test_pe.py`,
|
||||
`test_windows_stdio.py`, `test_riscv.py`, `test_windows_cpp_x86.py`,
|
||||
`test_windows_cpp_x8664.py`, and `test_kernel_proxy.py`.
|
||||
- The IDA plugin offers a front end comparable to [debugger.md](debugger.md).
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_history.py # pass = "Ran 4 tests … OK", exit 0
|
||||
```
|
||||
|
||||
- Coverage output end-to-end: `tests/test_qltool.py::test_qltool_coverage`
|
||||
(drcov) and `::test_qltool_json` (report).
|
||||
- r2 (needs `pip install qiling[RE]`): `cd tests && python3 test_r2.py`
|
||||
(skips without `r2libr`, `tests/test_r2.py:20`).
|
||||
- AFL (needs `pip install qiling[fuzz]` and AFL++): run a harness from
|
||||
`examples/fuzzing/linux_x8664/` (`fuzz.sh`); not automated.
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **New coverage/trace format**: subclass the base with a unique
|
||||
`FORMAT_NAME`; no registry edit needed.
|
||||
- **New extension**: keep imports to public `Qiling` API; put optional
|
||||
third-party dependencies behind a Poetry extra (root Dependencies
|
||||
deviation) and import them lazily.
|
||||
- **Do not** import extensions from `qiling/os`, `qiling/loader`, or
|
||||
`qiling/arch`; move shared code into the owning core module instead.
|
||||
- Improvement candidate (proposal): unify `trace.py` and `tracing/` behind
|
||||
one factory. Success check: `qltool` exposes trace formats like coverage
|
||||
formats and `tests/test_history.py` stays green.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- `afl/` and `r2/` depend on optional extras not installed by default;
|
||||
the IDA plugin cannot be CI-tested.
|
||||
- Two overlapping trace mechanisms exist (`trace.py` and `tracing/`).
|
||||
- `tests/test_perf.py` and `tests/view_perf_results.py` are ad-hoc
|
||||
performance tools without assertions.
|
||||
@@ -1,145 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# HW — peripheral emulation for bare-metal targets
|
||||
|
||||
## Goal
|
||||
|
||||
Emulate memory-mapped peripherals (GPIO, UART/USART, SPI, I2C, timers,
|
||||
DMA, interrupt controllers, flash, RTC, CRC, …) so MCU firmware runs
|
||||
against realistic hardware. Peripherals are instantiated from the
|
||||
chip `env` map naming which peripheral class sits at which MMIO base.
|
||||
Owns the manager, the peripheral base classes, and every chip-specific
|
||||
peripheral; the run loop belongs to [os-baremetal.md](os-baremetal.md).
|
||||
No roadmap milestone applies; maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — exercised by `tests/test_mcu.py` (observed: `Ran 18 tests … OK`)
|
||||
across GPIO, USART, EXTI, I2C, SPI, DMA, CRC, ADC, watchdog, timers, and
|
||||
NVIC on STM32F103/F407/F411/F429, GD32VF103, and SAM3X8E firmware.
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/hw/hw.py` | `QlHwManager` (`ql.hw`): creates peripherals from `env`, tracks MMIO regions, steps peripherals, save/restore; `QlPripheralHandler` MMIO bridge |
|
||||
| `qiling/hw/peripheral.py` | `QlPeripheralUtils` (read/write hooks, `monitor` decorator) and base `QlPeripheral` (ctypes register struct, `raw_read/raw_write`, `region`) |
|
||||
| `qiling/hw/connectivity.py` | `QlConnectivityPeripheral` + `PeripheralTube`: byte-stream peripherals (UART-like) with `send/recv` for tests |
|
||||
| `qiling/hw/{analog,char,dma,flash,gpio,i2c,intc,math,mem,misc,net,power,sd,spi,timer}/` | One directory per peripheral class; chip-specific implementations inside (e.g. `qiling/hw/char/stm32f4xx_usart.py`, `qiling/hw/intc/cm_nvic.py`, `qiling/hw/timer/cm_systick.py`) |
|
||||
| `qiling/hw/const/` | Register bit-field constants per chip family |
|
||||
| `qiling/hw/external_device/` | External device models attached to buses |
|
||||
| `qiling/extensions/mcu/{stm32f1,stm32f4,gd32vf1,nxp,atmel,bes}/` | Chip `env` dicts: entries typed `core`, `memory`, `mmio`, `peripheral` with `struct`, `base`, `kwargs` |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Local patterns (observed, enforced by
|
||||
`QlHwManager.create`):
|
||||
|
||||
- A peripheral is a class whose inner `Type(ctypes.Structure)` lists
|
||||
registers in datasheet order with offset comments
|
||||
(`qiling/hw/char/stm32f4xx_usart.py:12-49`); `__init__(ql, label,
|
||||
**kwargs)` builds `self.instance` with reset values (`:51-58`).
|
||||
- Register access overrides `read/write` decorated with
|
||||
`@QlPeripheral.monitor()` and falls back to `raw_read/raw_write`
|
||||
(`:60-70`, `qiling/hw/peripheral.py:157-173`).
|
||||
- Class names are looked up by `struct` string through
|
||||
`ql_get_module_function('qiling.hw', struct)` (`qiling/hw/hw.py:81`), so
|
||||
every peripheral must be exported from `qiling/hw/__init__.py`.
|
||||
- Bit-field constants live in `qiling/hw/const/<chip>_<periph>.py` as
|
||||
`IntEnum`-style classes.
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- **Creation**: `QlHwManager.create(label, struct, base, kwargs)`
|
||||
instantiates the class and records `region[label]` from
|
||||
`QlPeripheral.region` offset by `base` (`qiling/hw/hw.py:67-93`); an
|
||||
unknown `struct` logs a warning and returns `None` (FIXME at `:95`).
|
||||
- **MMIO**: the loader calls `setup_mmio(begin, size, info)`
|
||||
(`qiling/loader/mcu.py:127`), which maps a `QlPripheralHandler` through
|
||||
`QlMemoryManager.map_mmio` (`qiling/hw/hw.py:143-146`); reads/writes are
|
||||
routed to `find(address)` and the peripheral's `read/write`
|
||||
(`:37-58`, `:124-133`).
|
||||
- **Stepping**: `QlHwManager.step()` (`qiling/hw/hw.py:135-140`) calls
|
||||
`step()` on every peripheral that defines it, e.g.
|
||||
`qiling/hw/timer/cm_systick.py:29`; the MCU run loop invokes it once per
|
||||
instruction in step mode.
|
||||
- **Interrupts**: NVIC peripherals hold `arch.interrupt_handler` and raise
|
||||
pending IRQs into the CPU (`qiling/hw/intc/cm_nvic.py:55`, `:127`).
|
||||
- **User hooks**: `hook_read/hook_write` with `QL_INTERCEPT` stages
|
||||
(`qiling/hw/peripheral.py:34-46`); `watch()` enables verbose access logs.
|
||||
- **Snapshots**: `QlHwManager.save/restore` (`qiling/hw/hw.py:165-171`)
|
||||
serialize each peripheral's ctypes register struct to bytes and back
|
||||
(`qiling/hw/peripheral.py:258-262`); `QlPripheralHandler.__getstate__`
|
||||
(`qiling/hw/hw.py:24`) strips the manager reference so the MMIO handler
|
||||
can be pickled with the memory map.
|
||||
- Peripheral fidelity is demand-driven: registers behave as observed
|
||||
firmware needs, not per full datasheets (root deviations).
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/hw/hw.py:60` - `QlHwManager` - `create` (`:67`), `delete` (`:97`),
|
||||
`load_env` (`:107`), `load_all` (`:118`), `find` (`:124`), `step` (`:135`),
|
||||
`setup_mmio` (`:143`), `save/restore` (`:165`/`:171`); attribute access
|
||||
`ql.hw.usart1` via `__getattr__` (`:162`).
|
||||
- `qiling/hw/hw.py:17` - `QlPripheralHandler` - `QlMmioHandler`
|
||||
implementation bridging MMIO to peripherals.
|
||||
- `qiling/hw/peripheral.py:132` - `QlPeripheral(QlPeripheralUtils)` -
|
||||
`Type` struct, `raw_read/raw_write` (`:157`/`:163`), `read/write`
|
||||
(`:168`/`:172`), `region` (`:232`).
|
||||
- `qiling/hw/peripheral.py:15` - `QlPeripheralUtils` - `hook_read/hook_write/
|
||||
hook_del` (`:34-50`), `monitor` (`:60`).
|
||||
- `qiling/hw/connectivity.py:49` - `QlConnectivityPeripheral` -
|
||||
`send/recv` tubes used by tests to talk to UARTs.
|
||||
- `qiling/extensions/mcu/stm32f4/stm32f407.py` (and siblings) - chip `env`
|
||||
passed as `Qiling(..., env=...)`.
|
||||
|
||||
## Interactions
|
||||
|
||||
- Created by [core.md](core.md) only when `ql.baremetal`
|
||||
(`qiling/core.py:191`); populated by the MCU loader
|
||||
([loader.md](loader.md), `qiling/loader/mcu.py:110-131`).
|
||||
- Driven by [os-baremetal.md](os-baremetal.md) `QlOsMcu.run` /
|
||||
`MCUTask.on_interrupted` via `step()`.
|
||||
- Interrupt controllers call into [arch.md](arch.md) `QlArchCORTEX_M`.
|
||||
- MMIO regions are mapped through `QlMemoryManager.map_mmio`
|
||||
([os-base.md](os-base.md)).
|
||||
- YAML profiles are parsed by `profile_setup` ([core.md](core.md)) and
|
||||
merged into `ql.env` by the loader.
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_mcu.py # pass = "Ran 18 tests … OK", exit 0
|
||||
```
|
||||
|
||||
- Firmware fixtures live in `examples/rootfs/mcu/{stm32f103,stm32f407,
|
||||
stm32f411,stm32f429,gd32vf103,sam3x8e}`.
|
||||
- Each case asserts UART output or register state through
|
||||
`ql.hw.<label>` (`tests/test_mcu.py:57-76` for USART input).
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **New peripheral**: subclass `QlPeripheral` (or
|
||||
`QlConnectivityPeripheral` for byte streams), export it from
|
||||
`qiling/hw/__init__.py`, add constants under `qiling/hw/const/`, reference
|
||||
it from the chip `env`, and add a firmware-driven case to
|
||||
`tests/test_mcu.py`.
|
||||
- **Register layout changes** must keep `Type` field offsets matching the
|
||||
datasheet; `field_description` (`qiling/hw/peripheral.py:182`) is used
|
||||
for logging only.
|
||||
- **Do not** put run-loop or scheduling logic in peripherals; `step()` must
|
||||
be side-effect-bounded to one tick.
|
||||
- Improvement candidate (proposal): cache `find(address)` lookups
|
||||
(TODO at `qiling/hw/hw.py:123`). Success check: `tests/test_mcu.py`
|
||||
runtime drops with unchanged results.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- Chip coverage limited to families under `qiling/extensions/mcu/`; adding
|
||||
a chip means writing its `env` map plus any missing peripheral classes.
|
||||
- `QlHwManager.create` returns `None` for unknown peripherals instead of
|
||||
failing (FIXME at `qiling/hw/hw.py:95`).
|
||||
- No unit tests target peripherals in isolation; all coverage is
|
||||
firmware-driven.
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Firmware Routes
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing MCU/blob execution, peripheral scheduling or device models.
|
||||
|
||||
## Routes
|
||||
|
||||
| Source paths / task trigger | Responsibility | Read next |
|
||||
| --- | --- | --- |
|
||||
| `qiling/os/mcu/`, `qiling/os/blob/`, `qiling/extensions/multitask.py`, execution/count/timeout cases in `tests/test_mcu.py`, `test_blob.py`, `examples/src/blob/`; scheduling and raw execution | Bare-metal runtime | [Bare-metal](../modules/baremetal.md) |
|
||||
| `qiling/hw/`, `qiling/extensions/mcu/`, register/device cases in `tests/test_mcu.py`, `examples/mcu/`; chip maps, MMIO, interrupts, external devices | Hardware models | [Hardware](../modules/hardware.md) |
|
||||
|
||||
`tests/test_mcu.py` has shared coverage: route instruction semantics to the
|
||||
CPU owner and image/vector loading to loaders through the root runtime
|
||||
branch. Timing and peripheral effects belong to the rows above.
|
||||
`extensions/multitask.py` and `extensions/mcu/` are explicit exceptions to
|
||||
general extensions ownership. UEFI services use the root operating-system branch.
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Operating System Routes
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing guest memory/services, syscall dispatch, OS APIs or interrupts.
|
||||
|
||||
## Routes
|
||||
|
||||
| Source paths / task trigger | Responsibility | Read next |
|
||||
| --- | --- | --- |
|
||||
| `qiling/os/*.py`, `tests/test_pathutils.py`, `test_struct.py`; memory/heap, paths, file wrappers, call marshalling, structs | Shared OS services | [OS base](../modules/os-base.md) |
|
||||
| `qiling/os/posix/` except `kernel_proxy/`, `linux/`, `freebsd/`, `macos/`, `qnx/`; `tests/test_posix.py`, ELF syscall/thread/kernel, Android/QNX/Mach-O/RISC-V/network cases; `examples/src/{linux,freebsd,macos,qnx}/` | Syscall/process personalities | [POSIX](../modules/posix.md) |
|
||||
| `qiling/os/windows/`, API cases in `tests/test_pe*.py`, `test_windows*.py`, `examples/src/windows/`; Win32/NT handlers and objects | Windows APIs | [Windows](../modules/windows.md) |
|
||||
| `qiling/os/uefi/`, `tests/test_uefi.py`, `qiling/profiles/uefi.ql`; services, protocols, DXE/SMM state | UEFI services | [UEFI](../modules/uefi.md) |
|
||||
| `qiling/os/dos/`, `tests/test_dos*.py`, DOS examples; interrupts, files/disks/terminal | DOS personality | [DOS](../modules/dos.md) |
|
||||
|
||||
Image parsing in shared integration tests belongs to the runtime/loading
|
||||
branch of the root Task Index. `qiling/profiles/` parsing belongs to core;
|
||||
OS-specific values belong to the corresponding row above. `qiling/os/disk.py`
|
||||
is shared OS-base ownership; follow its DOS partner only for interrupt consumers.
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Runtime Routes
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing facade composition, CPU behavior, image loading or shared profiles.
|
||||
|
||||
## Routes
|
||||
|
||||
| Source paths / task trigger | Responsibility | Read next |
|
||||
| --- | --- | --- |
|
||||
| `qiling/{__init__,core,core_hooks,core_hooks_types,core_struct,utils,const,exception,host,log}.py`, `qiling/profiles/`, hook/lifecycle cases in `tests/test_edl.py`; options, factories, hooks, snapshots, logging, profile merge | Facade and lifecycle | [Core](../modules/core.md) |
|
||||
| `qiling/arch/`, `qiling/cc/`, `tests/test_cpu_models.py`, `test_riscv.py`, `test_shellcode.py`, `examples/shellcodes/`; register/CPU/ABI behavior | Architecture adapters | [CPU and ABI](../modules/arch.md) |
|
||||
| `qiling/loader/`, image/entry/relocation cases in `tests/test_elf*.py`, `test_pe*.py`, `test_macho*.py`, `test_uefi.py`, `test_dos*.py`, `test_mcu.py`, `test_blob.py`; executable parsing | Image mapping | [Loaders](../modules/loaders.md) |
|
||||
|
||||
Shared tests route by the changed assertion: syscall/API behavior belongs
|
||||
to the matching OS owner in the root Task Index, firmware timing/device
|
||||
behavior to its firmware branch. Profile parsing is core; individual key
|
||||
semantics belong to the consuming loader/OS. Example binaries use the same
|
||||
behavior owners; fixture preparation belongs to CLI/build tooling.
|
||||
@@ -1,152 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# Kernel proxy — hybrid syscall forwarding to a real Linux kernel
|
||||
|
||||
## Goal
|
||||
|
||||
Let a user forward individually chosen Linux syscalls from the emulated
|
||||
guest to a real Linux kernel through a helper process, without touching
|
||||
the POSIX dispatcher. Owns the `KernelProxy` API, the IPC protocol, the
|
||||
proxy subprocess, the `ql_proxy_fd` fd object, and argument descriptors.
|
||||
It must not change `load_syscall`, existing handlers, or the fd table
|
||||
code (`TODO.md:65-79`). Serves the "Hybrid Kernel Architecture" roadmap in
|
||||
`TODO.md`, Phase 0.
|
||||
|
||||
## Status
|
||||
|
||||
`in progress (Phase 0)` — Phase 0 (proof of concept) is implemented:
|
||||
lifecycle, raw and buffer-carrying forwarding, FD translation, user-hook
|
||||
interplay. `tests/test_kernel_proxy.py` has 21 cases; 20 pass and
|
||||
`test_ptr_out_writes_back_to_guest_memory` errors with `EBADF` at
|
||||
`tests/test_kernel_proxy.py:451` because the test closes proxy-side fds in
|
||||
the Qiling process (see Open Gaps). Phases 1–5 (`TODO.md:271-622`) are not
|
||||
started. Linux hosts only (`qiling/os/posix/kernel_proxy/__init__.py:62`).
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/os/posix/kernel_proxy/__init__.py` | `KernelProxy`: starts the subprocess, resolves guest/host syscall numbers, registers CALL hooks, marshals buffers |
|
||||
| `qiling/os/posix/kernel_proxy/ipc.py` | Wire format, `ProxyClient` (Qiling side) and `ProxyServer` (proxy side) |
|
||||
| `qiling/os/posix/kernel_proxy/proxy.py` | Subprocess entry point: `libc.syscall()` execution and proxy-side fd ops |
|
||||
| `qiling/os/posix/kernel_proxy/proxy_fd.py` | `ql_proxy_fd`: fd-table object whose real fd lives in the proxy |
|
||||
| `qiling/os/posix/kernel_proxy/argtypes.py` | `INT`, `FD`, `PtrIn`, `PtrOut`, `PtrInOut` descriptors |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. This package uses `from __future__ import
|
||||
annotations`, `TYPE_CHECKING` guards, and module docstrings with usage
|
||||
examples (`qiling/os/posix/kernel_proxy/__init__.py:6-28`). Errors are `QlErrorArch`,
|
||||
`QlErrorSyscallNotFound`, `QlErrorSyscallError`, and
|
||||
`QlProxyConnectionError` (`qiling/exception.py:77`). The proxy process logs
|
||||
through the standard `logging` module, not `ql.log`, because it has no
|
||||
`Qiling` instance (`qiling/os/posix/kernel_proxy/proxy.py:28`).
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- **Integration is only `set_syscall`**: `forward_syscall` registers a
|
||||
CALL hook named `ql_syscall_<name>`
|
||||
(`qiling/os/posix/kernel_proxy/__init__.py:136-166`, `:210`), so
|
||||
user ENTER/EXIT hooks still fire around it and a later user CALL hook
|
||||
overrides it (`tests/test_kernel_proxy.py:186-257`).
|
||||
- **Two syscall tables**: guest numbers come from the guest arch table and
|
||||
host numbers from the host `platform.machine()` table, both read from
|
||||
`qiling/os/linux/map_syscall.py`
|
||||
(`qiling/os/posix/kernel_proxy/__init__.py:97-126`, `:278-302`).
|
||||
Unsupported host architectures raise `QlErrorArch`.
|
||||
- **Wire format** is length-prefixed binary over a Unix socketpair with
|
||||
three message types: `SYSCALL`, `SYSCALL_EX` (with in/out buffers), and
|
||||
`FD_OP` (`qiling/os/posix/kernel_proxy/ipc.py:25-72`). Buffers are copied
|
||||
from/to guest memory by the forwarder
|
||||
(`qiling/os/posix/kernel_proxy/__init__.py:237-265`); the proxy never
|
||||
touches guest memory.
|
||||
- **FD translation**: `FD` args must refer to a `ql_proxy_fd`, otherwise
|
||||
the forwarder raises (`qiling/os/posix/kernel_proxy/__init__.py:213-235`,
|
||||
`tests/test_kernel_proxy.py:397`); `returns_fd=True` wraps the result and
|
||||
stores it in the next free `ql.os.fd` slot
|
||||
(`qiling/os/posix/kernel_proxy/__init__.py:304-311`).
|
||||
- **Lifecycle**: the subprocess starts in `__init__`
|
||||
(`qiling/os/posix/kernel_proxy/__init__.py:73-95`) with the repository
|
||||
root prepended to `PYTHONPATH`; `stop()` terminates then kills
|
||||
(`:313-330`); `__del__` calls `stop()`. The forwarder closure holds a
|
||||
weakref so registered hooks do not keep the proxy alive
|
||||
(`tests/test_kernel_proxy.py:498`).
|
||||
- **Trust boundary**: forwarded syscalls execute on the host with the
|
||||
caller's privileges and no namespace isolation (`TODO.md:398-405` is a
|
||||
proposal). Only explicitly forwarded syscalls leave the sandbox; nothing
|
||||
is forwarded by default.
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/os/posix/kernel_proxy/__init__.py:54` - `KernelProxy(ql)` -
|
||||
`forward_syscall(name, returns_fd=False, arg_types=None)` (`:136`),
|
||||
`stop()` (`:313`).
|
||||
- `qiling/os/posix/kernel_proxy/__init__.py:168` - `_make_forwarder` - the
|
||||
CALL-hook closure; `_translate_args` (`:213`), `_collect_buffers`
|
||||
(`:237`), `_writeback_buffers` (`:259`), `_alloc_fd` (`:304`).
|
||||
- `qiling/os/posix/kernel_proxy/ipc.py:85` - `ProxyClient` - `syscall`
|
||||
(`:91`), `syscall_ex` (`:108`), `fd_read/fd_write/fd_close/fd_dup/
|
||||
fd_fcntl/fd_ioctl` (`:173-196`).
|
||||
- `qiling/os/posix/kernel_proxy/ipc.py:202` - `ProxyServer` -
|
||||
`recv_request` (`:208`) and the three `send_*_response` methods.
|
||||
- `qiling/os/posix/kernel_proxy/proxy.py:120` - `main()` - request loop;
|
||||
`raw_syscall` (`:41`), `raw_syscall_ex` (`:54`), `handle_fd_op` (`:84`).
|
||||
- `qiling/os/posix/kernel_proxy/proxy_fd.py:22` - `ql_proxy_fd` -
|
||||
`read/write/close/fileno/dup/fcntl/ioctl` matching the `ql_file` surface.
|
||||
- `qiling/os/posix/kernel_proxy/argtypes.py:46,55,64` - `PtrIn`, `PtrOut`,
|
||||
`PtrInOut` (size as int or callable over the raw args).
|
||||
|
||||
## Interactions
|
||||
|
||||
- Consumes [os-posix.md](os-posix.md): `QlOsPosix.set_syscall`,
|
||||
`ql.os.fd`, and the `map_syscall.py` tables; existing handlers such as
|
||||
`ql_syscall_read/write/close` operate on `ql_proxy_fd` unchanged.
|
||||
- Uses [core.md](core.md) `ql.mem.read/write` for buffer marshalling and
|
||||
`ql.log` for diagnostics.
|
||||
- Has no dependency from any other module; nothing in `qiling/` imports it
|
||||
(verified by grep), so it is strictly optional.
|
||||
- Examples in `TODO.md:86-109` show the intended user flow (epoll
|
||||
forwarding); `examples/` has no script yet.
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_kernel_proxy.py # Linux host; currently "Ran 21 tests … FAILED (errors=1)"
|
||||
```
|
||||
|
||||
- Expected after the Open Gaps fix: `Ran 21 tests … OK`. All cases except
|
||||
the pure unit tests `test_ipc_roundtrip` and `test_ptr_size_callable`
|
||||
build a `Qiling` on `examples/rootfs/x8664_linux/bin/x8664_hello` and
|
||||
invoke the registered hook directly.
|
||||
- Non-Linux hosts skip the whole class (`tests/test_kernel_proxy.py:17`).
|
||||
- No end-to-end test runs a guest binary that actually issues a forwarded
|
||||
syscall (`TODO.md:251-259` lists the intended validation).
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **Adding a forwarded-syscall feature** must not modify
|
||||
`qiling/os/posix/posix.py`; extend `argtypes.py` and the forwarder
|
||||
instead, and add a message type to `ipc.py` if the wire format grows.
|
||||
- **Wire-format changes** must update both `ProxyClient` and
|
||||
`ProxyServer`, and the format comment at
|
||||
`qiling/os/posix/kernel_proxy/ipc.py:40-53`.
|
||||
- **Security review** focuses on `_collect_buffers`/`_writeback_buffers`
|
||||
sizes (guest-controlled via `PtrOut(size=callable)`) and on which
|
||||
syscalls a harness chooses to forward; there is no allow-list.
|
||||
- Improvement candidate (accepted by the test's own intent): the
|
||||
`PtrOut` test should release proxy-side fds through `FD_OP CLOSE`, not
|
||||
host `os.close`. Success check: `test_ptr_out_writes_back_to_guest_memory`
|
||||
passes.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- `tests/test_kernel_proxy.py:449-451` closes proxy-side fds in the wrong
|
||||
process; the test errors on every Linux host.
|
||||
- Phase 1 networking (`ql_proxy_socket`, socket CALL hooks,
|
||||
`TODO.md:271-361`), Phase 2 epoll/poll/namespaces (`:363-427`), Phase 3
|
||||
real threading (`:429-525`), Phase 4 signals (`:527-567`), Phase 5 API
|
||||
and fallback (`:569-622`) are unimplemented proposals.
|
||||
- No `examples/` script and no documentation outside `TODO.md` and the
|
||||
module docstring.
|
||||
@@ -1,161 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# Loader — binary format loaders
|
||||
|
||||
## Goal
|
||||
|
||||
Map the target binary into emulated memory and prepare execution state:
|
||||
parse the format, load segments/sections and dependencies (ld.so, DLLs,
|
||||
dylibs), build stack/argv/auxv or OS structures (PEB/TEB, UEFI protocols),
|
||||
and set entry/exit points. One loader per format, selected by OS type. It
|
||||
must not implement syscalls or APIs. No roadmap milestone applies;
|
||||
maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — every format loader is exercised by its OS suite; the **How to
|
||||
Test** command proves the ELF path end-to-end (observed: `Ran 1 test … OK`).
|
||||
PE and Mach-O loading are proven only on their host-gated suites (see
|
||||
Open Gaps).
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/loader/loader.py` | Base `QlLoader` + `Image` record; `find_containing_image`, `get_image_by_name`, save/restore, abstract `run()` |
|
||||
| `qiling/loader/elf.py` | ELF for Linux/FreeBSD/QNX: segments, interpreter, stack/auxv, kernel-module (`.ko`) loading |
|
||||
| `qiling/loader/pe.py` | Windows PE/driver: DLL resolution, `QlPeCache`, PEB/TEB/LDR via `Process` |
|
||||
| `qiling/loader/pe_uefi.py` | UEFI DXE/SMM module loading, protocol installation, module chaining |
|
||||
| `qiling/loader/macho.py` + `macho_parser/` | Mach-O executables and kexts with an in-tree parser |
|
||||
| `qiling/loader/dos.py` | DOS COM/EXE/MBR (`ComParser`) |
|
||||
| `qiling/loader/mcu.py` | MCU firmware: Intel HEX (`IhexParser`), ELF, or raw bin; wires peripherals from the profile |
|
||||
| `qiling/loader/blob.py` | Raw blobs loaded at a fixed address |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Local patterns:
|
||||
|
||||
- Loader classes are named `QlLoader<MODULE>` so `select_loader` can derive
|
||||
them (`qiling/utils.py:297-320`); the OS→module table lives there.
|
||||
- Parsing uses `pyelftools`, `pefile`, and the in-tree `macho_parser`;
|
||||
segments are mapped through `ql.mem.map` with an `info` label naming the
|
||||
image so `QlMemoryManager.get_lib_base` can find it by basename
|
||||
(`qiling/os/memory.py:254`).
|
||||
- The base loader owns `images` and `skip_exit_check`
|
||||
(`qiling/loader/loader.py:26-27`); concrete loaders set `entry_point`,
|
||||
and the exit point lives on `ql.os`. The ELF loader also sets
|
||||
`is_driver` (`qiling/loader/elf.py:121`) and `elf_entry` (`:221`).
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- `loader.run()` executes at the end of `Qiling.__init__`
|
||||
(`qiling/core.py:195`); it may read `ql.os`, `ql.mem`, `ql.arch`, and the
|
||||
profile, but the OS run loop has not started. Loaders sit above the OS
|
||||
personalities in import order: they import OS structs, hooks, and API
|
||||
tables (`qiling/loader/elf.py:24-27`, `qiling/loader/pe.py:22-26`,
|
||||
`qiling/loader/pe_uefi.py:16-24`, `qiling/loader/macho.py:16-25`,
|
||||
`qiling/loader/dos.py:12`), and no OS module imports a loader.
|
||||
- **Shellcode path**: with `code=…` the ELF and PE loaders map a
|
||||
shellcode region from profile `[CODE]` values and return early
|
||||
(`qiling/loader/elf.py:72-83`, `qiling/os/os.py:90-94`).
|
||||
- **ELF**: `load_with_ld` (`qiling/loader/elf.py:149`) maps the binary
|
||||
and its interpreter and builds the initial stack (argv/envp/auxv per `AUXV`, `:32`);
|
||||
`skip_exit_check = (elf_entry != entry_point)` (`:393`) tells core whether
|
||||
the exit trap can be written immediately. `ET_REL` inputs are treated as
|
||||
kernel modules (`load_driver` `:596`, `lkm_dynlinker` `:440`).
|
||||
- **PE**: `Process` (`qiling/loader/pe.py:77`) owns PEB/TEB/LDR construction and
|
||||
`import_symbols` (address → `{dll, name, ordinal}`), the table the Windows
|
||||
API hook consults on every instruction (`qiling/os/windows/windows.py:172`).
|
||||
`QlPeCache` (`qiling/loader/pe.py:49`) persists parsed DLL data behind the `libcache`
|
||||
kwarg; invalidation is manual.
|
||||
- **UEFI**: `map_and_load` (`qiling/loader/pe_uefi.py:78`) and
|
||||
`execute_module` (`:227`) chain DXE/SMM modules, installing the loaded
|
||||
image protocol into a `DxeContext`/`SmmContext`
|
||||
(`qiling/os/uefi/context.py:80`, `:87`).
|
||||
- **MCU**: `run()` (`qiling/loader/mcu.py:135`) merges the YAML profile into
|
||||
`ql.env`, then creates peripherals and MMIO regions from every env entry
|
||||
(`:127`, `:130`) and hooks Unicorn interrupts to the arch exception
|
||||
handler; the loader is the only place hardware gets instantiated.
|
||||
- **Trust boundary**: every loader parses an untrusted image. Sizes and
|
||||
offsets from headers reach `ql.mem.map/write` and `struct` unpacking;
|
||||
keep bounds checks at the parse site (see root Review deviations).
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/loader/loader.py:21` - `QlLoader` - base; `Image` NamedTuple
|
||||
(`:15`), `run()` (`:62`), `save/restore` (`:51`/`:58`).
|
||||
- `qiling/loader/elf.py:68` - `QlLoaderELF` - `run` (`:72`),
|
||||
`load_with_ld` (`:149`), `load_driver` (`:596`).
|
||||
- `qiling/loader/pe.py:821` - `QlLoaderPE(QlLoader, Process)` - `run`
|
||||
(`:829`), `load` (`:887`); `Process.load_dll` (`:229`); `QlPeCache` (`:49`).
|
||||
- `qiling/loader/pe_uefi.py:26` - `QlLoaderPE_UEFI` - `run` (`:370`),
|
||||
`map_and_load` (`:78`), `execute_module` (`:227`).
|
||||
- `qiling/loader/macho.py:70` - `QlLoaderMACHO` - `run` (`:91`).
|
||||
- `qiling/loader/dos.py:36` - `QlLoaderDOS` - `run` (`:50`); `ComParser` (`:16`).
|
||||
- `qiling/loader/mcu.py:57` - `QlLoaderMCU` - `run` (`:135`); `IhexParser` (`:15`).
|
||||
- `qiling/loader/blob.py:10` - `QlLoaderBLOB` - `run` (`:16`).
|
||||
- `qiling/utils.py:297` - `select_loader(ostype, libcache)` - OS→loader
|
||||
mapping (LINUX/FREEBSD/QNX→elf, MACOS→macho, WINDOWS→pe, UEFI→pe_uefi,
|
||||
DOS→dos, MCU→mcu, BLOB→blob).
|
||||
|
||||
## Interactions
|
||||
|
||||
- Instantiated by [core.md](core.md) (`qiling/core.py:183`); `run()` fires
|
||||
at `qiling/core.py:195`.
|
||||
- Uses `ql.mem` ([os-base.md](os-base.md)) for mapping and `ql.arch`
|
||||
([arch.md](arch.md)) for initial register/stack state.
|
||||
- ELF: [os-posix.md](os-posix.md) `QlOsLinux.run` drives ld.so to
|
||||
`elf_entry`, then applies library patches (`qiling/os/linux/linux.py:173-186`);
|
||||
kernel modules use `qiling/os/linux/kernel_api/`.
|
||||
- PE/UEFI: [os-windows.md](os-windows.md) consumes `import_symbols`, heap,
|
||||
and handle managers; the UEFI loader populates `qiling/os/uefi/context.py`.
|
||||
- MCU: reads the YAML profile owned by [core.md](core.md) and calls
|
||||
`QlHwManager.setup_mmio/create` ([hw.md](hw.md)).
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 -m unittest test_elf.ELFTest.test_elf_linux_x8664 # pass = "OK", exit 0
|
||||
```
|
||||
|
||||
- Full ELF coverage: `python3 test_elf.py` (`tests/test_elf.py:30`,
|
||||
observed `Ran 53 tests … OK (skipped=2)`); kernel modules:
|
||||
`python3 test_elf_ko.py` after unzipping the rootkit fixture (see
|
||||
[os-posix.md](os-posix.md)).
|
||||
- DOS: `python3 test_dos.py`; MCU: `python3 test_mcu.py`; BLOB:
|
||||
`python3 -m unittest test_blob.BlobTest.test_uboot_arm`; UEFI:
|
||||
`python3 test_uefi.py` (all observed `OK`).
|
||||
- PE: `tests/test_pe.py` on a Windows host only; Mach-O: `tests/test_macho.py`
|
||||
on a macOS host only.
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **Format parsing changes**: keep header-derived sizes bounded before
|
||||
they reach `ql.mem`; add a regression sample under `examples/rootfs`
|
||||
(submodule) and a case in the owning suite.
|
||||
- **Changing entry/exit semantics** (`entry_point`, `elf_entry`,
|
||||
`skip_exit_check`) affects `Qiling.write_exit_trap`
|
||||
(`qiling/core.py:546`) and every OS `run()`; run `tests/test_elf.py`,
|
||||
`tests/test_uefi.py`, and `tests/test_mcu.py`.
|
||||
- **`import_symbols` layout** is a contract with
|
||||
`qiling/os/windows/windows.py:172-199`; change both together.
|
||||
- **Do not** call OS services from loaders beyond heap/handle allocation
|
||||
the Windows/UEFI loaders already use; syscalls and APIs belong to the OS
|
||||
docs.
|
||||
- Improvement candidate (proposal): version the `QlPeCache` on-disk format
|
||||
so stale caches are detected. Success check: a cache written by an older
|
||||
format is rejected and rebuilt in `tests/test_pe.py`.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- PE and Mach-O loading depend on host-collected libraries
|
||||
(`examples/scripts/dllscollector.bat`, `examples/scripts/dylibcollector.sh`);
|
||||
not runnable from a Linux checkout.
|
||||
- `QlPeCache` invalidation is manual (delete cache files).
|
||||
- macOS kext loading carries FIXMEs about IOKit providers
|
||||
(`qiling/os/macos/macos.py:79-100`).
|
||||
- `TODO.md:646-649` proposes replacing the label-string search in
|
||||
`QlMemoryManager.get_lib_base` (`qiling/os/memory.py:254`), which loaders
|
||||
rely on, with a structured mapping.
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# CPU Architecture and Calling Conventions
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/arch/, qiling/cc/, CPU models, register access, instruction mode or ABI slots.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns architecture adapters, the Unicorn instance, registers, assembler/
|
||||
disassembler access and function calling conventions. **In progress:**
|
||||
CPU-model, shellcode and RISC-V samples pass; this is not an exhaustive ISA
|
||||
or cross-platform compatibility certification.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [arch/arch.py](../../qiling/arch/arch.py): `QlArch` | CPU, pointer width, stack, context interface |
|
||||
| [arch/register.py](../../qiling/arch/register.py): `QlRegisterManager` | Name-to-Unicorn registers and PC/SP aliases |
|
||||
| [arch/](../../qiling/arch/), [arch/models.py](../../qiling/arch/models.py) | Concrete x86/ARM/MIPS/RISC-V/PPC/Cortex-M adapters and CPU enums |
|
||||
| [cc/__init__.py](../../qiling/cc/__init__.py): `QlCC`, `QlCommonBaseCC` | Register/stack slots, result, return address, frame unwind |
|
||||
| [arch/cortex_m.py](../../qiling/arch/cortex_m.py): `QlArchCORTEX_M` | Exception/vector state and task-aware Unicorn |
|
||||
| [test_cpu_models.py](../../tests/test_cpu_models.py), [test_riscv.py](../../tests/test_riscv.py), [test_shellcode.py](../../tests/test_shellcode.py) | Model selection and instruction/ABI samples |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
The [root baseline](../../ARCHITECTURE.md#code-conventions) applies. Adapter
|
||||
names match `select_arch` class construction (`QlArch` + enum name);
|
||||
`cc` intentionally retains public camelCase methods. Keep register maps,
|
||||
CPU enums and architecture properties together. Python/native-engine
|
||||
versions come from the root manifest, not individual adapters.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- `pointersize = bits // 8`; stack operations use architecture-sized memory
|
||||
access and update SP only for push/pop. Register aliases must match GDB
|
||||
and saved context expectations (`arch.py`, `register.py`).
|
||||
- Calling-convention argument **slots** need not equal argument indices:
|
||||
multiword arguments, shadow space, register exhaustion, stack return
|
||||
addresses and unwind rules differ (`cc/__init__.py`, `cc/intel.py`).
|
||||
- Endianness, CPU mode and Thumb PC handling affect execution and instruction
|
||||
decoding. Preserve `effective_pc` and Cortex-M exception return behavior
|
||||
when changing ARM mode; inspect the core Thumb workaround too.
|
||||
- Concrete adapters instantiate engine/assembler/disassembler objects;
|
||||
supported enums do not imply support for every OS/CPU combination.
|
||||
Cortex-M uses `MultiTaskUnicorn` and hardware interrupt state.
|
||||
- CPU behavior belongs here; OS-specific selector/TLS policy belongs in the
|
||||
OS personality even when architecture helpers implement the mechanism.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [core](core.md) for factory changes, [OS base](os-base.md) for fcall
|
||||
marshalling, [POSIX](posix.md) for syscall ABIs (separate from function ABIs),
|
||||
[debugger](debugger.md) for register wire order/step behavior, and
|
||||
[bare-metal](baremetal.md)/[hardware](hardware.md) for Cortex-M interrupts.
|
||||
Guest integers and host ctypes layouts must retain target width/endian;
|
||||
never substitute host pointer size for a guest ABI.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| New arch/model | Concrete adapter, model enum, core selection | Update core routes/support claims; CPU-model + instruction test |
|
||||
| Register/mode change | Maps, PC/SP, save/restore, disassembly | Read debugger; shellcode and relevant target regression |
|
||||
| Function ABI | Slot width, return register/address, unwind | Read OS base plus API consumer; execute target function test |
|
||||
| Cortex-M exception | Vector state, task/context switching | Read firmware partners; MCU snapshot/interrupt checks |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_cpu_models test_riscv test_shellcode`
|
||||
with root dependencies and relevant rootfs samples. All selected modules
|
||||
passed during rebuild; CPU-model suite has 7 tests. Their scope is selected
|
||||
models/instructions and guest execution, not instruction-set conformance.
|
||||
For PPC changes use `test_elf.ELFTest.test_elf_linux_powerpc`; for ABI changes
|
||||
run the consuming OS API/syscall regression as well.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
There is no independent exhaustive calling-convention suite. RISC-V/PPC
|
||||
Linux thread classes are unset in `qiling/os/linux/linux.py`; do not infer
|
||||
thread support from CPU execution. Native engine upgrades need root-level
|
||||
compatibility review and representative architecture tests.
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Bare-metal Execution
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/os/mcu/, qiling/os/blob/, qiling/extensions/multitask.py or firmware run/stop/count/timeout behavior.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns MCU/raw-BLOB run loops and the task-aware Unicorn scheduler used by
|
||||
MCU emulation. **In progress:** MCU snapshot/USART and ARM U-Boot samples
|
||||
pass; raw BLOB and complete fast-mode timing remain unverified. Peripheral
|
||||
models and image mapping are separate owners.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [mcu/mcu.py](../../qiling/os/mcu/mcu.py): `QlOsMcu`, `MCUTask` | Instruction stepping, fast mode and peripheral scheduling |
|
||||
| [blob/blob.py](../../qiling/os/blob/blob.py): `QlOsBlob` | Raw code execution, function-call helpers and optional heap |
|
||||
| [extensions/multitask.py](../../qiling/extensions/multitask.py): `MultiTaskUnicorn`, `UnicornTask` | Task contexts, stop/resume, gevent/native-engine coordination |
|
||||
| [test_mcu.py](../../tests/test_mcu.py), [test_blob.py](../../tests/test_blob.py) | Firmware stepping, I/O, snapshots and raw code |
|
||||
| [examples/src/blob/](../../examples/src/blob/), [examples/mcu/](../../examples/mcu/) | Raw fixture build and device/firmware examples |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use [root conventions](../../ARCHITECTURE.md#code-conventions); preserve
|
||||
legacy `runable` and task method spellings. `extensions/multitask.py` is
|
||||
owned here despite its source prefix. Nonblocking task work yields through
|
||||
gevent; don't block the cooperative loop with ordinary sleep. Raw fixture
|
||||
Makefile uses `arm-none-eabi-*`, Cortex-A7/Thumb, freestanding/no standard
|
||||
library flags; firmware builds are not part of the Python package build.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Normal MCU mode executes one instruction then `hw.step()`. `count` is
|
||||
instruction count; timeout is explicitly unsupported in this mode.
|
||||
`effective_pc` is used where available (`QlOsMcu.run`).
|
||||
- Fast mode creates an `MCUTask`; `count` means scheduling rounds. Scheduler
|
||||
timeout code uses milliseconds (`MultiTaskUnicorn._timeout_main`), unlike
|
||||
core's microsecond API. Do not silently equate these units.
|
||||
- `MCUTask.on_interrupted` advances peripherals on successful slices; CPU
|
||||
errors log and request stop instead of modeled HardFault delivery.
|
||||
Task locking serializes engine execution; it is not parallel guest CPU
|
||||
emulation. Preserve nested/context/stop behavior when changing scheduling.
|
||||
- BLOB uses profile code/load/entry fields, optional profile heap and
|
||||
architecture-specific fcall. Default end is load address + code length;
|
||||
caller begin/end overrides affect execution (`QlOsBlob.run`).
|
||||
- MCU and BLOB share a firmware-facing run boundary but have different
|
||||
timing/hardware semantics; BLOB does not automatically model peripherals.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [hardware](hardware.md) for step/interrupt timing, [arch](arch.md) for
|
||||
Cortex-M context/effective PC and [loaders](loaders.md) for env/vector/image
|
||||
initialization. Read [core](core.md) for `stop`/count/timeout API changes;
|
||||
[OS base](os-base.md) for BLOB heap/fcall. Do not move firmware scheduler
|
||||
semantics into unrelated POSIX/Windows run loops.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| Step/count/timeout | Both normal and fast paths, task stop/error state | MCU test with explicit mode/units; hardware/arch partners |
|
||||
| Snapshot/resume | CPU task context, memory and peripheral state | MCU snapshot regression; core/loaders if saved state changes |
|
||||
| Raw code/heap | BLOB profile, entry/end, fixture build | U-Boot/raw BLOB case; OS-base/loader contract |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_mcu.MCUTest.test_mcu_snapshot_stm32f411 test_mcu.MCUTest.test_mcu_usart_input_stm32f411 test_blob.BlobTest.test_uboot_arm`
|
||||
passed 3 cases during rebuild. Root dependencies and matching STM32/U-Boot
|
||||
fixtures are required. Run `python test_mcu.py` for broader device/timing
|
||||
changes and `python test_blob.py` for raw behavior; the full suites are not
|
||||
certified by the selected cases.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
`test_blob.BlobTest.test_blob_raw` fails because rootfs lacks
|
||||
`blob/example_raw.bin`; [fixture source](../../examples/src/blob/Makefile)
|
||||
exists but was not built or copied into the submodule during this rebuild.
|
||||
Fast-mode faults currently stop cleanly instead of delivering hardware
|
||||
HardFault. Timeout units differ by execution path; full scheduler timing
|
||||
and MCU exception fidelity remain unverified.
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# CLI, Packaging and Development Tooling
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/cli.py, qltool, qltui.py, manifests/lock, Dockerfile, CI/test drivers or fixture preparation.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns CLI/TUI entry behavior, package/resources, build and test wiring, and
|
||||
fixture preparation mechanics. **In progress:** selected checkout CLI tests,
|
||||
fresh installed-wheel smoke and sdist/wheel build pass; full interactive
|
||||
TUI, container and host CI matrices remain unverified. Guest behavior in
|
||||
examples/tests belongs to its runtime owner.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [qiling/cli.py](../../qiling/cli.py): `run`, `handle_run`, `handle_code` | Argument parsing, input conversion, emulation and output options |
|
||||
| [qltool](../../qltool), [qltui.py](../../qltui.py): `get_data` | Checkout launcher and interactive front end |
|
||||
| [pyproject.toml](../../pyproject.toml), [poetry.lock](../../poetry.lock) | Package/dependencies/extras, console entry and reproducible resolution |
|
||||
| [.github/workflows/](../../.github/workflows/), [Dockerfile](../../Dockerfile), [.editorconfig](../../.editorconfig) | Build/test environment and source-format baseline |
|
||||
| [test_qltool.py](../../tests/test_qltool.py): `InstalledQltool_Test`; [test_onlinux.sh](../../tests/test_onlinux.sh), [test_pe.bat](../../tests/test_pe.bat), [test_macho.sh](../../tests/test_macho.sh) | CLI contracts, installed package smoke and platform test drivers; `test_perf.py`/`view_perf_results.py` profile existing suites |
|
||||
| [.gitmodules](../../.gitmodules), [examples/scripts/](../../examples/scripts/), [examples/src/](../../examples/src/), [README.md](../../README.md), [docs/](../../docs/) | External fixtures, collectors/build inputs and existing user-facing documentation |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use the [root baseline](../../ARCHITECTURE.md#code-conventions).
|
||||
`argparse` actions convert enum/environment inputs and build `Qiling` keyword
|
||||
arguments. Preserve option names and exit status. TUI uses questionary/fx/
|
||||
termcolor; testing its import is not testing its prompts. Keep build metadata
|
||||
in the manifest, resolve its lock with Poetry, and retain package license
|
||||
and resource attribution. Documentation here contains coding context only.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Checkout `qltool` and installed console entry call `qiling.cli:run`.
|
||||
Wheel must include `qltui.py`, profiles and debugger XML; checkout-only
|
||||
relative imports/paths must not be required by installed execution.
|
||||
- CLI/TUI environment input uses `pickle.load` when the argument names an
|
||||
existing file, and `literal_eval` otherwise (`cli.py:__arg_env`,
|
||||
`qltui.py:env_arg`). Pickle files require trusted producers; neither path
|
||||
establishes a dictionary-schema validation guarantee.
|
||||
- Package version comes from distribution metadata with an uninstalled
|
||||
fallback in `qiling/__init__.py`; manifest, lock and CLI tests must agree.
|
||||
- Native engine pins and platform-marked extras are compatibility inputs.
|
||||
Container uses Python 3.13 slim with build tools in a separate stage;
|
||||
container success is a separate check from wheel validation.
|
||||
- Test paths commonly assume `tests/` as cwd. Rootfs is an external Git
|
||||
submodule; fixtures may need system libraries or extracted archives.
|
||||
Keep user changes and fixture revision separate from source edits.
|
||||
- C/C++/assembly fixtures have local Makefiles and cross-toolchains. Change
|
||||
fixture source with its behavioral owner; do not silently replace a
|
||||
binary/submodule pin to hide a failing assertion. Ignored `jexamples/`
|
||||
is local scratch, not a declared shipped package or required CI suite.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [core](core.md) for options/run configuration, [debugger](debugger.md)
|
||||
for debugger flags and [extensions](extensions.md) for coverage/reporting.
|
||||
Fixture preparation is tooling ownership; test assertions and guest payload
|
||||
semantics route to their OS/CPU/device owner. Workflow publishing steps are
|
||||
outside architecture's coding scope; package build/test contracts remain here.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| CLI/TUI option | Parser, argument mapping, exit/result | Checkout CLI case; affected runtime owner; installed smoke for public entry |
|
||||
| Dependency/resource/build | Manifest/lock, wheel contents, workflow | Metadata/build/installed checks below; relevant engine owner |
|
||||
| CI/fixture setup | Platform driver, rootfs/version, collectors | Run applicable host suite or record missing prerequisite; behavioral owner |
|
||||
|
||||
## Verification
|
||||
|
||||
**Shared test-resource rule (preserved):** serialize suites sharing fixed
|
||||
localhost ports or fixtures; do not run aggregates beside their constituents.
|
||||
`test_posix.py` imports ELF/CLI cases, ELF uses port 8000, debugger tests use
|
||||
fixed ports including 9999, and `test_onlinux.sh` runs scripts sequentially.
|
||||
|
||||
From root after setup, with Poetry 2, build and Twine installed:
|
||||
|
||||
```sh
|
||||
poetry check --lock
|
||||
python -m build
|
||||
python -m twine check --strict dist/*
|
||||
```
|
||||
|
||||
Install the resulting wheel in a fresh virtual environment, then from root:
|
||||
`python -I tests/test_qltool.py InstalledQltool_Test -v`. Three tests run
|
||||
CLI/version/profile behavior and TUI import outside the checkout. For
|
||||
checkout behavior from `tests/`: `python test_qltool.py`; rootfs samples
|
||||
are required for argument, JSON, filter and coverage cases.
|
||||
|
||||
Rebuild evidence: sdist plus wheel-from-sdist build passed, artifact metadata
|
||||
checks and `poetry check --lock` passed, 8 selected CLI tests passed, and 3 installed-wheel cases
|
||||
passed in a second fresh environment. See [root verification](../../ARCHITECTURE.md#verification)
|
||||
for common setup and Linux aggregate. No configured lint/type-check command exists.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
Full Windows/macOS fixtures, raw BLOB sample and complete host matrix were
|
||||
not available/verified. `.github/workflows/build-ci.yml` uses
|
||||
`matrix.contrainer` instead of `matrix.container`, so the intended container
|
||||
branch lacks reliable coverage. Interactive TUI, fixture rebuilds and Docker
|
||||
image build were not exercised. Build success alone does not certify guest
|
||||
execution or all optional integrations.
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Core Runtime
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/core*.py, utils.py, const.py, host.py, log.py, exception.py, __init__.py or profile composition.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns the public facade, component selection, hooks, packing, logging and
|
||||
snapshot orchestration. **In progress:** runtime is implemented; sampled
|
||||
execution passes, but hook lifecycle and complete state recovery are not
|
||||
exhaustively verified. CPU execution and OS semantics belong to partners.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [core.py](../../qiling/core.py): `Qiling` | Construction, run/stop, patch, save/restore and convenience APIs |
|
||||
| [core_hooks.py](../../qiling/core_hooks.py): `QlCoreHooks`; `core_hooks_types.py` | Unicorn hook fan-out, callback contracts, handles/removal |
|
||||
| [core_struct.py](../../qiling/core_struct.py): `QlCoreStructs` | Architecture-sized packing/unpacking |
|
||||
| [utils.py](../../qiling/utils.py): `select_arch`, `profile_setup`; [profiles/](../../qiling/profiles/) | Dynamic component factories, detection, configuration merge |
|
||||
| [const.py](../../qiling/const.py), [exception.py](../../qiling/exception.py), [log.py](../../qiling/log.py), [host.py](../../qiling/host.py), [__init__.py](../../qiling/__init__.py) | Enums, errors, logging, host detection, public import/version |
|
||||
| [test_shellcode.py](../../tests/test_shellcode.py), [test_elf.py](../../tests/test_elf.py), [test_mcu.py](../../tests/test_mcu.py), [test_edl.py](../../tests/test_edl.py) | Lifecycle, hooks, patching and snapshots through real execution |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use the [root baseline](../../ARCHITECTURE.md#code-conventions). Public facade
|
||||
properties and hook callbacks are compatibility surfaces; preserve legacy
|
||||
spellings. **Observed:** type hints coexist with wildcard imports; no local
|
||||
formatter/type-checker override exists. Errors inside callbacks are stored
|
||||
by `hookcallback` and re-raised by `emu_start`, not silently consumed.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Construction validates target/rootfs, selects architecture before packing
|
||||
and hooks, then creates profile/loader/memory/OS/hardware before
|
||||
`loader.run()`. Construction maps the image; callers can hook before `run()`.
|
||||
- `run(begin,end,timeout,count)` configures OS execution, patches and stop
|
||||
guards. Timeout is microseconds at the core/Unicorn interface; MCU semantics
|
||||
differ. `stop()` delegates for multithread or bare-metal execution.
|
||||
- Factories derive modules/class names from enums with explicit format/arch
|
||||
mappings. A new enum alone is insufficient (`utils.py`). Hook callback
|
||||
signatures and `QL_HOOK_BLOCK` behavior must survive dispatcher changes.
|
||||
- Profiles use defaults plus OS/user overrides; MCU uses YAML-derived maps
|
||||
(`profile_setup`). Parsing belongs here; the consumer owns each key.
|
||||
- `save` includes only selected components. `restore` applies only included
|
||||
keys; memory precedes registers/CPU-dependent state. Pickled snapshot files
|
||||
require trusted producers; component support varies (`core.py`).
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [arch](arch.md) when CPU/register contracts change, [OS base](os-base.md)
|
||||
for memory/call contracts, [loaders](loaders.md) for construction/image state,
|
||||
[bare-metal](baremetal.md) for scheduling, and [debugger](debugger.md) for
|
||||
stop/attach behavior. Core composes implementations through factories. **Required:** do not add
|
||||
module-level imports of concrete arch/OS/loader classes to core; use the
|
||||
existing factories (annotation-only imports belong under `TYPE_CHECKING`).
|
||||
|
||||
**Preserved project dependency rule:** keep dependencies toward underlying
|
||||
services; do not expand existing reverse edges incidentally. Source exceptions
|
||||
are Cortex-M and MCU importing `extensions.multitask`, lazy r2 use in
|
||||
`arch/utils.py`, and OS memory annotations in `arch/x86_utils.py`. Loaders
|
||||
legitimately use OS structs/hooks, GDB uses Linux procfs, and CLI uses
|
||||
coverage/report. This is a design constraint with documented exceptions,
|
||||
not a claim that the tree is an acyclic import graph.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| Constructor/profile/factory option | Initialization ordering, defaults, enum maps, consumer | Update this owner and affected runtime owner; shellcode + consumer test |
|
||||
| Hooks, patching, stop behavior | Dispatch/removal and internal exception propagation | Read debugger if resume changes; ELF hook and debugger regressions |
|
||||
| Snapshot contents | Save/restore pairs for every changed component | Read affected state owners; ELF/MCU snapshot tests, trusted-input contract |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`, after root setup: `python -m unittest test_shellcode`;
|
||||
`python -m unittest test_elf.ELFTest.test_memory_search` checks memory-facing
|
||||
facade behavior. Both passed during rebuild on Linux/Python 3.13/Unicorn 2.1.3.
|
||||
For snapshot changes use matching cases in `test_elf.py` and
|
||||
`test_mcu.MCUTest.test_mcu_snapshot_stm32f411` (the MCU case also passed).
|
||||
Fixtures are relative to `tests/`. Passing these does not certify every
|
||||
hook kind, debugger combination or snapshot component.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
`core.py:emu_start` retains a documented ARM Thumb workaround. Snapshot
|
||||
completeness depends on each component's implementation; base OS state is
|
||||
empty. A dedicated exhaustive hook lifecycle suite was not found. Broader
|
||||
refactoring ideas in [TODO.md](../../TODO.md) are proposals, not implemented
|
||||
behavior; keep changes scoped to an evidenced problem.
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Debuggers
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/debugger/, GDB packets/register XML, Qdb commands, stepping, breakpoints or replay.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns GDB remote protocol and Qdb's interactive/script debugger, register
|
||||
rendering, branch prediction and snapshots. **In progress:** selected GDB
|
||||
signal/interrupt regressions and one Qdb script pass; complete protocol,
|
||||
architecture and replay coverage is unverified.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [debugger.py](../../qiling/debugger/debugger.py): `QlDebugger` | Shared debugger interface |
|
||||
| [gdb/gdb.py](../../qiling/debugger/gdb/gdb.py): `QlGdb`; [gdb/utils.py](../../qiling/debugger/gdb/utils.py) | Packet transport, command handling, stop/continue and memory/files |
|
||||
| [gdb/xmlregs.py](../../qiling/debugger/gdb/xmlregs.py): `QlGdbFeatures`; [gdb/xml/](../../qiling/debugger/gdb/xml/) | Target register descriptions and wire ordering |
|
||||
| [qdb/qdb.py](../../qiling/debugger/qdb/qdb.py): `QlQdb`; [qdb/utils.py](../../qiling/debugger/qdb/utils.py): `SnapshotManager` | Commands, breakpoints, snapshot/replay and target helpers |
|
||||
| [qdb/](../../qiling/debugger/qdb/) | Architecture-specific renderers, predictors, disassembly and command helpers |
|
||||
| [test_debugger.py](../../tests/test_debugger.py), [test_qdb.py](../../tests/test_qdb.py), [qdb_scripts/](../../tests/qdb_scripts/), [test_windows_debugger.py](../../tests/test_windows_debugger.py) | Protocol regression clients and debugger scripts |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use [root conventions](../../ARCHITECTURE.md#code-conventions).
|
||||
Qdb preserves `cmd.Cmd`'s `do_*` convention and target-specific helpers.
|
||||
GDB protocol register order/width/encoding comes from target XML, not
|
||||
Python attribute order. Keep XML licensing/attribution and target meaning;
|
||||
no repository regeneration pipeline is established for these assets.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Core configures debugger before OS execution. GDB defaults to loopback
|
||||
port 9999; attach/detach addresses depend on file versus shellcode/MCU
|
||||
paths. Thumb attach addresses are aligned (`QlGdb.__init__`).
|
||||
- GDB reads/writes emulated registers/memory and exposes supported file
|
||||
requests. Preserve packet checksums/framing, target endian/register width
|
||||
and stop replies. Unicorn faults map to signals; single-step success
|
||||
reports SIGTRAP (`UC_ERROR_SIGMAP`, packet handlers).
|
||||
- Async Ctrl-C and `vCont` actions must stop/resume the right emulation
|
||||
state. Transport handling must not introduce unbounded guest/remote sizes
|
||||
reaching host allocation or file access; debugger access is a host boundary.
|
||||
- Qdb breakpoint handling stops the emulator and flushes translated blocks
|
||||
to avoid resuming past a breakpoint. Replay relies on available saved
|
||||
state; it cannot promise reversal of host filesystem/socket effects.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [core](core.md) for run/stop/hook lifecycle, [arch](arch.md) for register
|
||||
and step semantics, [loaders](loaders.md) for image entry/address changes,
|
||||
[OS base](os-base.md) for memory and [POSIX](posix.md) for GDB procfs/file
|
||||
behavior. Remote register descriptions are consumed by external clients;
|
||||
maintain protocol compatibility rather than exposing internal refactors.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| GDB packet/register | Parser, reply encoding, XML maps | Protocol regression; arch owner for layout |
|
||||
| Step/interrupt/resume | Stop state, fault signals, async input | SIGTRAP/vCont/Ctrl-C cases; core owner |
|
||||
| Qdb command/replay | Command helper, predictor, snapshots | Matching Qdb script; state owner for replay change |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_debugger.DebuggerTest.test_gdbdebug_stepi_reports_sigtrap test_debugger.DebuggerTest.test_gdbdebug_vcont_signal_actions test_debugger.DebuggerTest.test_gdbdebug_async_interrupt`
|
||||
and `python -m unittest test_qdb.DebuggerTest.test_qdb_arm_hello` passed
|
||||
3 GDB and 1 Qdb cases. Loopback sockets, free test ports, engine dependencies
|
||||
and target fixtures are required. For broader changes run
|
||||
`python test_debugger.py` and `python test_qdb.py`; Windows debugger checks
|
||||
need Windows fixtures. Follow [shared test-resource rules](cli-build.md#verification)
|
||||
for port conflicts. Tests use in-process clients, not every external GDB.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
Full protocol/architecture support and reverse-execution side effects are
|
||||
unverified. XML declarations do not prove every target is runnable. The
|
||||
Qdb README and scripts provide command examples, not a comprehensive test
|
||||
oracle; preserve package resource inclusion when changing XML assets.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# DOS and BIOS Interrupts
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/os/dos/, DOS/BIOS interrupt leaves, terminal behavior or DOS file/disk semantics.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns DOS/BIOS interrupt behavior, DOS handles, terminal state and run-loop
|
||||
semantics. **In progress:** the direct DOS sample test passes; EXE/MBR,
|
||||
interactive display and full interrupt coverage are unverified. DOS image
|
||||
loading and shared disk/file abstractions are separate owners.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [dos.py](../../qiling/os/dos/dos.py): `QlOsDos`, `Flags` | Interrupt interception, flags, terminal lifetime and execution |
|
||||
| [interrupts/__init__.py](../../qiling/os/dos/interrupts/__init__.py) | Interrupt-number handler registry |
|
||||
| [interrupts/](../../qiling/os/dos/interrupts/) | BIOS/DOS leaf implementations, console, disk and file services |
|
||||
| [utils.py](../../qiling/os/dos/utils.py) | DOS-specific helpers |
|
||||
| [test_dos.py](../../tests/test_dos.py), [test_dos_exe.py](../../tests/test_dos_exe.py), [profiles/dos.ql](../../qiling/profiles/dos.ql) | Program/format samples and consumed DOS settings |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
[Root conventions](../../ARCHITECTURE.md#code-conventions) suffice. Preserve
|
||||
interrupt identifiers and guest register names, including AH/AL leaves;
|
||||
return status may live in carry/zero flags and registers rather than a
|
||||
Python return value. Handler names mirror interrupt numbers. Curses is a
|
||||
host dependency; Windows uses the manifest's platform-marked curses package.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Each interrupt builds `(intno, AH)` for user CALL/ENTER/EXIT lookup.
|
||||
Built-ins are selected by interrupt number and handle their own leaves.
|
||||
ENTER/EXIT callbacks receive `ql`; unsupported interrupt numbers raise
|
||||
`NotImplementedError` (`QlOsDos.hook_syscall`).
|
||||
- `run()` uses the loader's `start_address` unless explicitly overridden,
|
||||
applies exit overrides and catches/re-raises Unicorn errors after logging.
|
||||
The implementation's binary path is distinct from shellcode behavior.
|
||||
- DOS file handles belong to this personality; segment:offset arithmetic
|
||||
uses real-mode address conventions. Do not treat these handles or far
|
||||
pointers as host descriptors/addresses.
|
||||
- `KERNEL.version` and `ticks_per_second` are read from the DOS profile.
|
||||
Curses terminal setup requires an appropriate terminal; destructor cleanup
|
||||
restores it when initialized. Preserve screen/keyboard state lifetime.
|
||||
- Disk and filesystem handlers reach shared host-backed objects; changes to
|
||||
guest offsets/lengths must keep host access within intended resources.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [loaders](loaders.md) for COM/EXE/MBR layout and initial registers,
|
||||
[arch](arch.md) for 8086 registers/segmentation, and [OS base](os-base.md)
|
||||
for `disk.py`, paths and file objects. Windows/UEFI API dispatch rules do
|
||||
not apply to DOS interrupt leaves.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| Interrupt/leaf | Registry, AH handling, flags and override callbacks | DOS regression for the changed register/flag contract |
|
||||
| Program startup | Start address, segment registers, loader format | Read loaders/arch; matching EXE/COM test |
|
||||
| Console/file/disk | Curses lifetime, handle/offset validation | Read OS base for shared changes; relevant device/fixture test |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_dos` passed one program test during
|
||||
rebuild with the existing DOS rootfs fixture. Use `python test_dos_exe.py`
|
||||
for EXE-loader changes; it was not part of the verified subset. Terminal
|
||||
cases need usable curses/TTY support. A successful sample does not certify
|
||||
all BIOS services or error leaves.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
Interrupt coverage is incremental; unsupported leaves must remain explicit.
|
||||
Interactive console, EXE/MBR and complete DOS-state recovery were not
|
||||
verified. File/disk behavior requires both guest-register and host-resource
|
||||
checks; don't infer containment solely from emulated execution.
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Instrumentation and Integrations
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/extensions/ except mcu/ and multitask.py, fuzzing harnesses, coverage/trace formats, sanitizers or analysis adapters.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns coverage/history, trace output, AFL integration, heap sanitizer,
|
||||
reports, pipes, r2/IDA adapters and Windows SDK stub tooling.
|
||||
**In progress:** history and UEFI sanitizer tests pass; optional fuzz/RE/IDA
|
||||
integrations and all output formats remain unverified. Firmware extension
|
||||
paths are explicitly owned by hardware/bare-metal modules.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [coverage/utils.py](../../qiling/extensions/coverage/utils.py): `CoverageFactory`, `collect_coverage`; [coverage/formats/](../../qiling/extensions/coverage/formats/) | Format discovery, hook lifecycle, history and output |
|
||||
| [tracing/utils.py](../../qiling/extensions/tracing/utils.py), [tracing/formats/](../../qiling/extensions/tracing/formats/), [trace.py](../../qiling/extensions/trace.py) | Trace collectors and log/history tracing |
|
||||
| [afl/afl.py](../../qiling/extensions/afl/afl.py): `ql_afl_fuzz`, `ql_afl_fuzz_custom`; [afl/](../../qiling/extensions/afl/) | Input/crash callback bridge and engine integrations |
|
||||
| [sanitizers/heap.py](../../qiling/extensions/sanitizers/heap.py): `QlSanitizedMemoryHeap` | Allocation wrapping and memory-error hooks |
|
||||
| [report/](../../qiling/extensions/report/), [pipe.py](../../qiling/extensions/pipe.py), [r2/](../../qiling/extensions/r2/), [idaplugin/](../../qiling/extensions/idaplugin/), [winsdkapi.py](../../qiling/extensions/winsdkapi.py) | Reporting, I/O adapters, optional analysis and stub generation |
|
||||
| [test_history.py](../../tests/test_history.py), [test_uefi.py](../../tests/test_uefi.py), [test_r2.py](../../tests/test_r2.py), [examples/fuzzing/](../../examples/fuzzing/) | History/sanitizer/integration evidence and harness sources |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use [root conventions](../../ARCHITECTURE.md#code-conventions); optional
|
||||
packages use manifest extras `fuzz` and `RE`. Format classes expose
|
||||
`FORMAT_NAME` and activate/deactivate/dump methods; imports register subclasses
|
||||
with factories. No common C standard is declared for fuzz harnesses; inspect
|
||||
the local `fuzz.c`/build instructions. Generated SDK stubs require review;
|
||||
they are not automatically complete Windows API behavior.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Coverage context manager always deactivates hooks and dumps output in
|
||||
`finally`. Tracing likewise deactivates and writes when an output path is
|
||||
supplied. New formats must register before the factory is constructed.
|
||||
- History derives executable-image ranges from memory metadata; labels and
|
||||
image identities matter to filtering (`coverage/formats/history.py`).
|
||||
- AFL delegates input placement and crash validation to user callbacks,
|
||||
configures engine exit addresses and supports persistent iterations.
|
||||
The harness owns resetting all state/input that survives an iteration;
|
||||
callbacks must not confuse guest addresses with host buffers (`afl.py`).
|
||||
- Sanitized heap wraps allocations with memory hooks/canaries. Caller
|
||||
failure callbacks determine response; inspect actual hook coverage before
|
||||
claiming detection of every memory error.
|
||||
- Reports and traces serialize observed emulator state; they are not full
|
||||
snapshots. r2/IDA depend on separately available runtimes and may use
|
||||
concrete emulator types. Host file output and native integrations retain
|
||||
their own resource lifetimes.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [core](core.md) for hooks/lifecycle, [OS base](os-base.md) for heap and
|
||||
mapping state, [arch](arch.md) for register/decoder output, and
|
||||
[CLI/build](cli-build.md) when flags or output choices change. Read
|
||||
[Windows](windows.md) when modifying SDK-generated hook contracts.
|
||||
Firmware-only exceptions use the root firmware branch, not this owner.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| Coverage/trace format | Base class, exports/factory, cleanup/output | History/CLI output regression; root CLI owner for flags |
|
||||
| Fuzzing lifecycle | Input/crash callbacks, exits, persistent state | Harness-specific run with correct extra/runtime; core/arch as affected |
|
||||
| Sanitizer/report/adapter | Hook lifetime, serialized fields, external API | Direct UEFI/history or adapter test; consumer owner |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_history` passed 4 tests;
|
||||
`python test_uefi.py` passed 2 direct integration tests including sanitizer
|
||||
behavior. For coverage output run
|
||||
`python -m unittest test_qltool.Qltool_Test.test_qltool_coverage` with UEFI
|
||||
fixtures; that coverage case also passed during rebuild. For r2 changes use `python test_r2.py` after root setup with `RE`;
|
||||
this optional check was not run. AFL/IDA require their external runtimes and
|
||||
matching harnesses; a base-package import does not validate them.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
No complete fuzz/IDA/trace/report regression suite is configured. Existing
|
||||
coverage/tracing factory similarity is observed, not a mandate to refactor
|
||||
unrelated formats. Native optional integration compatibility and all
|
||||
persistent-state reset contracts remain unverified.
|
||||
@@ -0,0 +1,86 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Hardware Models and MMIO
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/hw/, qiling/extensions/mcu/, chip environment maps, peripheral registers or external devices.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns hardware entity creation, MMIO dispatch, peripheral register/state
|
||||
models, interrupt controllers and external-device connections. **In progress:**
|
||||
selected STM32 USART and snapshot tests pass; device coverage is partial,
|
||||
not a cycle-accurate silicon model.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [hw/hw.py](../../qiling/hw/hw.py): `QlHwManager`, `QlPripheralHandler` | Entity/region registry, lookup, MMIO fallback and step/save/restore |
|
||||
| [hw/peripheral.py](../../qiling/hw/peripheral.py): `QlPeripheral`, `QlPeripheralUtils` | ctypes registers, raw access, monitoring and read/write interception |
|
||||
| [hw/connectivity.py](../../qiling/hw/connectivity.py), [hw/external_device/](../../qiling/hw/external_device/) | Peripheral I/O connections and external device models |
|
||||
| [hw/](../../qiling/hw/), [hw/const/](../../qiling/hw/const/) | Device-class implementations, register masks and controller behavior |
|
||||
| [extensions/mcu/](../../qiling/extensions/mcu/) | Chip-specific env dictionaries consumed by loaders/hardware |
|
||||
| [test_mcu.py](../../tests/test_mcu.py), [examples/mcu/](../../examples/mcu/) | Firmware-driven device and state checks |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use the [root baseline](../../ARCHITECTURE.md#code-conventions).
|
||||
Observed device classes mirror chip/peripheral names and nested `Type`
|
||||
ctypes register layouts; masks live in `hw/const`. `create` resolves class
|
||||
names through `qiling.hw` exports, so add new exports and env entries together.
|
||||
Register tables are source code, not a documented generated artifact;
|
||||
no common regeneration command is established. Preserve local register fidelity.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Env entries provide `type`, `base`, `size` and/or `struct`, `kwargs`.
|
||||
Loader creates memory/MMIO/core devices; hardware creation records
|
||||
entity-relative regions rebased to guest addresses (`hw.py`, `loader/mcu.py`).
|
||||
- MMIO handler converts address to peripheral offset. Unclaimed regions
|
||||
use a little-endian backing bytearray; missing peripheral implementations
|
||||
log a warning. Neither behavior proves an unmodeled device works.
|
||||
- Register layout, reset values, access width, write-one-to-clear effects
|
||||
and interrupt flags are device contracts. `raw_read/raw_write` use ctypes
|
||||
memory copies: validate changed offsets/widths against register storage
|
||||
before reaching host memory.
|
||||
- `step()` calls devices with step methods. Ordering/frequency is inherited
|
||||
from the MCU run loop, not physical elapsed time. Save/restore must keep
|
||||
device state, mappings and callbacks coherent with CPU/memory snapshots.
|
||||
- Monitoring honors ENTER/CALL/EXIT hooks; replacement/read return behavior
|
||||
differs from ordinary observation. Preserve hook identity/removal and
|
||||
side-effect order when extending register access.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [bare-metal](baremetal.md) for scheduling, [arch](arch.md) for interrupt
|
||||
entry/return, [loaders](loaders.md) for env initialization and [OS base](os-base.md)
|
||||
for MMIO mapping. `extensions/mcu/` belongs here, not to optional analysis
|
||||
extensions. External devices may expose host I/O; treat those resources
|
||||
according to the concrete connection implementation.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| New peripheral/chip | Register struct, masks, exports, env shape | Device-specific firmware assertion; loader if env schema changes |
|
||||
| MMIO/access hook | Offset/width bounds, fallback storage, side effects | Read OS base; register read/write and hook lifecycle evidence |
|
||||
| Interrupt/timing/snapshot | Device step/state, controller and CPU | Read bare-metal/arch; MCU snapshot and relevant firmware test |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_mcu.MCUTest.test_mcu_snapshot_stm32f411 test_mcu.MCUTest.test_mcu_usart_input_stm32f411`
|
||||
passed 2 cases with current STM32 fixtures. These exercise snapshot state
|
||||
and UART input, not every register or device family. For broader changes
|
||||
use `python test_mcu.py` and matching cases/examples; external devices need
|
||||
their own environment. Inspect guest output/register assertions for real pass evidence.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
`QlHwManager.find` performs linear device-region lookup (its source TODO
|
||||
notes potential caching); no performance claim is made. Missing devices
|
||||
may fall back to storage. A dedicated exhaustive peripheral suite and full
|
||||
cycle timing model were not found; accepted scope remains demand-driven.
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Linux Kernel Proxy
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/os/posix/kernel_proxy/, forwarded syscalls, proxy descriptors or the hybrid-kernel design in TODO.md.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns opt-in forwarding of selected Linux syscalls through a helper process,
|
||||
argument/buffer transport and proxy-owned descriptors. **In progress:**
|
||||
Phase 0 code exists and 21 tests passed, but actual guest dispatch and
|
||||
cross-ABI buffer semantics have unresolved gaps. Later phases in
|
||||
[TODO.md](../../TODO.md) remain designs, not implemented capability.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [__init__.py](../../qiling/os/posix/kernel_proxy/__init__.py): `KernelProxy` | Helper lifecycle, hook registration, syscall-number/argument translation |
|
||||
| [argtypes.py](../../qiling/os/posix/kernel_proxy/argtypes.py): `PtrIn`, `PtrOut`, `PtrInOut` | Integer/callable buffer-size descriptors; INT/FD tags |
|
||||
| [ipc.py](../../qiling/os/posix/kernel_proxy/ipc.py): `ProxyClient` | Length-prefixed binary request/reply protocol |
|
||||
| [proxy.py](../../qiling/os/posix/kernel_proxy/proxy.py), [proxy_fd.py](../../qiling/os/posix/kernel_proxy/proxy_fd.py) | libc syscall execution and guest wrapper around helper-owned fds |
|
||||
| [test_kernel_proxy.py](../../tests/test_kernel_proxy.py), [TODO.md](../../TODO.md) | IPC, fd, marshalling tests and established phased design |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
[Root conventions](../../ARCHITECTURE.md#code-conventions) apply. Descriptor
|
||||
classes use dataclasses/type hints; `ql_proxy_fd` follows existing fd-object
|
||||
naming. **Required project-specific constraint:** integrate through
|
||||
`set_syscall` CALL hooks and existing fd interfaces only; do not modify
|
||||
`load_syscall`, existing handlers or the emulation loop for this feature
|
||||
(`TODO.md`, “What Changes vs What Doesn't”).
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Constructor requires a Linux host and starts a Python helper connected by
|
||||
Unix socketpair. `forward_syscall` explicitly registers selected names;
|
||||
no automatic forwarding or dispatcher fallback is implemented.
|
||||
- Guest and host syscall numbers are resolved separately. Host support is
|
||||
the explicit architecture map in `_load_host_syscall_table`; descriptor
|
||||
bytes are not automatically converted between guest/host structure ABIs.
|
||||
- INT passes integers; FD requires a proxy-owned guest fd. Pointer tags copy
|
||||
bytes to/from helper buffers; `returns_fd=True` allocates a guest wrapper.
|
||||
Raw host pointers and ordinary guest fds are not interchangeable with
|
||||
translated buffers/proxy fds. Size callbacks currently see translated args.
|
||||
- Wire messages include SYSCALL, SYSCALL_EX and FD_OP. Short/closed reads
|
||||
raise `QlProxyConnectionError`; sizes are encoded in protocol fields but
|
||||
no comprehensive resource quota is established (`ipc.py`, `proxy.py`).
|
||||
- Helper calls are real host operations with helper-process credentials,
|
||||
not a separate security sandbox. Call `stop()` in caller `finally` cleanup;
|
||||
`KernelProxy` has no context-manager protocol.
|
||||
Do not promise automatic cleanup merely because `ql.run()` returned.
|
||||
Proxy descriptors cannot be closed by parent-process fd numbers.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [POSIX](posix.md) for CALL signature introspection, errno and fd
|
||||
consumers; [OS base](os-base.md) for memory/file abstractions. The helper
|
||||
owns actual descriptors; the emulator owns wrappers and guest memory.
|
||||
Buffer sizes, raw syscalls and IPC payloads are host trust/resource boundaries.
|
||||
No additional runtime package is required beyond the existing manifest.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| Forwarded call/argument type | Hook signature, guest/host maps, buffer layout | Read POSIX; direct marshalling plus actual guest trap regression |
|
||||
| IPC/fd operation | Both protocol endpoints, proxy wrapper/lifetime | Round-trip/error tests and POSIX fd consumer |
|
||||
| Phase advancement | Current code versus TODO design | Keep this status current; prove named end-to-end behavior |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_kernel_proxy` passed 21 tests on
|
||||
Linux x86_64/Python 3.13 with current hello fixtures. Helpers invoke real
|
||||
Linux syscalls. These tests exercise IPC and direct hook invocation; they
|
||||
do not establish complete guest-trap integration or cross-architecture
|
||||
struct conversion. Run an actual emulated syscall case for dispatch changes.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
`KernelProxy._make_forwarder` returns `(ql, *args)`, while POSIX dispatch
|
||||
counts only ordinary positional parameters; tests calling the hook directly
|
||||
can miss missing guest arguments. `test_ptr_out_writes_back_to_guest_memory`
|
||||
closes helper-owned fd numbers with parent `os.close`; its passing result
|
||||
can depend on coincident parent fd allocation and does not prove cleanup.
|
||||
Buffer quotas, ABI translation and concurrent IPC access need evidence;
|
||||
none should be implied by Phase 0 passing tests. Preserve scope and record
|
||||
required fixes before claiming later TODO milestones.
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Executable Loaders
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/loader/, image parsing, relocation, interpreter/library resolution or initial guest state.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns executable parsing/mapping and initial image, stack, entry and import
|
||||
state for ELF, PE, Mach-O, UEFI PE, DOS, MCU and BLOB. **In progress:** Linux,
|
||||
UEFI and selected firmware loading were exercised; Windows fixtures and
|
||||
macOS coverage remain incomplete. Loaders do not own ongoing OS semantics.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [loader.py](../../qiling/loader/loader.py): `QlLoader`, `Image` | Common image ranges, lookup and saved metadata |
|
||||
| [elf.py](../../qiling/loader/elf.py): `QlLoaderELF` | Segments, interpreter, auxv, initial stack, kernel relocation |
|
||||
| [pe.py](../../qiling/loader/pe.py): `QlLoaderPE` | PE/DLL imports, exports, relocations, process structures/cache |
|
||||
| [pe_uefi.py](../../qiling/loader/pe_uefi.py): `QlLoaderPE_UEFI` | DXE/SMM contexts, service tables and module queue |
|
||||
| [macho.py](../../qiling/loader/macho.py), [macho_parser/](../../qiling/loader/macho_parser/), [dos.py](../../qiling/loader/dos.py), [mcu.py](../../qiling/loader/mcu.py), [blob.py](../../qiling/loader/blob.py) | Other parsers and initialization paths |
|
||||
| [tests/](../../tests/) `test_elf*.py`, `test_pe*.py`, `test_macho*.py`, `test_uefi.py`, `test_dos*.py`, `test_mcu.py`, `test_blob.py` | Format-specific integration evidence |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use the [root baseline](../../ARCHITECTURE.md#code-conventions); factory class
|
||||
names are format-derived and include legacy uppercase/camelCase forms.
|
||||
Keep format-specific logic local. External parser libraries are declared
|
||||
in the root manifest; MCU Intel HEX parsing and Mach-O helpers are local.
|
||||
Fixture C/assembly belongs to its behavioral OS owner; preparation to tooling.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Core creates memory and OS before invoking `run`; loaders may initialize
|
||||
OS structures and hardware. Preserve entry/exit addresses, image ranges
|
||||
and metadata consumed by patches, reports and debuggers (`QlLoader`).
|
||||
- ELF maps load segments/interpreter and creates guest stack/auxv;
|
||||
ELF driver loading uses a separate relocation/import path (`elf.py`).
|
||||
- PE maps DLLs and records `import_symbols` consumed by Windows dispatch.
|
||||
Optional disk caches use pickle (`QlPeCache`); cache contents are trusted
|
||||
host artifacts, not safe arbitrary guest inputs.
|
||||
- UEFI loading creates contexts/heaps/protocols and queues images. Its
|
||||
saved state includes selected queue/event members and heap state;
|
||||
this does not establish complete firmware snapshot support.
|
||||
- MCU profile/env declares memory/MMIO/core devices; reset writes firmware
|
||||
and initializes CPU context. Raw `.bin` input needs its mapping address;
|
||||
HEX parsing implements selected record kinds (`mcu.py`).
|
||||
- Header-derived addresses/sizes/offsets cross allocation and mapping
|
||||
boundaries. Validate changed arithmetic against file bounds and guest
|
||||
address width; preserve page alignment and section permissions.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [OS base](os-base.md) for mapping/heap/path contracts; read the changed
|
||||
format's [POSIX](posix.md), [Windows](windows.md), [UEFI](uefi.md),
|
||||
[DOS](dos.md) or [bare-metal](baremetal.md) owner when initial state changes.
|
||||
Read [hardware](hardware.md) for MCU env/MMIO changes, [core](core.md) for
|
||||
factory/profile changes, [debugger](debugger.md) for image/entry reporting.
|
||||
OS imports in loaders are intentional construction dependencies.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| Parser/relocation | Format loader, segment/header arithmetic | Format regression plus its OS owner; malformed-input checks for changed validation |
|
||||
| Stack/import/entry layout | Guest-width structs, profile keys, symbols | Consumer owner and relevant executable/API test |
|
||||
| Cache/snapshot | Save/restore, cache format and consumers | Core and relevant OS state owner; round-trip evidence |
|
||||
| Firmware mapping | MCU env/reset or UEFI contexts | Read corresponding firmware/service owners; execution test |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_elf.ELFTest.test_elf_linux_x8664 test_blob.BlobTest.test_uboot_arm`
|
||||
and `python test_uefi.py`. These passed during rebuild with local fixtures.
|
||||
Use direct UEFI script execution: its test bodies are guarded by `__main__`.
|
||||
For changed formats, use the corresponding test files in Code Map; Windows
|
||||
and macOS need their system-library fixtures. Passing ELF does not verify PE.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
Raw BLOB test requires missing `examples/rootfs/blob/example_raw.bin` in the
|
||||
examined rootfs; Windows hello tests lacked `Windows/registry` directories.
|
||||
Mach-O and full PE/kernel-module suites were not verified. Intel HEX handling
|
||||
is partial; no general malformed-image corpus was found. Treat these as
|
||||
verification gaps, not supported-input guarantees.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Shared Operating System Services
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/os/*.py, guest memory/heap, filesystem paths/mappings, shared file wrappers or API marshalling.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns shared memory/heap, path virtualization, mapped files/disks, typed
|
||||
function calls, guest structs, standard streams, statistics and OS base
|
||||
interfaces. **In progress:** path/struct and integration samples pass;
|
||||
complete memory/heap/OS-state coverage remains unverified.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [os.py](../../qiling/os/os.py): `QlOs` | Shared streams, profile, API hooks, resolver and call protocol |
|
||||
| [memory.py](../../qiling/os/memory.py): `QlMemoryManager`, `QlMemoryHeap` | Guest mapping metadata, read/write, MMIO and heap chunks |
|
||||
| [path.py](../../qiling/os/path.py): `QlOsPath` | Guest path dialect, symlinks, rootfs/host conversion |
|
||||
| [mapper.py](../../qiling/os/mapper.py): `QlFsMapper`; [filestruct.py](../../qiling/os/filestruct.py), [disk.py](../../qiling/os/disk.py) | Host/file-object bindings, persistent streams and disk helpers |
|
||||
| [fcall.py](../../qiling/os/fcall.py), [struct.py](../../qiling/os/struct.py), [utils.py](../../qiling/os/utils.py), [stats.py](../../qiling/os/stats.py), [thread.py](../../qiling/os/thread.py), [const.py](../../qiling/os/const.py) | ABI-driven calls, ctypes guest layouts, strings, statistics, shared abstractions |
|
||||
| [test_pathutils.py](../../tests/test_pathutils.py), [test_struct.py](../../tests/test_struct.py), [test_elf.py](../../tests/test_elf.py) | Path/struct assertions and memory integration |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
[Root conventions](../../ARCHITECTURE.md#code-conventions) apply. Keep target
|
||||
width and endian explicit in memory/ctypes helpers; use `ql.mem` for guest
|
||||
pointers. Function-call helpers preserve existing camelCase methods.
|
||||
Observed duck typing lets descriptors and mapped objects implement only
|
||||
needed file operations; do not impose unrelated concrete host types.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Mapping metadata uses sorted non-overlapping half-open ranges, while
|
||||
Unicorn's region end is inclusive. Keep metadata and engine mappings in
|
||||
sync across map/unmap/protect/restore (`memory.py`). MMIO callbacks receive
|
||||
offsets and access widths; heap allocation bookkeeping is separate.
|
||||
Mapping labels are compatibility data: `get_lib_base` strips boxed prefixes
|
||||
and matches basenames; core library patches depend on this lookup.
|
||||
- Unmapped filesystem opens resolve a virtual path and check containment
|
||||
under rootfs (`QlFsMapper.__open_new`). Explicit mappings deliberately
|
||||
bind host paths, factories or objects and bypass this ordinary path.
|
||||
Preserve guest Windows/POSIX dialect handling and symlink tests.
|
||||
- API prototypes preserve parameter order. Resolvers translate strings,
|
||||
wide strings and GUID pointers; calling convention owns slots/stack
|
||||
unwind. ENTER/EXIT hooks may alter parameters/results (`fcall.py`, `os.py`).
|
||||
`QlOs.call` must not write PC after `QL_STATE.STOPPED`; Unicorn can otherwise
|
||||
resume despite `emu_stop`. Preserve this guard when changing call returns.
|
||||
- Standard streams may lack `fileno`; base OS adapts embedded interpreters
|
||||
and persistent file wrappers avoid closing host standard descriptors.
|
||||
- Base `QlOs.save()` returns empty state and `restore()` is a stub. A
|
||||
component snapshot must not be assumed to capture host files/resources.
|
||||
Keep personality-specific behavior in `QlOs` subclasses; extend shared base
|
||||
behavior only when it is a shared contract.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [arch](arch.md) for ABI/width changes, the consuming OS owner for paths,
|
||||
errno/API semantics, [loaders](loaders.md) for mapped layout/labels, [core](core.md) for library-patch lookup, and
|
||||
[hardware](hardware.md) for MMIO. Read [kernel proxy](kernel-proxy.md) when
|
||||
changing descriptor operations that proxy-owned objects must implement.
|
||||
Host opens, ctypes and string reads are trust boundaries; guest values need
|
||||
validation at the operation that consumes them.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| Paths/mapping | Dialect conversion, symlink containment, explicit bindings | Path tests plus consuming OS filesystem case |
|
||||
| Memory/heap/MMIO | Range convention, engine metadata, allocator state | ELF memory search; MCU snapshot for MMIO state |
|
||||
| Struct/call/strings | Target packing, slots, resolvers, unwind | Struct tests; arch plus API consumer tests |
|
||||
| Descriptor/disk | Wrapper methods, lifetime/dup and host errors | POSIX or DOS owner; proxy owner if fd polymorphism changes |
|
||||
|
||||
## Verification
|
||||
|
||||
Use root's shared-primitives command from `tests/`: it ran 21 cases: 16 passed,
|
||||
5 explicitly skipped (host-specific path behavior and wchar cases).
|
||||
`python -m unittest test_elf.ELFTest.test_memory_search` also passed.
|
||||
Root dependencies and ELF rootfs fixture are needed for integration.
|
||||
These do not prove memory-manager/heap coverage; changed allocator or call
|
||||
semantics need their own existing consumer regression extended as appropriate.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
Four wchar struct cases are marked broken/skipped in `test_struct.py`.
|
||||
`os/utils.py` string readers do not establish a global maximum guest-string
|
||||
length. Base OS save/restore is partial by design. There is no dedicated
|
||||
exhaustive memory/heap/fcall test suite; proposals in `TODO.md` must retain
|
||||
behavior and gain focused evidence before implementation claims.
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# POSIX Personalities and Syscalls
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/os/posix/ except kernel_proxy/, or Linux, FreeBSD, macOS and QNX personalities.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns syscall ABIs/dispatch, common POSIX handlers, Linux processes/threads,
|
||||
futex/procfs and FreeBSD/macOS/QNX personalities, including driver API
|
||||
support. **In progress:** representative Linux, RISC-V and QNX execution
|
||||
passes; full syscall, threading, network and macOS behavior is unverified.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [posix/posix.py](../../qiling/os/posix/posix.py): `QlOsPosix`, `QlFileDes` | Hooks, argument dispatch, identity/network configuration and fd table |
|
||||
| [posix/syscall/](../../qiling/os/posix/syscall/), [posix/syscall/abi/](../../qiling/os/posix/syscall/abi/) | Common handlers, syscall-register conventions and return encodings |
|
||||
| [linux/linux.py](../../qiling/os/linux/linux.py): `QlOsLinux`; [linux/thread.py](../../qiling/os/linux/thread.py), [linux/futex.py](../../qiling/os/linux/futex.py), [linux/procfs.py](../../qiling/os/linux/procfs.py) | Linux traps, cooperative guest threads, futex state and virtual procfs |
|
||||
| [freebsd/](../../qiling/os/freebsd/), [macos/](../../qiling/os/macos/), [qnx/](../../qiling/os/qnx/) | OS-specific maps, syscalls, structures, kernel APIs and messages |
|
||||
| [test_posix.py](../../tests/test_posix.py), [test_elf.py](../../tests/test_elf.py), [test_elf_multithread.py](../../tests/test_elf_multithread.py), [tests/](../../tests/) `test_elf_ko.py`, `test_android.py`, `test_qnx.py`, `test_macho*.py`, `test_tendaac15_httpd.py` | Syscalls, programs, threads and platform integration |
|
||||
| [examples/src/](../../examples/src/) `linux/`, `freebsd/`, `macos/`, `qnx/`; [profiles/](../../qiling/profiles/) | Guest fixture sources and consumed OS settings |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use [root conventions](../../ARCHITECTURE.md#code-conventions). Common
|
||||
handlers are `ql_syscall_<name>(ql, ...)`; ordinary positional parameter
|
||||
names/counts matter because dispatcher introspection reads guest arguments.
|
||||
OS-specific handlers precede common POSIX implementations. Keep per-OS
|
||||
layouts and error semantics; do not unify superficially similar handlers.
|
||||
Fixture Makefiles specify cross-compilers locally; no common C standard is declared.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Dispatch maps syscall ID to name, resolves CALL/ENTER/EXIT overrides by
|
||||
name or number, then OS-specific/common handler. ENTER may replace args;
|
||||
EXIT may replace result; `None` leaves the return register untouched
|
||||
(`QlOsPosix.load_syscall`). Missing handlers warn and raise only when
|
||||
`debug_stop` requests it; automatic host forwarding is absent.
|
||||
- Syscall ABI owns register decoding and negative errno encoding. Function
|
||||
calling conventions are separate. Hook signatures must match dispatch
|
||||
introspection, particularly wrappers using only `*args`.
|
||||
- Fds 0–2 track standard streams; descriptors are polymorphic objects.
|
||||
Closing/duplicating/replacing them must preserve ownership and errors.
|
||||
- Linux guest threads use gevent/cooperative Unicorn contexts; this does
|
||||
not imply full native Linux scheduling or signal delivery. `sched.py`
|
||||
can call host fork; socket handlers operate on host sockets.
|
||||
- Linux `/proc/self` mappings are installed only when not overridden.
|
||||
Profile identity and network keys affect observable guest behavior.
|
||||
macOS kernel and QNX message interfaces retain their own state/layouts.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [OS base](os-base.md) for memory/paths/fds, [arch](arch.md) for syscall
|
||||
register contracts, [loaders](loaders.md) for process entry/auxv/driver
|
||||
imports, and [kernel proxy](kernel-proxy.md) for forwarded descriptors.
|
||||
Proxy integration must use existing hooks and fd polymorphism; it does not
|
||||
own dispatch. Filesystem/socket/fork operations expose host resources.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| New/fixed syscall | Mapping table, ABI, OS override/common handler | Update this contract; matching `test_posix`/ELF test and affected OS |
|
||||
| Thread/futex/signal | Thread contexts, yielding, futex waiters | `test_elf_multithread.py`; mark unsupported semantics explicitly |
|
||||
| FD/path/socket | Host object lifetime, errno, guest structure layouts | OS-base checks and syscall regression; proxy checks if shared operations change |
|
||||
| macOS/QNX/kernel API | Local maps/structs, loader imports | Platform-specific test and loader owner |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_elf.ELFTest.test_elf_linux_x8664 test_riscv test_qnx`
|
||||
passed during rebuild. It checks sample Linux/RISC-V/QNX flows, not every
|
||||
handler. For POSIX changes run `python test_posix.py` (also imports ELF,
|
||||
RISC-V and CLI suites), and the relevant thread/kernel/network/platform
|
||||
script. Follow [shared test-resource rules](cli-build.md#verification) when
|
||||
running these suites. Root Linux aggregate is the broader CI gate; its fixtures, extracted
|
||||
kernel sample and network/host requirements apply. Full aggregate was not run.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
macOS CI is commented out. Some thread classes are unset for RISC-V/PPC;
|
||||
signal syscall state is not proof of full delivery. Existing kernel-module
|
||||
archives need fixture preparation. `TODO.md` hybrid phases are plans; proxy
|
||||
Phase 0 implementation is documented separately. Current representative
|
||||
checks do not establish kernel fidelity or host isolation.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# UEFI Services
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/os/uefi/, DXE/SMM service tables, protocols, events or UEFI module execution.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns UEFI service/API behavior, protocol definitions, DXE/SMM contexts and
|
||||
firmware events/callbacks. **In progress:** two direct UEFI integration tests
|
||||
pass; they do not establish complete firmware or SMM conformance. The UEFI
|
||||
PE loader owns mapping, service-table installation and module queues.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [uefi.py](../../qiling/os/uefi/uefi.py): `QlOsUefi` | Calls, module callbacks, execution and selected OS state |
|
||||
| [fncc.py](../../qiling/os/uefi/fncc.py): `dxeapi` | Typed service hooks and interception |
|
||||
| [context.py](../../qiling/os/uefi/context.py): `DxeContext`, `SmmContext` | Context-owned heaps, protocols and configuration state |
|
||||
| [bs.py](../../qiling/os/uefi/bs.py), [rt.py](../../qiling/os/uefi/rt.py), [ds.py](../../qiling/os/uefi/ds.py), [smst.py](../../qiling/os/uefi/smst.py), [smm.py](../../qiling/os/uefi/smm.py), [protocols/](../../qiling/os/uefi/protocols/) | Service tables, SMM environment and protocol hooks |
|
||||
| [UefiSpec.py](../../qiling/os/uefi/UefiSpec.py), [ProcessorBind.py](../../qiling/os/uefi/ProcessorBind.py), [guids.csv](../../qiling/os/uefi/guids.csv), [__init__.py](../../qiling/os/uefi/__init__.py) | ABI types and GUID names |
|
||||
| [test_uefi.py](../../tests/test_uefi.py), [qiling/profiles/uefi.ql](../../qiling/profiles/uefi.ql) | API interception and sanitized heap integration |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
The [root baseline](../../ARCHITECTURE.md#code-conventions) applies; retain
|
||||
specification casing and ordered ctypes fields. Service handlers use
|
||||
`dxeapi` and explicit parameter types. `type64.py` is reference-only and
|
||||
raises on import; do not use it as runtime definitions. No reproducible
|
||||
regeneration pipeline for that reference file is established; runtime
|
||||
layout changes belong in the actual spec/bindings/protocol definitions.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- Function calls use cdecl for 32-bit and ms64 for 64-bit state. Service
|
||||
hooks resolve by function name; `set_api` adapts user targets, and the
|
||||
decorator applies CALL/ENTER/EXIT hooks (`uefi.py`, `fncc.py`).
|
||||
- Context protocols and heaps are initialized with the PE/UEFI loader.
|
||||
Preserve DXE versus SMM context selection, GUID identity, native-width
|
||||
fields and protocol callback signatures.
|
||||
- Module-enter/module-exit callbacks can thwart subsequent execution;
|
||||
queue/event/notification state and saved heap state live partly in the
|
||||
loader. `QlOsUefi.save()` only extends base state with its entry point.
|
||||
- Guest structure addresses and buffer lengths must be validated before
|
||||
shared memory/heap access. A handler stub returning success is not proof
|
||||
of specification conformance.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [loaders](loaders.md) for image queues/context installation/snapshots,
|
||||
[OS base](os-base.md) for struct/fcall/heap changes, [arch](arch.md) for ABI
|
||||
width, and [extensions](extensions.md) when sanitizer/coverage hooks change.
|
||||
UEFI shares PE parsing infrastructure, not Windows API semantics.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| Service/protocol | Descriptor layout, decorator, GUID and context | Direct UEFI script plus target service assertions |
|
||||
| Event/module scheduling | Queue notifications, loader context callbacks | Read loader; module order/interception regression |
|
||||
| SMM or snapshot | OS/loader state split and context memory | Read loader and OS base; explicit restoration/state evidence |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: **`python test_uefi.py`** with root dependencies and
|
||||
`examples/rootfs/x8664_efi` fixtures. Both tests passed during rebuild;
|
||||
they exercise interception and sanitized heap behavior. Do not substitute
|
||||
`python -m unittest test_uefi`: actual bodies are guarded by `__main__`.
|
||||
For package/coverage output changes also use the CLI coverage test through
|
||||
[extensions](extensions.md).
|
||||
|
||||
## Known Gaps
|
||||
|
||||
Coverage is narrow relative to the service/protocol surface. Full SMM,
|
||||
32-bit firmware and complete state restoration were not verified.
|
||||
`type64.py` is an intentionally unusable reference artifact, not a second
|
||||
supported runtime ABI. Keep unimplemented service behavior explicit.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
eatmycode_version: "2.0.0"
|
||||
---
|
||||
|
||||
# Windows API Emulation
|
||||
|
||||
Owner: [Project architecture](../../ARCHITECTURE.md)
|
||||
|
||||
Read when: changing qiling/os/windows/, Win32/NT APIs, handles, registry, fibers or guest thread/process state.
|
||||
|
||||
## Responsibility and Status
|
||||
|
||||
Owns Win32/NT API emulation, process environment, handles, registry,
|
||||
clipboard, fibers and guest thread management. **In progress:** source is
|
||||
implemented but PE hello verification is blocked by missing registry
|
||||
fixtures; comprehensive Windows behavior is unverified. PE parsing belongs
|
||||
to loaders; UEFI and DOS have separate owners.
|
||||
|
||||
## Code Map
|
||||
|
||||
| Path / symbol | Role |
|
||||
| --- | --- |
|
||||
| [windows.py](../../qiling/os/windows/windows.py): `QlOsWindows` | Process setup, API dispatch and execution |
|
||||
| [fncc.py](../../qiling/os/windows/fncc.py): `winsdkapi` | API prototype/CC decorator and ENTER/EXIT hooks |
|
||||
| [dlls/](../../qiling/os/windows/dlls/), [api.py](../../qiling/os/windows/api.py) | DLL hook implementations and exported namespace |
|
||||
| [handle.py](../../qiling/os/windows/handle.py): `HandleManager`; [registry.py](../../qiling/os/windows/registry.py): `RegistryManager` | Guest object IDs and hive/overlay state |
|
||||
| [thread.py](../../qiling/os/windows/thread.py), [fiber.py](../../qiling/os/windows/fiber.py), [structs.py](../../qiling/os/windows/structs.py), [wdk_const.py](../../qiling/os/windows/wdk_const.py) | Scheduling, guest layouts and driver constants |
|
||||
| [test_pe.py](../../tests/test_pe.py), [test_pe_sys.py](../../tests/test_pe_sys.py), [test_peshellcode.py](../../tests/test_peshellcode.py), [tests/](../../tests/) `test_windows*.py`; [examples/src/windows/](../../examples/src/windows/) | User/kernel/shellcode and C/C++ fixtures |
|
||||
|
||||
## Local Conventions
|
||||
|
||||
Use the [root baseline](../../ARCHITECTURE.md#code-conventions).
|
||||
`hook_<API>` functions use `@winsdkapi(cc=..., params=...)`, then receive
|
||||
`(ql, address, params)` via the wrapper. Preserve Windows API capitalization,
|
||||
A/W variants and target-width typedefs. Extend the owning DLL file/API export
|
||||
path; the SDK stub helper is owned by extensions and is not an authoritative
|
||||
implementation. No universal Windows fixture compiler version is declared.
|
||||
|
||||
## Contracts and Invariants
|
||||
|
||||
- `hook_winapi` recognizes addresses in loader `import_symbols`, resolves
|
||||
names/ordinals and user CALL hooks, then built-in hooks. Missing APIs warn
|
||||
or fail under `debug_stop`; implementation exceptions become
|
||||
`QlErrorSyscallError` (`windows.py`).
|
||||
- `winsdkapi` selects calling convention and delegates decoding, ENTER/EXIT
|
||||
hooks and unwind to shared OS/fcall. Stack cleanup and return width must
|
||||
match x86 versus x64; API success/error conventions remain API-specific.
|
||||
- Handles are guest identities around host/Python objects, not native
|
||||
Windows handles. Registry hives come from rootfs; the manager persists
|
||||
its overlay after normal `run()` completion. Maintain host-file/path
|
||||
boundaries when adding APIs.
|
||||
- Process PEB/TEB, segment bases and thread contexts couple this module to
|
||||
PE initialization and architecture utilities. Base OS save/restore does
|
||||
not capture the entire Windows environment.
|
||||
|
||||
## Dependencies and Boundaries
|
||||
|
||||
Read [loaders](loaders.md) when import symbols or process structures change,
|
||||
[OS base](os-base.md) for pointers, paths and fcall, [arch](arch.md) for
|
||||
calling conventions and segments, [debugger](debugger.md) for breakpoints.
|
||||
Win32 hook behavior should stay within this owner; do not move it into
|
||||
shared dispatch just to support one DLL or driver.
|
||||
|
||||
## Change Guide
|
||||
|
||||
| Change trigger | Inspect / extend | Required docs / checks |
|
||||
| --- | --- | --- |
|
||||
| DLL/API handler | Decorator prototype, export, A/W and error handling | Matching PE/API test; OS base if marshalling changes |
|
||||
| Handle/registry/thread state | Ownership/lifetime, guest structs, persistence | Existing Windows regression + PE initialization case |
|
||||
| Driver API or shellcode | WDK structures, loader path and CC | `test_pe_sys.py` or `test_peshellcode.py`; loader/arch owners |
|
||||
|
||||
## Verification
|
||||
|
||||
From `tests/`: `python -m unittest test_pe.PETest.test_pe_win_x86_hello test_pe.PETest.test_pe_win_x8664_hello`.
|
||||
Both were attempted and failed before emulation because the matching rootfs
|
||||
`Windows/registry` directory is absent (`registry.py`). These are blocked
|
||||
checks, not passing evidence. With complete system libraries/registry
|
||||
fixtures, use the matching Windows scripts and `test_pe.bat` under its
|
||||
Windows CI prerequisites. See [tooling](cli-build.md) for fixture preparation.
|
||||
|
||||
## Known Gaps
|
||||
|
||||
Full Windows verification requires collected system libraries and registry
|
||||
fixtures; Linux checks do not substitute for the Windows CI job. API
|
||||
coverage is demand-driven and includes stubs; inspect the target hook before
|
||||
claiming support. Complete OS snapshots and all guest threading semantics
|
||||
remain unverified.
|
||||
@@ -1,124 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# OS bare-metal — MCU firmware and raw blobs
|
||||
|
||||
## Goal
|
||||
|
||||
Execute code with no operating system. MCU mode runs microcontroller
|
||||
firmware (STM32, GD32V, NXP, Atmel SAM, BES) with peripheral emulation and
|
||||
interrupt delivery on a cooperative multitasking wrapper around Unicorn;
|
||||
BLOB mode runs raw binaries (e.g. u-boot) at a fixed address with no OS
|
||||
services. Owns the two run loops and the multitask Unicorn subclass; the
|
||||
peripherals themselves belong to [hw.md](hw.md). No roadmap milestone
|
||||
applies; maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — MCU covered by `tests/test_mcu.py` (observed: `Ran 18 tests … OK`,
|
||||
STM32F1/F4, GD32VF103, SAM3X8E firmware); BLOB by
|
||||
`test_blob.BlobTest.test_uboot_arm` (`OK`). `tests/test_edl.py` is not
|
||||
BLOB coverage: its ELF is sniffed as Linux and runs under `QlOsLinux`
|
||||
([os-posix.md](os-posix.md)).
|
||||
`test_blob.BlobTest.test_blob_raw` errors on the pinned rootfs submodule
|
||||
(see Open Gaps).
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/os/mcu/mcu.py` | `QlOsMcu` run loop (step mode and fast mode) + `MCUTask` |
|
||||
| `qiling/os/blob/blob.py` | `QlOsBlob`: minimal run loop for raw binaries |
|
||||
| `qiling/extensions/multitask.py` | `UnicornTask`, `NestedCounter`, `MultiTaskUnicorn(Uc)`: cooperative multitasking over Unicorn |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. `QlOsMcu`/`QlOsBlob` set `type = QL_OS.MCU|BLOB`
|
||||
as class attributes (`qiling/os/mcu/mcu.py:42`) so `select_os` and
|
||||
`ql.baremetal` work. Only MCU is in `QL_OS_BAREMETAL`
|
||||
(`qiling/const.py:74`); BLOB is an OS without services but with a normal
|
||||
memory manager.
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- **Step mode** (default): `QlOsMcu.run` loops `emu_start(pc, 0, count=1)`
|
||||
then `ql.hw.step()` until `exit_point`, `count`, or `stop()`
|
||||
(`qiling/os/mcu/mcu.py:74-90`); `timeout` is unsupported here.
|
||||
- **Fast mode** (`ql.os.fast_mode = True`): a single `MCUTask` runs under
|
||||
`MultiTaskUnicorn.tasks_start` with a scheduling interval in ms
|
||||
(`qiling/extensions/multitask.py:154-158`); `count` means scheduling
|
||||
rounds, and `hw.step()` runs on interruption (`qiling/os/mcu/mcu.py:27-38`).
|
||||
- **Interrupts**: NVIC peripherals call `arch.interrupt_handler`, which
|
||||
enters the vector inside `QlInterruptContext`
|
||||
(`qiling/arch/cortex_m.py:146-161`); the Unicorn exception hook is
|
||||
installed by the MCU loader (`qiling/loader/mcu.py:140`).
|
||||
- **Effective PC**: run loops read `arch.effective_pc` when present to
|
||||
keep the thumb bit consistent (`qiling/os/mcu/mcu.py:54-58`).
|
||||
- **BLOB**: `QlOsBlob.run` resolves entry/exit overrides, creates a heap
|
||||
only if the profile `[CODE]` section defines `heap_address`/`heap_size`,
|
||||
then runs `entry_point` → `exit_point` with the standard `emu_start`
|
||||
(`qiling/os/blob/blob.py:43-60`).
|
||||
- `ql.hw` exists only when `ql.baremetal` (`qiling/core.py:191`,
|
||||
`:357`); BLOB targets needing hardware must use MCU mode.
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/os/mcu/mcu.py:41` - `QlOsMcu(QlOs)` - `run` (`:53`), `stop`
|
||||
(`:49`), `fast_mode` flag (`:47`).
|
||||
- `qiling/os/mcu/mcu.py:17` - `MCUTask(UnicornTask)` - `on_start` (`:23`),
|
||||
`on_interrupted` (`:27`).
|
||||
- `qiling/extensions/multitask.py:26` - `UnicornTask` - task record with
|
||||
`pc`; `:152` - `MultiTaskUnicorn(Uc)` - `task_create`, `tasks_start`.
|
||||
- `qiling/os/blob/blob.py:14` - `QlOsBlob(QlOs)` - `run` (`:43`).
|
||||
- `qiling/core.py:357` - `Qiling.baremetal` - gates hardware-manager
|
||||
creation.
|
||||
|
||||
## Interactions
|
||||
|
||||
- Both subclass [os-base.md](os-base.md) `QlOs`.
|
||||
- MCU drives [hw.md](hw.md) through `ql.hw.step()`; peripherals and MMIO
|
||||
are created by the MCU loader ([loader.md](loader.md)) from the YAML
|
||||
profile and the `env` dict ([core.md](core.md) `profile_setup`,
|
||||
`qiling/utils.py:419`).
|
||||
- Interrupt entry/exit uses [arch.md](arch.md) `QlArchCORTEX_M`; the
|
||||
Cortex-M arch constructs a `MultiTaskUnicorn` instead of a plain `Uc`
|
||||
(`qiling/arch/cortex_m.py:78`; the import at `:22` is one of the three
|
||||
documented upward imports in the arch layer, see [arch.md](arch.md)).
|
||||
- Fuzzing MCU firmware (`examples/fuzzing/stm32f429/`) combines this mode
|
||||
with [extensions.md](extensions.md) AFL support.
|
||||
- Snapshots include hardware state when `save(hw=True)`
|
||||
(`tests/test_mcu.py:29-56`).
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 -m unittest test_blob.BlobTest.test_uboot_arm # pass = "OK", exit 0
|
||||
```
|
||||
|
||||
- MCU (also proves [hw.md](hw.md)): `cd tests && python3 test_mcu.py`
|
||||
— pass = `Ran 18 tests … OK`.
|
||||
- `python3 test_blob.py` as a whole fails on a clean checkout; see Open Gaps.
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **Run-loop changes** must keep step mode and fast mode behaviorally
|
||||
equivalent for `tests/test_mcu.py::test_mcu_hackme_stm32f429` and
|
||||
`::test_mcu_fastmode_stm32f429` (`tests/test_mcu.py:403`, `:427`).
|
||||
- **New chip family**: add the `env` map under `qiling/extensions/mcu/`,
|
||||
any missing peripherals under `qiling/hw/` ([hw.md](hw.md)), a YAML
|
||||
profile if needed, and a firmware sample plus test case.
|
||||
- **Do not** add OS services (syscalls, fs) to these run loops; that is a
|
||||
different OS personality.
|
||||
- Improvement candidate (proposal): support `timeout` in step mode.
|
||||
Success check: a `tests/test_mcu.py` case with `timeout` set terminates.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- `test_blob.BlobTest.test_blob_raw` (`tests/test_blob.py:85`) reads
|
||||
`examples/rootfs/blob/example_raw.bin` (`:96`), which the pinned
|
||||
`examples/rootfs` submodule (`f71f45f`) does not ship; upstream `master`
|
||||
now contains it, so bumping the submodule should fix it (unverified
|
||||
locally).
|
||||
- Chip coverage is limited to families under `qiling/extensions/mcu/`.
|
||||
- BLOB provides no services by design.
|
||||
@@ -1,161 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# OS base — shared OS-layer services
|
||||
|
||||
## Goal
|
||||
|
||||
Own the OS-agnostic services every OS personality builds on: the abstract
|
||||
`QlOs` class, the emulated memory manager and heap, function-call argument
|
||||
marshalling over calling conventions, filesystem virtualization (rootfs
|
||||
confinement plus host mapping), file-descriptor objects, green-thread base,
|
||||
struct helpers, and API-call statistics. It must not contain syscall or API
|
||||
implementations. No roadmap milestone applies; maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — path virtualization and struct helpers have dedicated unit tests
|
||||
(observed: `Ran 3 tests … OK (skipped=1)` and `Ran 18 tests … OK (skipped=4)`);
|
||||
everything else is exercised by every OS suite.
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/os/os.py` | Abstract `QlOs`: stdio, path/fs-mapper composition, `user_defined_api`, `call`, `set_api`, abstract `run()`, `emu_error` |
|
||||
| `qiling/os/memory.py` | `QlMemoryManager` (`ql.mem`: map/unmap/protect/search/MMIO) and `QlMemoryHeap` |
|
||||
| `qiling/os/fcall.py` | `QlFunctionCall`: read args / write retval using `qiling/cc/` conventions |
|
||||
| `qiling/os/mapper.py` | `QlFsMapper` / `QlFsMappedObject`: redirect emulated paths to host files or Python objects |
|
||||
| `qiling/os/path.py` | `QlOsPath`: rootfs confinement and virtual↔host path conversion |
|
||||
| `qiling/os/filestruct.py`, `disk.py` | `ql_file`/`PersistentQlFile` fd objects and raw-disk mapped object |
|
||||
| `qiling/os/thread.py` | `QlThread(Greenlet)` base for multithread emulation |
|
||||
| `qiling/os/stats.py` | `QlOsStats` / `QlWinStats`: API and syscall call statistics |
|
||||
| `qiling/os/struct.py` | `BaseStruct` and the `get_packed_struct` / `get_aligned_struct` factories |
|
||||
| `qiling/os/utils.py`, `const.py` | `QlOsUtils` (string readers, `print_function`), fcall parameter type constants |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Local patterns:
|
||||
|
||||
- Guest-facing structures are `ctypes` classes built through
|
||||
`get_packed_struct`/`get_aligned_struct` (`qiling/os/struct.py:226`,
|
||||
`:246`) so endianness and pointer width follow the target; canonical use:
|
||||
`qiling/os/posix/syscall/epoll.py:31-45`.
|
||||
- API parameter types are the `PARAM_*`/`STRING`/`WSTRING`/`GUID`
|
||||
constants in `qiling/os/const.py:9-32`; resolvers turn pointers into
|
||||
Python values (`qiling/os/os.py:96-104`).
|
||||
- Errors: `QlErrorNotImplemented`, `QlMemoryMappedError`, `QlOutOfMemory`
|
||||
from `qiling/exception.py`. `assert` is used for page-size and alignment
|
||||
checks in `qiling/os/memory.py:65`, `:281`, `:303`; `TODO.md:638-644`
|
||||
flags this as observed, not a convention.
|
||||
- Logging goes through `ql.log`; `QlOsUtils.print_function`
|
||||
(`qiling/os/utils.py:106`) is the single formatter for syscall/API log
|
||||
lines.
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- **Construction order**: `QlMemoryManager` is created before the OS
|
||||
(`qiling/core.py:188-189`); `QlOs.__init__` reads the profile, builds
|
||||
`QlOsPath`/`QlFsMapper` only for POSIX, Windows, and DOS
|
||||
(`qiling/os/os.py:44-49`), and sets a bit-width default exit point
|
||||
(`:82-87`).
|
||||
- **Rootfs confinement**: every guest path resolves through `QlOsPath`;
|
||||
`__is_safe_host_path` canonicalizes and requires the result to stay under
|
||||
the rootfs (`qiling/os/path.py:239-266`). `QlFsMapper` overrides win only
|
||||
for explicitly mapped virtual paths (`qiling/os/mapper.py:189`). This is
|
||||
the host trust boundary named in the root Review deviations;
|
||||
`tests/test_pathutils.py` enforces it.
|
||||
- **API call protocol**: `QlOs.call` (`qiling/os/os.py:198`) resolves
|
||||
params via the current `fcall`, invokes the handler with `onenter/onexit`
|
||||
hooks from `user_defined_api`, logs, records stats, and sets `pc` to the
|
||||
return address unless emulation already stopped (`:215-221`).
|
||||
- **Memory manager owns `map_info`**; all mapping goes through
|
||||
`map`/`map_mmio`/`unmap`/`protect` (`qiling/os/memory.py:622`, `:644`,
|
||||
`:451`, `:612`). MMIO regions are Unicorn callbacks bound to a
|
||||
`QlMmioHandler` (`:17`).
|
||||
- **Heap**: `QlMemoryHeap` (`:678`) is a simple chunk allocator used by
|
||||
Windows/UEFI APIs; the sanitizer extension wraps it.
|
||||
- **Threads**: `QlThread` is a gevent `Greenlet` (`qiling/os/thread.py:11`);
|
||||
cooperative scheduling only.
|
||||
- `QlOs.save/restore` are empty in the base (`qiling/os/os.py:106-109`);
|
||||
personalities that support snapshots override them.
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/os/os.py:24` - `QlOs` - `__init__` (`:29`), `call` (`:198`),
|
||||
`set_api` (`:225`), abstract `run()` (`:240`), `stop` (`:243`),
|
||||
`emu_error` (`:249`).
|
||||
- `qiling/os/memory.py:40` - `QlMemoryManager` - `map` (`:622`),
|
||||
`map_mmio` (`:644`), `read/write` (`:349`/`:383`), `read_ptr/write_ptr`
|
||||
(`:361`/`:393`), `search` (`:414`), `find_free_space` (`:540`),
|
||||
`get_lib_base` (`:254`).
|
||||
- `qiling/os/memory.py:678` - `QlMemoryHeap` - `alloc` (`:721`), `free`
|
||||
(`:771`), `size` (`:757`).
|
||||
- `qiling/os/fcall.py:21` - `QlFunctionCall(ql, cc, accessors)` - argument
|
||||
and return-value marshalling over `qiling/cc/`.
|
||||
- `qiling/os/mapper.py:64` - `QlFsMapper` - behind `Qiling.add_fs_mapper`
|
||||
(`qiling/core.py:701`); `open_ql_file` (`qiling/os/mapper.py:129`),
|
||||
`add_mapping` (`:189`).
|
||||
- `qiling/os/path.py:16` - `QlOsPath` - `virtual_to_host_path` (`:311`),
|
||||
`is_safe_host_path` (`:326`), `host_to_virtual_path` (`:268`).
|
||||
- `qiling/os/filestruct.py:18` - `ql_file` - fd object interface
|
||||
(`read/write/fileno/lseek/close/fstat/fcntl`); `PersistentQlFile` (`:97`)
|
||||
for stdio.
|
||||
- `qiling/os/thread.py:11` - `QlThread(Greenlet)`.
|
||||
|
||||
## Interactions
|
||||
|
||||
- Instantiated by [core.md](core.md): memory at `qiling/core.py:188`, the
|
||||
concrete OS at `:189` via `select_os`.
|
||||
- Uses [arch.md](arch.md) calling conventions inside `QlFunctionCall`.
|
||||
- Subclassed by [os-posix.md](os-posix.md) (`QlOsPosix`),
|
||||
[os-windows.md](os-windows.md) (`QlOsWindows`, `QlOsUefi`, `QlOsDos`),
|
||||
and [os-baremetal.md](os-baremetal.md) (`QlOsMcu`, `QlOsBlob`).
|
||||
- [loader.md](loader.md) maps segments through `QlMemoryManager`;
|
||||
[hw.md](hw.md) MMIO goes through `map_mmio`.
|
||||
- The heap is wrapped by `QlSanitizedMemoryHeap` in
|
||||
[extensions.md](extensions.md); fake stdio streams from
|
||||
`qiling/extensions/pipe.py` are assigned to `ql.os.stdin/stdout`.
|
||||
- [kernel-proxy.md](kernel-proxy.md) stores `ql_proxy_fd` objects in the
|
||||
POSIX fd table, relying on the `ql_file` duck-typed interface.
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_pathutils.py && python3 test_struct.py # pass = both print "OK", exit 0
|
||||
```
|
||||
|
||||
- `tests/test_pathutils.py:30` proves rootfs path virtualization (POSIX
|
||||
cases run on Linux; NT cases skip, `:74`); `tests/test_struct.py:202`
|
||||
proves the struct helpers (four wchar cases are skipped as broken,
|
||||
`:170`, `:185`).
|
||||
- Memory manager and heap have no unit tests; they are proven by every OS
|
||||
suite and `tests/test_uefi.py::test_x8664_uefi_santizier`.
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **Path handling changes** must keep `__is_safe_host_path` as the last
|
||||
check before any host `open`; add cases to `tests/test_pathutils.py`.
|
||||
- **Memory manager changes**: `map_info` labels are parsed by
|
||||
`get_lib_base` and by the coverage/report extensions; keep the
|
||||
`[boxed] name` label format.
|
||||
- **New fcall parameter types** go in `qiling/os/const.py` with a resolver
|
||||
in `QlOs.resolvers`; OS-specific aliases live in the OS module
|
||||
(`qiling/os/windows/api.py`).
|
||||
- **Do not** add OS-specific branches to `QlOs`; override in the
|
||||
personality instead (see `QlOsPosix.set_api`, `qiling/os/posix/posix.py:149`).
|
||||
- Improvement candidates (proposals): bound `read_cstring`
|
||||
(`qiling/os/utils.py:88`, `TODO.md:659-661`) so a missing terminator on
|
||||
MMIO cannot spin; implement `QlOs.save/restore` for Windows/UEFI
|
||||
(`TODO.md:663-665`). Success check: a regression case in
|
||||
`tests/test_elf.py` or `tests/test_uefi.py`.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- Multithread emulation (gevent) is opt-in via `multithread=True` and
|
||||
cooperative only; `TODO.md:429-525` sketches a real-thread design as a
|
||||
proposal.
|
||||
- `QlOs.run()` is a loose contract: each personality implements its own
|
||||
loop.
|
||||
- No unit tests for `QlMemoryManager`/`QlMemoryHeap`.
|
||||
@@ -1,168 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# OS POSIX — Linux, FreeBSD, macOS, QNX
|
||||
|
||||
## Goal
|
||||
|
||||
Emulate POSIX operating systems by intercepting syscalls: a shared
|
||||
`QlOsPosix` layer owns the per-arch syscall ABI, number→name mapping, the
|
||||
dispatcher, the fd table, and SysV IPC; per-OS subclasses add their
|
||||
personality (Linux futex/procfs/threads/kernel modules, macOS mach ports
|
||||
and kexts, QNX message passing, FreeBSD). It must not own the kernel proxy
|
||||
(see [kernel-proxy.md](kernel-proxy.md)). No roadmap milestone applies;
|
||||
maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — Linux is the flagship target with the largest suite (observed:
|
||||
`tests/test_posix.py` → `Ran 65 tests … OK (skipped=2)`); FreeBSD, QNX, and
|
||||
Android are covered on Linux hosts; macOS suites are host-gated (see Open
|
||||
Gaps).
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/os/posix/posix.py` | `QlOsPosix`: syscall ABI selection, dispatcher `load_syscall`, `set_syscall`, `QlFileDes` fd table, signals bitmap |
|
||||
| `qiling/os/posix/syscall/abi/` | `QlSyscallABI` per arch (id/arg/return registers) |
|
||||
| `qiling/os/posix/syscall/` | Syscall implementations, one module per family (`unistd`, `mman`, `socket`, `net`, `epoll`, `sched`, `futex`, `signal`, `ioctl`, …), re-exported by `__init__.py` |
|
||||
| `qiling/os/posix/const.py`, `const_mapping.py`, `structs.py`, `stat.py` | Per-arch constants (incl. MIPS socket options), flag decoders, structs |
|
||||
| `qiling/os/posix/shm.py`, `msq.py`, `filestruct.py` | SysV IPC and POSIX-specific fd objects (sockets, pipes) |
|
||||
| `qiling/os/linux/linux.py` | `QlOsLinux`: syscall hook wiring per arch, procfs, dynamic-ELF run loop |
|
||||
| `qiling/os/linux/thread.py`, `futex.py`, `procfs.py` | Green-thread manager per arch, futex emulation, `/proc` files |
|
||||
| `qiling/os/linux/map_syscall.py`, `syscall_nums.py`, `syscall.py` | Number→name tables per arch, Linux-only syscalls |
|
||||
| `qiling/os/linux/function_hook.py`, `fncc.py`, `kernel_api/` | ELF symbol hooking (`set_api` for user binaries) and `.ko` kernel API emulation |
|
||||
| `qiling/os/freebsd/`, `qiling/os/macos/`, `qiling/os/qnx/` | `QlOsFreebsd`, `QlOsMacos` (+ mach ports/tasks/kext), `QlOsQnx` (+ `message.py`) |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Local patterns (enforced by the dispatcher, not a
|
||||
linter):
|
||||
|
||||
- A syscall handler is a module-level function named
|
||||
`ql_syscall_<name>(ql, arg1, …)`; `SYSCALL_PREF`
|
||||
(`qiling/os/posix/posix.py:19`) is the prefix, and parameter names are
|
||||
read from the signature to log arguments (`:200-205`). Canonical example:
|
||||
`qiling/os/posix/syscall/sched.py:13` and `:133` (`clone`/`clone3`).
|
||||
- Return an `int` (negative errno on failure) or `None` to leave the return
|
||||
register untouched (`qiling/os/posix/posix.py:227-229`).
|
||||
- Constants are re-declared locally where the platform defines them
|
||||
(`qiling/os/posix/syscall/sched.py:14-39`); MIPS-specific socket values
|
||||
live in `qiling/os/posix/const.py:309-456`.
|
||||
- Guest structs use `get_packed_struct` (`qiling/os/posix/syscall/epoll.py:35`).
|
||||
- `qiling/os/posix/syscall/epoll.py:140-141` still contains tab-indented
|
||||
lines (the other such file is in [debugger.md](debugger.md)); do not add
|
||||
more.
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- **Entry**: `QlOsLinux.load` registers `hook_syscall` on the arch-specific
|
||||
trap (`hook_intno` for ARM/ARM64/MIPS/RISC-V/PPC, `hook_intno(0x80)` and
|
||||
`hook_insn(SYSCALL)` for x86, `qiling/os/linux/linux.py:53-116`).
|
||||
- **Dispatch** (`load_syscall`, `qiling/os/posix/posix.py:170`): read id via
|
||||
the ABI (`get_id`, `qiling/os/posix/syscall/abi/__init__.py:31`) → map to
|
||||
name via `qiling/os/<os>/map_syscall.py` → look up user CALL hook, else
|
||||
`qiling/os/<os>/syscall`, else `qiling/os/posix/syscall` → run ENTER hook,
|
||||
handler, EXIT hook → set return value. Unknown syscalls log a warning and
|
||||
raise `QlErrorSyscallNotFound` only when `debug_stop` is set
|
||||
(`qiling/os/posix/posix.py:255-258`).
|
||||
- **User overrides** go through `set_syscall(target, handler, intercept)`
|
||||
(`qiling/os/posix/posix.py:132`), keyed by name or number; `set_api` on
|
||||
non-driver binaries hooks ELF symbols via `FunctionHook` (`:149-153`,
|
||||
`qiling/os/linux/function_hook.py:499`).
|
||||
- **fd table**: `QlFileDes` (`qiling/os/posix/posix.py:22`) is a
|
||||
fixed-size list of duck-typed
|
||||
objects (`ql_file`, sockets, pipes, `ql_proxy_fd`); handlers index it and
|
||||
call `read/write/close/fstat/…` without type checks.
|
||||
- **Run loop** (`QlOsLinux.run`, `qiling/os/linux/linux.py:148`): shellcode
|
||||
runs once; `multithread=True` hands control to
|
||||
`QlLinuxThreadManagement` (`qiling/os/linux/thread.py:537`); otherwise
|
||||
run ld.so to `elf_entry`, apply lib patches, then run to the exit point.
|
||||
- **Processes**: `clone` without `CLONE_VM` forks the host process
|
||||
(`qiling/os/posix/syscall/sched.py:50-59`); threads are gevent greenlets.
|
||||
- **Networking** uses host sockets; the profile `[NETWORK]` section
|
||||
controls IPv6 and bind-to-localhost (`qiling/os/posix/posix.py:58-61`).
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/os/posix/posix.py:48` - `QlOsPosix(QlOs)` - ABI table (`:69-78`),
|
||||
fd table, IPC, signal bitmap.
|
||||
- `qiling/os/posix/posix.py:170` - `load_syscall()` - the dispatcher.
|
||||
- `qiling/os/posix/posix.py:132` - `set_syscall(target, handler, intercept)`.
|
||||
- `qiling/os/posix/posix.py:100` - `__get_syscall_mapper(archtype)` - loads
|
||||
`qiling/os/<os>/map_syscall.py:get_syscall_mapper`.
|
||||
- `qiling/os/posix/syscall/abi/__init__.py:14` - `QlSyscallABI` -
|
||||
`get_id` (`:31`), `get_params` (`:39`), `set_return_value` (`:53`).
|
||||
- `qiling/os/linux/linux.py:26` - `QlOsLinux(QlOsPosix)` - `load` (`:53`),
|
||||
`setup_procfs` (`:123`), `hook_syscall` (`:137`), `run` (`:148`).
|
||||
- `qiling/os/linux/map_syscall.py:14` - `get_syscall_mapper(archtype)` and
|
||||
the per-arch `*_syscall_table` dicts (also read by the kernel proxy).
|
||||
- `qiling/os/freebsd/freebsd.py:16`, `qiling/os/macos/macos.py:24`,
|
||||
`qiling/os/qnx/qnx.py:23` - the other personalities.
|
||||
|
||||
## Interactions
|
||||
|
||||
- Subclasses [os-base.md](os-base.md) `QlOs`; fd objects come from
|
||||
`qiling/os/filestruct.py` and `qiling/os/posix/filestruct.py`.
|
||||
- Syscall entry is an arch hook registered through [core.md](core.md)
|
||||
`QlCoreHooks`; registers are read through [arch.md](arch.md).
|
||||
- Runs images prepared by [loader.md](loader.md) (`QlLoaderELF`,
|
||||
`QlLoaderMACHO`); kernel modules use `hook_kernel_api`
|
||||
(`qiling/os/linux/kernel_api/hook.py:14`).
|
||||
- Path/fs access goes through `QlOsPath`/`QlFsMapper` ([os-base.md](os-base.md)).
|
||||
- [kernel-proxy.md](kernel-proxy.md) plugs in purely through
|
||||
`set_syscall(…, QL_INTERCEPT.CALL)` and the fd table.
|
||||
- Fuzzing harnesses in [extensions.md](extensions.md) hijack stdin with
|
||||
`qiling/extensions/pipe.py` and syscalls via `set_syscall`.
|
||||
- [debugger.md](debugger.md) is attached by `Qiling.run` before `os.run()`.
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_posix.py # pass = "Ran 65 tests … OK (skipped=2)", exit 0
|
||||
```
|
||||
|
||||
- `tests/test_posix.py` aggregates `test_elf.py`, `test_riscv.py`, and
|
||||
`test_qltool.py`; the POSIX members the CI script `tests/test_onlinux.sh`
|
||||
adds are `test_elf_multithread.py` (`Ran 24 tests … OK (skipped=1)`,
|
||||
includes TCP/UDP/HTTP and `clone3`), `test_elf_ko.py`, `test_qnx.py`,
|
||||
`test_android.py`, `test_edl.py`, `test_tendaac15_httpd.py`. Run them
|
||||
one at a time (fixed localhost ports; see the root Verification and
|
||||
Review Map).
|
||||
- `test_elf_ko.py` needs `unzip -P infected m0hamed_rootkit.ko.zip` in
|
||||
`examples/rootfs/x86_linux/kernel/` first (CI does this,
|
||||
`.github/workflows/build-ci.yml:73`); without it one case errors with
|
||||
`QlErrorFileNotFound`.
|
||||
- macOS: `tests/test_macho.sh` on a macOS host with collected dylibs.
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **Adding a syscall**: implement `ql_syscall_<name>` in the matching
|
||||
`qiling/os/posix/syscall/<family>.py` (or `qiling/os/<os>/syscall.py` if
|
||||
OS-specific), ensure the name exists in every relevant
|
||||
`map_syscall.py` table, and add a case to `tests/test_elf.py` with a
|
||||
sample in the rootfs submodule. Coverage is demand-driven (root
|
||||
deviations); do not add unrequested syscalls.
|
||||
- **ABI changes** touch `qiling/os/posix/syscall/abi/<arch>.py` only.
|
||||
- **fd-table contract**: any new fd object must implement the `ql_file`
|
||||
method set used by `unistd.py`/`socket.py` handlers.
|
||||
- **Do not** call host syscalls directly from handlers except through the
|
||||
existing fd objects; host forwarding is the kernel proxy's job.
|
||||
- Improvement candidates (proposals): replace the bare `except:` in
|
||||
`qiling/os/posix/syscall/select.py:78` (`TODO.md:628-636`; the
|
||||
`filestruct.py` sites listed there are already fixed); un-skip
|
||||
`tests/test_elf.py:441` once the ARM sample is rebuilt. Success check:
|
||||
named tests pass without skips.
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- macOS suites need a macOS host; the CI job is commented out
|
||||
(`.github/workflows/build-ci.yml:83-93`).
|
||||
- Signals are largely stubs and threading is cooperative; `TODO.md:7-21`
|
||||
states the problem and `TODO.md:271-567` proposes phases 1–4 (networking,
|
||||
real threads, signals) on top of the kernel proxy.
|
||||
- Two skipped tests: `tests/test_elf.py:441` (ARM sample) and `:887`
|
||||
(stdin hijacking); `tests/test_elf_multithread.py:185` (ARM big-endian
|
||||
invalid instruction).
|
||||
@@ -1,153 +0,0 @@
|
||||
---
|
||||
eatmycode_version: "1.2.0"
|
||||
---
|
||||
|
||||
# OS Windows family — Windows, UEFI, DOS
|
||||
|
||||
## Goal
|
||||
|
||||
Emulate API-call-driven (rather than syscall-driven) environments.
|
||||
Windows: Win32/NT API emulation with handles, registry, fibers, threads,
|
||||
and PE process structures. UEFI: boot/runtime/SMM services and a protocol
|
||||
database for DXE/SMM modules. DOS: BIOS/DOS interrupt services for
|
||||
COM/EXE/MBR targets. No roadmap milestone applies; maturity-based status.
|
||||
|
||||
## Status
|
||||
|
||||
`done` — UEFI and DOS suites run on Linux (observed: `Ran 2 tests … OK`,
|
||||
`Ran 1 test … OK`); the Windows PE suite runs in CI on a Windows host with
|
||||
collected system DLLs and is not runnable from a Linux checkout.
|
||||
|
||||
## Code Structure
|
||||
|
||||
| File | Role |
|
||||
| ---- | ---- |
|
||||
| `qiling/os/windows/windows.py` | `QlOsWindows`: fcall selector, heap, GDT, component setup, `hook_winapi`, run loop |
|
||||
| `qiling/os/windows/fncc.py`, `api.py` | `@winsdkapi` decorator (cc + typed params) and Windows type aliases over `qiling/os/const.py` |
|
||||
| `qiling/os/windows/dlls/` | API implementations per DLL (`kernel32/` split by header, `ntdll.py`, `msvcrt.py`, `advapi32.py`, `user32.py`, …) |
|
||||
| `qiling/os/windows/handle.py`, `registry.py`, `fiber.py`, `thread.py`, `clipboard.py` | Kernel-object emulation: `HandleManager`, `RegistryManager` (python-registry hives), `FiberManager`, `QlWindowsThreadManagement`, `Clipboard` |
|
||||
| `qiling/os/windows/structs.py`, `const.py`, `wdk_const.py`, `utils.py` | Guest structs and constants (user + WDK) |
|
||||
| `qiling/os/uefi/uefi.py` | `QlOsUefi` run loop and diagnostics (`emit_context`, `emit_stack`) |
|
||||
| `qiling/os/uefi/bs.py`, `rt.py`, `ds.py`, `smm.py`, `smst.py`, `st.py`, `protocols/` | Boot/runtime/DXE/SMM service tables and protocol implementations (`@dxeapi`) |
|
||||
| `qiling/os/uefi/context.py`, `hob.py`, `Uefi*.py`, `type32.py`/`type64.py` | DXE/SMM contexts, HOB list, EDK2-derived type definitions |
|
||||
| `qiling/os/dos/dos.py`, `interrupts/` | `QlOsDos` flag helpers and `INT 10h/13h/15h/16h/19h/1Ah/20h/21h` handlers |
|
||||
|
||||
## Language and Conventions
|
||||
|
||||
Python; root rules apply. Local patterns:
|
||||
|
||||
- A Windows API is `hook_<Name>(ql, address, params)` decorated with
|
||||
`@winsdkapi(cc=STDCALL|CDECL|MS64, params={...})` where param types are
|
||||
the aliases in `qiling/os/windows/api.py`; canonical example:
|
||||
`qiling/os/windows/dlls/kernel32/fibersapi.py:10-57`. The C prototype
|
||||
is kept as a comment above each hook.
|
||||
- UEFI services use `@dxeapi(params=…)` (`qiling/os/uefi/fncc.py:11`),
|
||||
e.g. `qiling/os/uefi/bs.py:23-35`.
|
||||
- DOS interrupt handlers are plain functions registered in the
|
||||
`handlers` map (`qiling/os/dos/interrupts/__init__.py:24`).
|
||||
- Unimplemented APIs are logged and raise only under `debug_stop`
|
||||
(`qiling/os/windows/windows.py:196-199`); implementation errors raise
|
||||
`QlErrorSyscallError` (`:189-194`).
|
||||
- Stub generation for new DLLs: `qiling/extensions/winsdkapi.py` emits
|
||||
`@winsdkapi` skeletons from Windows SDK JSON.
|
||||
|
||||
## Design and Invariants
|
||||
|
||||
- **Dispatch is address-based**: `hook_winapi` runs on every instruction
|
||||
(`hook_code`, `qiling/os/windows/windows.py:130`) and matches `address`
|
||||
against `ql.loader.import_symbols` filled by the PE loader; user CALL
|
||||
overrides in `user_defined_api` win over `dlls/` implementations
|
||||
(`:172-186`).
|
||||
- **Calling conventions**: x86 selects STDCALL/CDECL per API; x86-64 always
|
||||
uses MS64 (`:41-65`); `QlOs.call` handles marshalling and the return jump
|
||||
([os-base.md](os-base.md)).
|
||||
- **Process state** comes from the profile (`[OS32]/[OS64]` heap, `[PATH]`,
|
||||
`[USER]`, `[KERNEL]`; `qiling/profiles/windows.ql`) and the PE loader
|
||||
(PEB/TEB/LDR); the GDT is set up by the OS
|
||||
(`qiling/os/windows/windows.py:132`).
|
||||
- **Registry** is read from hive files under the rootfs and saved back at
|
||||
the end of `run()` (`qiling/os/windows/windows.py:217`).
|
||||
- **UEFI**: `run()` creates the SMM environment, then executes the loaded
|
||||
module chain from the loader (`qiling/os/uefi/uefi.py:202-221`); service
|
||||
tables are callable tables installed in guest memory by
|
||||
`qiling/os/uefi/st.py`/`smst.py`; variables are managed in `rt.py` with
|
||||
known gaps (`qiling/os/uefi/rt.py:204-205`).
|
||||
- **DOS**: `hook_syscall` installs an interrupt hook that dispatches on
|
||||
`(intno, AH)` with ENTER/CALL/EXIT user overrides keyed by that tuple
|
||||
(`qiling/os/dos/dos.py:83-105`).
|
||||
- **Trust boundary**: DLLs, hives, and PE images come from the rootfs; the
|
||||
root Review deviations apply to any header-derived size that reaches
|
||||
host allocations.
|
||||
|
||||
## Key Types and Entry Points
|
||||
|
||||
- `qiling/os/windows/windows.py:33` - `QlOsWindows(QlOs)` - `load` (`:125`),
|
||||
`__setup_components` (`:156`), `hook_winapi` (`:172`), `run` (`:201`).
|
||||
- `qiling/os/windows/fncc.py:17` - `winsdkapi(cc, params, passthru)`.
|
||||
- `qiling/os/windows/handle.py:29` - `HandleManager`; `registry.py:164` -
|
||||
`RegistryManager`; `fiber.py:20` - `FiberManager`; `thread.py:94` -
|
||||
`QlWindowsThreadManagement`.
|
||||
- `qiling/os/uefi/uefi.py:22` - `QlOsUefi(QlOs)` - `run` (`:202`),
|
||||
`set_api` (`:199`), `save/restore` (`:43`/`:49`).
|
||||
- `qiling/os/uefi/context.py:12` - `UefiContext` - `DxeContext` (`:80`),
|
||||
`SmmContext` (`:87`), protocol database and configuration tables.
|
||||
- `qiling/os/dos/dos.py:33` - `QlOsDos(QlOs)` - `hook_syscall` (`:83`),
|
||||
flag helpers (`:65-80`).
|
||||
- `qiling/os/os.py:225` - `QlOs.set_api` - user API override by name
|
||||
(Windows/UEFI) or `(intno, ah)` tuple (DOS).
|
||||
|
||||
## Interactions
|
||||
|
||||
- All three subclass [os-base.md](os-base.md) `QlOs`; Windows and UEFI
|
||||
allocate from `QlMemoryHeap`.
|
||||
- [loader.md](loader.md): `QlLoaderPE` builds PEB/TEB and `import_symbols`;
|
||||
`QlLoaderPE_UEFI` installs protocols into `qiling/os/uefi/context.py`;
|
||||
`QlLoaderDOS` sets real-mode state.
|
||||
- Argument marshalling uses `QlFunctionCall` with `qiling/cc/intel.py`
|
||||
conventions ([arch.md](arch.md)).
|
||||
- Windows threads subclass `QlThread` ([os-base.md](os-base.md)) and honor
|
||||
`QL_HOOK_BLOCK` when switching (`qiling/os/windows/thread.py:120`).
|
||||
- `examples/uefi_sanitized_heap.py` pairs UEFI with the heap sanitizer from
|
||||
[extensions.md](extensions.md); `tests/test_uefi.py:24` proves it.
|
||||
- [debugger.md](debugger.md): `tests/test_windows_debugger.py` covers GDB
|
||||
on PE targets (Windows host).
|
||||
|
||||
## How to Test
|
||||
|
||||
```sh
|
||||
cd tests && python3 test_uefi.py && python3 test_dos.py # pass = "OK" twice, exit 0 (Linux-runnable)
|
||||
```
|
||||
|
||||
- Windows host only: run `examples\scripts\dllscollector.bat` from the
|
||||
repository root, then `cd tests && test_pe.bat` (runs `test_pe.py`,
|
||||
`test_windows_stdio.py`, `test_peshellcode.py`, `test_windows_debugger.py`,
|
||||
`test_dos.py`, `test_pe_sys.py`; CI: `.github/workflows/build-ci.yml:54-62`).
|
||||
- `tests/test_windows_cpp_x86.py`/`_x8664.py` are not in `test_pe.bat`.
|
||||
|
||||
## Review and Refactor Guide
|
||||
|
||||
- **Adding a Win32 API**: put `hook_<Name>` in the DLL module matching
|
||||
the exporting DLL (`kernel32/` is split by SDK header); use the
|
||||
`winsdkapi.py` generator for the signature; add a sample-driven case to
|
||||
`tests/test_pe.py`. Coverage is demand-driven (root deviations).
|
||||
- **Changing `import_symbols` handling** must stay in sync with
|
||||
`qiling/loader/pe.py:86`.
|
||||
- **UEFI protocol additions** go in `qiling/os/uefi/protocols/` and are
|
||||
registered through the context's protocol database; test via
|
||||
`tests/test_uefi.py` with a sample in the rootfs submodule.
|
||||
- **DOS interrupts** are added to the `handlers` map with a leaf switch on
|
||||
`AH`.
|
||||
- **Do not** put Windows-only fcall selection in `QlOs`; it lives in
|
||||
`QlOsWindows.__init__`.
|
||||
- Improvement candidates (proposals): implement `QlOs.save/restore` for
|
||||
Windows (`TODO.md:663-665`); replace the remaining bare `except:` in
|
||||
`qiling/os/windows/registry.py:127` (`TODO.md:628-636`).
|
||||
|
||||
## Open Gaps / Roadmap
|
||||
|
||||
- Windows tests require a Windows host: system DLLs and hives are not
|
||||
redistributable.
|
||||
- Win32 API surface is demand-driven; fibers, registry, handles, and DLL
|
||||
resolution have known gaps (`TODO.md:667-670`).
|
||||
- UEFI variables ignore namespaces and access attributes
|
||||
(`qiling/os/uefi/rt.py:204-205`); SMM covers common protocols only.
|
||||
+1
-1
Submodule examples/rootfs updated: f71f45fe1a...55893d6fb2
Reference in New Issue
Block a user