`vmm_place_vmcode()` computed the shrink step as `size / 16` where size
is a `size_t` pointer, so the step was derived from a stack address and
the "don't go too small" check always broke out of the loop. A failed
vmcode reservation therefore never retried with a smaller size. Use
`*size` instead.
This PR fixes two bugs identified in #8159:
- The `api.rseq` test register an rseq with `sizeof(*tls_rseq)` which is
64, but the address is only 32-aligned. The fix is to add
`__attribute__((aligned(64)))` to the declaration.
- After adding `__attribute__((aligned(64)))`, our `try_struct_rseq()`
still cannot find it. This is because the linux kernel requires
`rseq_len` to match the registered length (64), and we were not testing
64 in `try_struct_rseq()`.
Also added a print on registration failure in
`api.rseq/api.rseq_splitvma` to help with debugging in the future.
Tested on my local workstation with kernel 7.1.6 and glibc 2.43
- Before: `api.rseq` fails at assertion `res == 0 || (res == -1 && errno
== ENOSYS)`.
- Now: `api.rseq` fails at assertion `restarts > 0`, like the other
tests in #8159.
Issue: #8159
Changes the drmemtrace scheduler to store a different kernel template
(both switch and syscalls) per drmemtrace version. If it can't find a
template for a given input's version (for syscalls) or output's version
(for switches) it raises an error.
The scheduler's template file reading is extended to handle multiple
back-to-back versions.
Changes raw2trace to handle multi-versioned template files by ignoring
entries that are not for the current version.
Updates raw2trace_directory to support reading a zipfile-format template
file. Adds an error check for raw2trace failing to find any syscalls
(before, there was no error when the plain reader failed to read the
zipfile).
Updates the mock switch and syscall files to store back-to-back
templates for versions 6, 7, and 8.
Relaxes the view and invariant_checker complaints about varying versions
to not apply to template files.
Adds scheduler unit tests for invalid versions and multiple versions.
Adds mutiple versions to the raw2trace static injection unit test.
Adds an end-to-end test of raw2trace using the checked-in mock syscall
templates (this did not exist before: burst_syscall_inject uses its own
generated templates).
Manually tested with the new multi-version checked-in mock files. Also
did manual testing of error detection:
```
$ bin64/drrun -t drmemtrace -tool record_filter -filter_modify_marker_value 12,5 \
-outdir foo -indir ../src/clients/drcachesim/tests/drmemtrace.threadsig.x64.tracedir
Output 975332 entries from 975332 entries.
$ bin64/drrun -t drmemtrace -indir foo \
-sched_syscall_file ../src/clients/drcachesim/tests/mock_syscall_sequences.x64.zip \
-sched_switch_file ../src/clients/drcachesim/tests/mock_switch_sequences.x64.zip \
-core_sharded -cores 1
ERROR: failed to run analyzer: Failed on drmemtrace.threadsig.872894.3824.trace.zip: \
Failed to find syscall sequence for version 5 and number 15
```
Issue: #4915
When DynamoRIO attaches to a running application, it must locate the
application’s previously registered `struct rseq` in static TLS. Naming
TLS pages or changing their memory attributes can split the allocation
into multiple VMAs. Searching only the VMA containing the thread pointer
can therefore miss valid registrations. This PR extends attach-time
discovery to neighboring mappings to handle split TLS allocations.
- Search adjacent read-write mappings, stopping at gaps or non-writable
mappings. Limit extensions to 1 MB from the thread pointer while
preserving the full search of its containing mapping.
- Search outward from the thread pointer on both sides together so both
static TLS and struct pthread are checked before reaching the stack.
- Fix an off-by-one in `try_struct_rseq()` so re-registration uses the
matched registration size.
- Add a new test `api.rseq_splitvma` which places the registered `struct
rseq` in a separate VMA using `PR_SET_VMA_ANON_NAME` (with a
`MADV_DONTFORK` fallback), reproducing the original attach failure
without this fix and passing with it.
Fixes#8174
This PR is a follow up on #8169. Instead of hardcoding the kernel module
object file list, we are reusing existing `CORE_SRCS` and `ARCH_SRCS` to
generate the object file list. Overall, the current process is easier to
maintain and less error prone. It's also easier to replicate in a
different build system. The trade-offs are there are some files we don't
actually need getting pulled in, so the build time is slightly longer
(<10s longer) and the built `.ko` file is slightly larger (~130KB larger
with DEBUG ON, ~20KB larger with DEBUG OFF).
- Use existing `CORE_SRCS` and `ARCH_SRCS` defined in
`core/CMakeLists.txt` for kernel module object file list generation.
- Prevent `emit_utils.c` from being included in `ARCH_SRCS` twice when
`ARCH_NAME` and `ARCH_NAME_SHARED` are the same in
`core/CMakeLists.txt`. This isn't a bug per se: both CMake and kbuild
can handle duplicate files. I just thought this would be cleaner and
less confusing.
- Disable annotations (by `undef ANNOTATIONS`) in `configure.cmake.h`.
- Use `limits_wrapper.h` in `core/hotpatch.c`.
- Remove redundant `lib/dr_annotations.h` in `core/jit_opt.c`.
Issue: #8021
In our internal system we need to support a different build system where
CMake is not available. This PR moves several module build rules from
CMake into a checked-in Kbuild so different build systems can share the
same implementation by supplying a prepared source tree, generated
headers, and an object list.
- Replace `Kbuild.in` with a checked-in `Kbuild`. CMake generates
`dr_objs` in a separate `Kbuild.sources` file.
- Make Kbuild paths relative to the module build tree. CMake prepares
this tree using symlinks to sources and generated headers.
- Move shared include paths, defines, and C/assembly flags from CMake
into Kbuild.
- Move `x86.asm` → `x86.S` preprocessing and placeholder replacement
into Kbuild.
- Move development stub generation into Kbuild. Generate weak stubs with
function declarations from the component objects before linking,
allowing the module to build in one Kbuild invocation.
- Select assembly flags for GNU `as` or LLVM’s integrated assembler.
- Keep the existing kernel source list and add a TODO to generate the
object list from the shared core source lists.
Issue: #8021Fixes#8112
Includes the scheduler error string in errors returned to the analyzer.
Changes the analyzer's scheduler_ field to become a unique_ptr, to
better support code sharing with subclasses, including error
reporting.
Adds a per-stream error string to avoid races accessing the
overall scheduler object's string for non-initialization errors.
Adds an exit path in the scheduler's rebalance_queues() if there are no
live outputs. Withtout this, an error in one output thread can cause
another doing a rebalance to hit a floating point crash, which was hit
and confirmed fixed.
Tested on forthcoming #4915 kernel template versioning changes which
return an error if the right version is missing. Without this change,
the error showed up as a vague "Failed to read from trace". With this
change it says:
```
ERROR: failed to run analyzer: Failed on allasm_x86_64_ver5.trace.zip: \
Failed to find syscall sequence for version 5 and number 1
```
Issue: #4915
During the initial address-space scan, DynamoRIO can mistake a read-only
MAP_SHARED view of an ELF file (for example, `r--s`) for a loaded
module. It then projects the ELF’s PT_LOAD segments into its internal
module map even when those segments are not actually mapped there.
This PR:
- Validates shared ELF candidates by requiring their executable PT_LOAD
segments to be mapped at the expected addresses and file offsets from
the same file.
- Adds regression coverage for one-page and full-span flat views, plus a
valid shared PT_LOAD layout.
The validation is limited to MAP_SHARED mappings as standard ELF loaders
typically map PT_LOAD segments with MAP_PRIVATE, so normal loaded
modules do not take this path.
Fixes#8117
When the version marker is updated in the drmemtrace record_filter tool,
the version stored in the header is now updated as well.
Adds an error check in the reader that the marker version matches the
most recent header version (a new header is allowed to be different, to
support multiple versions in the same file for kernel templates).
(Previously, there was no error reported on such a mistmach.) This was
tested with the record_filter before its update here:
```
$ bin64/drrun -t drmemtrace -tool record_filter -filter_trace_types 29 \
-filter_modify_marker_value 12,8 -outdir foo \
-indir ../src/clients/drcachesim/tests/drmemtrace.threadsig.x64.tracedir/
Output 975206 entries from 975332 entries.
$ bin64/drrun -t drmemtrace -tool view -indir foo 2>&1 | head
Version mismatch: header 7 != marker 8
```
Removes redundant footer and header checks in reader_t::operator++.
Adds a record_filter unit test for the new behavior.
Issue: #4915
Extends the drmemtrace first-last-iter non-expanded repeated-string
scheme to include filters in offline mode. This is accomplished for
d-filters by inserting a clean call at the end of the loop and having it
walk the addresses and apply the d-cache filter, with raw2trace modified
to read all the entries. The same clean call is invoked on a fault
interrupting a repstring. For i-filters, raw2trace is modified to
perform its loop walk and record synthesis for this mode just like
non-filtered.
The clean call outputs the buffer if it hits the end; it rolls back the
buffer header to prevent the markers causing problems with raw2trace.
This is tested by adding a 4K OP_lods in allasm_repstr and running the
new tests with a 128-record buffer limit.
The TRACE_MARKER_TYPE_INSTRUCTION_COUNT value was incorrectly including
the nofetch records: that is fixed here as a side effect of not
expanding the repstring.
The drreg issue filed as #8153 is fixed here by removing used registers
from the drreg vector. This issue only happens with this filter change,
so it belongs in this PR.
Adds 3 new tests based on allasm_repstr with i, d, and i+d filtering.
Adds filter cases to the burst_repstr test.
Updates raw2trace_unit_tests.
Fixes#4914
Reduces the verbosity used in the scheduler_unit_tests and
raw2trace_unit_tests to remove 31K lines (!) from their output. I had to
go all the way down to 0 to accomplish this as even at 1 there are so
many tests the output adds up. These tests are designed as unit tests
which should be completely deterministic, so we should not need extra
verbosity on the Github servers.
Before:
```
$ clients/bin64/tool.scheduler.unit_tests ~/dr/git/src/clients/drcachesim/tests 2>&1 | wc
25421 182840 1556574
$ clients/bin64/tool.drcacheoff.raw2trace_unit_tests 2>&1 | wc
6607 45631 308008
```
After:
```
$ clients/bin64/tool.scheduler.unit_tests ~/dr/git/src/clients/drcachesim/tests 2>&1 | wc
564 2397 47550 OUT
$ clients/bin64/tool.drcacheoff.raw2trace_unit_tests 2>&1 | wc
139 258 2310
```
Issue: #8161
Deprecates the "non-fetched" instruction record in drmemtrace. Increases
the drmemtrace trace version number, with all new traces simply omitting
any instruction record on non-initial x86 repeated string loops.
Adds a new TRACE_TYPE_INSTR_STRING_LOOP type to make it easier to
identify string loops ahead of time, as their long sequences of loads
and stores may warrant planning in some consumers. This type is
considered type_is_instr().
Online traces and filtered-offline traces still generate fetch records,
which are removed in the reader for online and raw2trace for offline.
Future work will eliminate this for offline; doing so for online is
complex, covered by #4948, and not currently planned.
Adds buffer output for syscall templates to avoid overflow now that
no-fetch records cannot be used as output points.
Adds invariant checks that we do not see no-fetch (or maybe-fetch)
records in new traces.
These checks are once-only per shard. We do expect the no-fetch
invariant to show up with the current kernel trace templates. We plan to
separately create new templates.
Removes "(fetched)" from basic_counts output and only outputs
"non-fetched" counts for legacy traces with non-zero counts. This also
removes the non-fetched that were being counted by the opcode_mix tool.
Updates documentation and tests.
Issue: #4915
After recently enabling -hw_cache_consistency on AArch64 the test
security-common.TestAllocWE started failing on Linux >= 5.13 when run on
recent AArch64 hardware.
Issue: #7585
Fixes misc-include-cleaner warnings seen when importing recent changes
into our environment:
- `core/unix/signal_linux_x86.c`: Includes `proc.h` for `XCR0_*
andarch_exports.h` for `ZMM_ENABLED()`.
- `core/lib/instrument.c`: Includes `dr_project_wide_defines.h` for
`BUFFER_SIZE_ELEMENTS` and `NULL_TERMINATE_BUFFER`.
- `core/lib/instrument.c`: Calls `d_r_snprintf()` and `d_r_strstr()`
directly in `report_client_lib_load_error()`. include-cleaner maps any
macro named like a C standard library function to the libc header, so
the `snprintf` and `strstr` macros in `globals.h` are reported as
needing `<stdio.h>` and `<string.h>`. Using the underlying names
resolves them to their declarations in `globals.h`.
- `suite/tests/linux/sigcontext.c`: Includes `dr_project_wide_defines.h`
for `TESTANY`.
clang-tidy with misc-include-cleaner no longer reports these warnings on
the changed lines.
Issue: #4103
Building the kernel module with clang fails with `-Werror`. This PR
fixes two errors while keeping the gcc builds working.
- `kernel_interface.h`: clang does not support the `gnu_printf` format
archetype (`-Wignored-attributes`). Use `__printf__` instead, which is
portable across both gcc and clang.
- `instrument.c`: clang inlines `add_client_lib()` into
`instrument_load_client_libs()`. That combines the 2 KiB option buffer
with the 3 KiB error-message buffers from the load-failure path and
exceeds the module's `-Wframe-larger-than=4096`:
```
error: stack frame size (5176) exceeds limit (4096) in
'instrument_load_client_libs' [-Werror,-Wframe-larger-than]
```
Move the load-failure reporting into a NOINLINE helper
(`report_client_lib_load_error()`) so those buffers are on the stack
only when a client library fails to load. No functional change.
Issue: #8021
Follow up on #8123. This PR adds CPU-local DR state and the kernel OS
hooks needed to advance the module initialization through OS,
configuration heap, initial-stack, and loader setup code. Initialization
now stops before `d_r_arch_init()`.
- Add a fixed 1 KiB per-CPU buffer, with allocation and lifetime managed
by the Linux module loader.
- Validate storage capacity and alignment during `d_r_os_init()`.
- Implement local-state and dcontext accessors.
- Add a no-op `os_loader_init_prologue()`. Linux has already loaded and
relocated the module; kernel client loading remains future work.
Issue: #8021
The vector register save/restore paths (append_save_simd_reg,
append_restore_simd_reg, insert_push_all_registers, and
insert_pop_all_registers) copied eight registers at a time with LMUL=8.
An LMUL=8 group store packs the eight registers contiguously, so
register N's bytes land at &simd[N] only when VLEN is exactly 256 bits
and fills a whole dr_simd_t slot. On any other VLEN the mcontext layout
contract VREG_OFFSET(reg) == &simd[reg - VR0] is broken.
Copy one register at a time with LMUL=1 instead, so each register's
vlenb bytes always land in the low bytes of its own dr_simd_t slot for
any supported VLEN <= 256 bits.
Issue: #3544
On x86 Linux with recent Intel CPUs (e.g., Xeon 8573C and 6973P-C),
`XCR0` can contain state components, such as AMX tile data, that are
absent from a particular signal frame. Copying all `XCR0` bits into that
frame's `XSAVE` bitmap can cause `sigreturn` to fail. This PR preserves
the frame's bitmap while marking the register state DynamoRIO actually
writes.
Fixes#7996
Define DRMEMTRACE_STATIC for the drmemtrace_nomain and
drmemtrace_drstatic static variants too, so func_trace no longer skips
the application executable when recording functions.
Fixes#8148
Adds conditionals to avoid drmemtrace file writes from compression
setup, headers, and teardown when the buffer handoff callback is set.
This fixes an assert in the burst_replaceall test when a new enough LZ4
is available to be used when linking statically. Confirmed that this
fixes the test on a machine where it was failing.
Issue: #8146
Increases the default drmemtrace trace buffer size by 8x from 4K records
to 8K records. This improves SPECPU2017 INT performance with no i/o by
14% on x86; with lz4 output it is estimated from 2 runs at half that.
Although a larger buffer provides better performance, we limit the
larger default to just the main thread to better scale to thousands of
threads.
Adds snappy support for buffers larger than 64K by splitting them up
into multiple to work with snappy's built-in 64K-per-output limit.
Documents in the options and release notes that this decreases the
granularity of -trace_for_instrs and -exit_after_instrs. Tweaks a few
tests using these options whose output assumed the old 4K default so
that their assumptions on window counts and exit points are met.
Issue #8125
This commit fixes two related state corruption bugs.
When hw_cache_consistency is enabled DynamoRIO sets writable/executable
pages to read-only in order to detect any writes to code regions. If
these writes occurred whilst app state was spilled, the signal handler
failed to restore app state before returning to d_r_dispatch, which
caused app state corruption.
Similarly, if this fault happened on an instruction that uses the stolen
register, we would enter the signal handler with the stolen register
restored to its app state. The signal handler assumed it contained the
tls pointer and left things in a corrupt state.
Discovered while testing simple Java commands with hw_cache_consistency
enabled on AArch64:
drrun -- javac HelloWorld.java
With this bug fixed I am not aware of any other issues with
hw_cache_consistency on AArch64 so this commit also turns
hw_cache_consistency on by deafult on AArch64.
Issue: #7585Fixes#2055, #5771
Silences a spurious curiosity about the recursion depth of private
module loading by adding a missing decrement: it was really reaching
only depth 3 for 21 modules, but complained about reaching depth 20.
Silences a spurious curiosity that a module is the VDSO by adding a
check for soname staring with "linux-gate.so".
Silences a vmareas assert on a size 0 mmap by adding an early exit.
Issue: #8138, #6152Fixes#8138
Inverts the drmemtrace trace buffer redzone check. Instead of re-setting
the buffer to 0 every time and the redzone to -1 and identifying when to
output by reading a non-0 value, we set the redzone to hold a sentinel
record value: the value 1, which would be a load/store of the address 1,
which seems pretty unlikely (we do see 0 in prefetches). If it does
happen, we'll just output early; the output code is updated here to
clobber the 1 so the same record won't trigger again.
For online, with 12-byte records, we set the 1st pointer-sized bytes to
the sentinel 1 and the final 4 to 0. This is a write to 0, which should
also be rare.
For 32-bit, we have to keep the memset, as loads and stores assume the
top word is already zeroed, but 32-bit is not a performance priority.
Changing the clean call skip is pretty simple on arm; on x86 it adds a
redirection jump. The filter mode change checks that leverage the
redzone check are more complicated and we change those to branches that
skip over the other skips.
On SPECCPU2017 INT this provides a 2% performance improvement, but it
enables larger buffer sizes without paying a memset cost. A future
update will make the main thread's buffer 8x larger, which is a 14%
no-i/o improvement and half that with LZ4 output.
Issue: #8125
Previously in #8096, we implemented
`dynamorio_app_init_part_one_options()` in `dynamorio_app_init()` to
parse and initialize DynamoRIO runtime options when loading the kernel
module.
This PR advances kernel module initialization into part two of
`dynamorio_app_init()` through `modules_init()` (including VMM/heap, VM
areas, decoder, processor, and module initialization). The rest of part
two is left for follow-up PRs to keep this one relatively small and
reviewable.
- Call `dynamorio_app_init()` from `dynamorio_module_init()`, and add a
temporary kernel-only early return after `modules_init()` in
`dynamorio_app_init_part_two_finalize()`.
- Resolve kernel `_text` and `_end` symbols to implement
`get_application_base()` and `get_application_end()`.
- Change default heap size to 256MB and allocate heap as RWX
(`PAGE_KERNEL_EXEC`). Implement `os_heap_reserve_in_region()`,
`os_heap_commit()`, `os_heap_decommit()`, and
`os_heap_get_commit_limit()`.
- Update `os_check_option_compatibility()` to configure VMM placement
for the pre-allocated module heap (`reachable_heap`, `vm_base_near_app`,
`vmm_block_size`, and clamping `vm_size`) and disable reset options via
`DISABLE_RESET()`.
- Switch `LINUX_KERNEL` from `HAVE_MEMINFO_MAPS` to
`HAVE_MEMINFO_QUERY`, implement `is_readable_without_exception*()` using
`copy_from_{user,kernel}_nofault()`, and stub out `all_memory_areas`
helpers.
- Implement `os_modules_init()`/`exit()`, `report_diagnostics()`, and
`diagnost_exit()` as no-ops in kernel mode.
Issue: #8021
On the first execution, we need to nop out a store's memory operations
but not its register side effects if it's mid-sequence: but not if it's
the final instruction (committing store). We relax the existing fatal
error on a store with multiple destinations if it's the final
instruction.
Adds a test case that fails without the fix.
Also tested on our internal application where this was first hit.
Fixes#8136
Adds two new drmemtrace options: -main_buf_records and
-trace_buf_records, controlling the buffer size (in record count) for
the main (initial) thread and subsequent threads. The main thread has a
different size to facilitate forthcoming increases (after we invert the
redzone check) without adding space for every thread.
Tested:
```
$ bin64/drrun -t drmemtrace -offline -main_buf_records 32K -verbose 2 \
-- suite/tests/bin/pthreads.pthreads
T177260 trace buffer size=262144 max=266240 redzone=4096
T177274 trace buffer size=32768 max=36864 redzone=4096
T177275 trace buffer size=32768 max=36864 redzone=4096
```
Issue: #8125
Kernel thread ID helpers return CPU numbers, which can be zero and
collide with `INVALID_THREAD_ID`. In debug builds, when module
initialization happens on CPU 0, DR will mistake an unowned mutex (whose
`owner` equals `INVALID_THREAD_ID`) for a mutex owned by the current
CPU, causing heap initialization to fail.
- Change `d_r_get_thread_id()` to return CPU ID plus one, and have
`get_sys_thread_id()` use the same helper.
- Add XXX comments about alternative approaches.
- Link issue #8131 to the existing XXX comments above
`INVALID_THREAD_ID`.
Fixes#8131
Adds a fastpath to the AArch64 memset using DC ZVA when setting to 0.
This speeds up drmemtrace by 2.3x on SPECCPU (see
https://github.com/DynamoRIO/dynamorio/issues/8125#issuecomment-5722352941),
as drmemtrace zeroes out its trace buffer for its redzone scheme. This
was a major cause of why AArch64 drmemtrace was so much slower than x86
drmemtrace.
Adds memset unit tests.
The core_unit_tests memset times compared to libc with the -1 removed
(so just 0) went from:
```
our_memset_time: 751
libc_memset_time: 12
```
To:
```
our_memset_time: 18
libc_memset_time: 11
```
Adding back the -1 it went from:
```
our_memset_time: 1458
libc_memset_time: 31
```
to:
```
our_memset_time: 635
libc_memset_time: 30
```
Further tested by successfully running SPECCPU (results at
https://github.com/DynamoRIO/dynamorio/issues/8125#issuecomment-5722352941).
Issue: #8125
Adds elision of AArch64 stack pointers by treating like the stolen
register and removing the displacement. This lets them use the existing
basic no-displacement and elision machinery. The newly added
GPR-generalized pre-post-indexed elision support takes over from there.
This speeds up SPECPU2017 INT by 10%.
Adds test cases to the burst_traceopts and raw2trace_unit_tests tests.
Fixes#4898
In drmemtrace, the buffer load added prior to a memory-accessing
instruction ends up being superfluous if that instruction's memref(s)
are all elided. Here we remove it to reduce overhead. Measurements show
this has on average just a 1% impact, mostly in the noise: presumably
because the buffer pointer is always in the L1 cache. It is still
worthwhile to shrink the instrumentation size and remove unnecessary
loads.
In the same vein of small instrumentation improvements: also adds a
conditional to limit delayed instruction counting to offline mode; this
was observed to cause extra code run during tracing on instructions
beyond the first one (but no extra instrumentation: instrument_instr has
a flag causing it to short-circuit).
Tested:
In the allasm_aarch64_prefetch app we have prefetches whose addresses
are all elided, so the superfluous loads really jump out:
Before:
```
0x0000000041d10058 f9800000 prfm $0x00 (%x0)
0x0000000041d1005c f940d782 ldr +0x01a8(%x28)[8byte] -> %x2
0x0000000041d10060 f8800000 prfum $0x00 (%x0)
0x0000000041d10064 f940d782 ldr +0x01a8(%x28)[8byte] -> %x2
0x0000000041d10068 f9800001 prfm $0x01 (%x0)
0x0000000041d1006c f940d782 ldr +0x01a8(%x28)[8byte] -> %x2
0x0000000041d10070 f8800001 prfum $0x01 (%x0)
0x0000000041d10074 f940d782 ldr +0x01a8(%x28)[8byte] -> %x2
0x0000000041d10078 f8800001 prfum $0x01 (%x0)
0x0000000041d1007c f940d782 ldr +0x01a8(%x28)[8byte] -> %x2
0x0000000041d10080 f9800002 prfm $0x02 (%x0)
0x0000000041d10084 f940d782 ldr +0x01a8(%x28)[8byte] -> %x2
0x0000000041d10088 f8800002 prfum $0x02 (%x0)
0x0000000041d1008c f940d782 ldr +0x01a8(%x28)[8byte] -> %x2
0x0000000041d10090 f9800002 prfm $0x02 (%x0)
```
After:
```
0x00000000492f0058 f9800000 prfm $0x00 (%x0)
0x00000000492f005c f8800000 prfum $0x00 (%x0)
0x00000000492f0060 f9800001 prfm $0x01 (%x0)
0x00000000492f0064 f8800001 prfum $0x01 (%x0)
0x00000000492f0068 f8800001 prfum $0x01 (%x0)
0x00000000492f006c f9800002 prfm $0x02 (%x0)
0x00000000492f0070 f8800002 prfum $0x02 (%x0)
0x00000000492f0074 f9800002 prfm $0x02 (%x0)
```
Issue: #8125
Extends the drmemtrace offline trace elision optimizations to support
any general-purpose base register being modified by an add, a subtract,
an x86 LEA, or an arm pre-indexed or post-indexed addressing mode.
This was measured to speed up x86 SPECCPU2017 INT by 2.3%: so a modest
speedup on top of stack push/pop elision which is already in place.
Adds test cases to burst_traceopts.
Adds test cases to raw2trace_unit_tests, which required adding
convenience macros for INSTR_CREATE_{ld,st}{r,p}_imm_{pre,post}index().
AArch64 still does not elide stack pointer bases: that is under #4898
and will be resolved separately, but it needs this pre-and-post-indexed
tracking in order to work.
Issue: #4913
Implement an AArch64 version of insert_selfmod_sandbox() modelled after
the x86 version.
The existing AArch64 self-modifying code support works by mangling ic
cache maintenance instructions to flush code cache fragments containing
invalidated addresses. This works fine for code which uses ic
instructions but there are two cases where cache maintenance
instructions are not required which this does not address:
- Hardware with coherent icaches where software is not required to
perform explicit cache maintenance.
- Concurrent Modification and Execution of Instructions (CMODX).
Sandboxing adds support for these cases by adding a check for
modifications at the beginning for fragments corresponding to writable
memory.
The new behaviour is currently off by default on AArch64 but can be
enabled with the -hw_cache_consistency option. From my testing I believe
that the new mangling code is correct but I have seem some app data
corruption crashes stemming from the signal handling code where an app
writes to a page that DR has made read-only to detect writes. I am
intending for fix this issue in a separate PR. Once this is fixed and
further testing shows no more bugs -hw_cache_consistency will become the
new default behaviour on AArch64.
AI tools were used to prototype and debug parts of this commit but all
generated code has been reviewed and revised by me. The documentation
changes are handwritten.
Issues: #2055, #5771, #7585
For non-filtered drmemtrace offline traces, disables unrolling of
repeated string loops. The first iteration's address(es) are recorded
before the rep-string, and the last iteration's are recorded *after*. A
nop is appended to avoid complexity if a block ends in a rep-string
(unlikely.
Raw2trace now unrolls the loop. It removes the records if it detects a
zero-iteration loop (something we could try to optimize to avoid writing
records; left for future work). Otherwise, it fills in between the start
end end addresses, handling the different types of string operations and
directions. It outputs the block-record buffer when needed to avoid
overflow.
This speeds up SPECCPU 2017 gcc ref subrun 5 by 10x! It also speeds up
leela by 18%.
If a rep string loop is interrupted by a handled fault, the tracer
detects this in the kernel xfer event and on decoding and identifying a
rep string, it inserts the end address(es). Raw2trace is updated to
support a partially-interrupted rep string. It also detects a rep string
that never started and removes the instruction (and encoding, if added)
and inserts a TRACE_MARKER_TYPE_UNCOMPLETED_INSTRUCTION marker.
Bumps the raw trace version so we can tell whether to expect this.
Adds raw2trace unit tests for regular, legacy, and filtered cases.
Adds a new burst_repstr test targeting interrupted string loops.
Further tested manually on allasm-repstr with its new backward loop: the
final trace was diffed versus the code base before this PR and found to
be identical except thread id and timestamp values.
Updates the instcount test to pass a lower window count to keep the same
behavior with its rep loop becoming a single instruction in the counter.
Updates the windows-asm test output as its block boundaries changed and
so its window boundaries changed.
Future work includes adding d-filter support via filtering in a clean
call after the loop, so we can then remove the no-fetch records for all
offline, and have the reader remove for online (#4915).
Issue: #4914
- Add an `options` module parameter to accept DynamoRIO options on
`insmod` and initialize them via `dynamorio_app_init_part_one_options()`
during module init.
- Added functions in `core/kernel_linux/os.c` that
`dynamorio_app_init_part_one_options()` depends on. They are exercised
by the init path and tested in the insmod CI.
- Implement `kernel_setenv()` for saving the options value.
- Implement `os_check_option_compatibility()` to enforce kernel-mode
options constraints.
- Update the kernel module insmod workflow to build with both
`-DDEBUG=ON` and `OFF`, and verify that runtime options (`-loglevel 2`)
are parsed and logged correctly in the debug build.
On configuration lookup: the shared init path still performs it, but
`os_open()` is intentionally not implemented in kernel mode and returns
`INVALID_FILE` unconditionally. So no config file can be found and
options can only be set using the module parameter.
The kernel-mode options constraints implemented in
`os_check_option_compatibility()` are mostly copied from drk and may
need future adjustments. For example, #8116 tracks CPU-private fragment
restrictions (`shared_bbs`, `shared_traces`, `coarse_units`).
This initial implementation only supports setting options at module load
time. Dynamic updates and per-process options are tracked by #8115.
Issue: #8021
The safe-read labels are hidden in assembly but not in their C
declarations. On ARM, this creates GOT-based address references that can
become stale when BOLT moves `safe_read_asm()`, preventing fault
recovery.
Mark the declarations hidden to generate direct references to ensure no
GOT entries are created for them. Tested on a large internal application
that uses BOLT.
Fixes#8087
When the decode cache is cleared on a filter endpoint, raw2trace needs
to also clear its last block fields.
Adds a unit test that fails without this fix.
Fixes#8110
Adds elision of the stack pointer base for repeated push/pop
instructions by tracking the displacement changes. Only supports x86
push/pop for now.
Adds reconstruction in raw2trace, using stack pointer tracking and an
instru_offline_t helper to detect elision breaks.
Bumps the raw trace version.
Adds burst_traceopts test cases to confirm the actual addresses in the
final trace match the non-optimized version.
Adds a raw2trace unit test sanity test case.
On SPECCPU2017 INT, this was found to improve performance by 9% on
average.
Issue: #4913
Adds the missing 8-byte versions INSTR_CREATE macros for MOVS, CMPS,
SCAS and their repeating counterparts, along with the 8-byte repeating
versions of STOS and LODS. Simplifies the memory sizes used in the
macros to avoid confusion by removing rex8.
Adds test case to api.ir_x86.
These are wanted to build tests for the drmemtrace repstr bound
optimizations.
Issue: #4914
Refactors the raw2trace scatter/gather skipped-memref code into a
separate helper function to keep the code easier to read and maintain.
This is a prelude to adding rep string expansion in raw2trace for
parallel helper function.
Issue: #7914, #4914
Fixes glibc struct rseq detection heuristic, which is broken after
recent kernel changes that increased the size of the struct
(https://lwn.net/Articles/885818). This affects DR mid-run attach uses
where DR did not have a chance to observe glibc's original rseq syscall
that registered its struct.
The struct rseq was 32 bytes with padding before, which was supposed to
be passed as rseq_len to the rseq syscall. Now rseq_len may be
getauxval(AT_RSEQ_FEATURE_SIZE), so we need to try multiple sizes. As we
do not want to depend on the getauxval library call in DR code, we
hardcode the value actually seen in a more recent environment where the
prior try_struct_rseq failed.
See https://lwn.net/Articles/1033957/ for more details.
Fixes: #8100
Replace the output-based custom command with an always-run target, so
kbuild checks source and header dependencies on every build.
Prebviously, header-only edits will not trigger a rebuild, which leaves
a stale .ko file.
Note that this doesn't mean every `cmake --target dynamorio_module`
rebuilds everything from scratch: kbuild itself still tracks
dependencies and re-compiles only affected ordinary objects. In other
words, the changes in this PR simply delegate dependencies tracking to
Kbuild.
Fixes#8097
This test assumes that the final output will be "Hello, world!" but if
the application shutdown code reaches a threshold there may be an
additional "Hit retrace threshold" message. This happens consistently on
CentOS 9 and SLES 15.
Update the .templatex file to allow optional output after "Hello,
world!"
Issue: #8079
- Set up a new GitHub workflow that (1) builds the kernel module against
the kernel inside the runner, and (2) loads/unloads the kernel module
and checks whether the expected logs are printed in `dmesg`
- Disable objtool validation with `OBJECT_FILES_NON_STANDARD := y` when
building the kernel module.
Our x86.asm doesn't adhere to kernel objtool/SLS conventions. The GitHub
runner kernel has `CONFIG_OBJTOOL_WERROR` enabled, so objtool reports
issues as errors instead of warnings.
- Add -mfunction-return=keep flag when compiling the kernel module.
By default the kernel build system adds `-mfunction-return=thunk-extern`
for Spectre mitigation, which generates `jmp __x86_return_thunk` instead
of `ret` for every function return. When objtool validation is disabled,
it doesn't generate the `.return_sites` section in the ELF section,
which is needed for the mitigation to work. `-mfunction-return=keep`
tells gcc to generate `ret` instead of `jmp __x86_return_thunk`.
- Switch to `__vmalloc_node_range` for kernel module heap allocation.
Before we were only testing again kernel 6.6, and the GitHub runner has
a newer kernel which has `module_alloc()` removed. After some research I
come to the conclusion that across 6.6+ LTS kernels,
`__vmalloc_node_range` is the only reliable API to allocate writable
memory within the 2GB reach of kernel text
(`MODULES_VADDR..MODULES_END`). Alternatives:
- `module_alloc()`: Removed in 6.10. Returns RW+NX.
- `execmem_alloc()`: Introduced in 6.10. Returns RW+NX in 6.10-6.12, but
read-only-execute (ROX) in 6.13+.
- `execmem_alloc_rw()`: Introduced in 6.14. Returns RW+NX.
- Auto-generate weak stubs for undefined symbols so the module loads
during incremental development (`generate_stubs.cmake`). When
`Module.symvers` is present, the build runs `nm -u` on the composite
object, filters out kernel-exported symbols, and generates a weak
`BUG()` stub for each remaining symbol — so an unexpected call at
runtime fails loudly with a kernel oops. When `Module.symvers` is
absent, stub generation is skipped and the build relies on
`KBUILD_MODPOST_WARN=1` for compile-only testing.
Issue: #8021.
Suyash Mahar