Running the suite on GCC Compile Farm machines found test scripts that
don't work as intended on some hosts:
- on_qemu() took KVM guests on POWER for qemu-user
- the native POWER10 target ignored CPU and shared ppc64le's directory
- readelf 2.35 appends "[<localentry>: 1]" to POWER10 function names
- compressed debug sections hid .debug_str from `readelf -p`
- some tests assumed Linux, glibc, GCC-only flags or g++
- an x86-64 executable was run without $QEMU on a non-x86 host
- old binutils lack multiple .symver and SHF_LINK_ORDER
- GCC 8's `gcc -r` links start files into relocatable output
- debug-macro-section.sh grepped for an offset format objdump doesn't print
With -g3, GCC emits a COMDAT group for the .debug_macro contents of each
header file, and other .debug_macro sections refer to it. mold doesn't
deduplicate these groups because it would resolve references to discarded
copies to 0 (https://github.com/rui314/mold/issues/438), but -r output
dropped the groups while keeping SHF_GROUP on their members. Now -r keeps
every copy as a COMDAT group. GNU ld -r deduplicates them and redirects
the references to the kept copies; without that redirection, mold leaves
deduplication to the final link, where GNU ld, lld and mold handle the
groups as they would in the original objects.
--print-dependencies reads the relocations of every input section,
including non-allocated ones whose CREL tables are not decoded when the
file is read. It read them as an array, which reinterpreted such a table
as RELA records and failed an assertion. C++ mold's get_rels() decoded
the table on demand instead.
A read-only pass doesn't need the table materialized, so iterate with
the streaming relocation iterator as --stats does, and document when
the array accessor can be used.
rustc's LoongArch targets use the medium code model, whose calls reach
±128 GiB, but the cc crate passes no -mcmodel to the C compiler unless
-Ccode-model is given explicitly, so C code got the normal code model,
whose direct calls reach only ±128 MiB. A debug build of mold has more
than 128 MiB of text, so calls from C code such as zstd, mimalloc and
lto-message.c to the PLT failed to link with R_LARCH_B26 overflows on
a native LoongArch host. We now give -Ccode-model=medium explicitly.
A section group whose flag word is 0 is not a COMDAT group. It is never
deduplicated; it only ties its members together so that they are kept or
discarded as a unit. annobin emits such groups, e.g. into crt1.o on
AlmaLinux 8. mold ignored them, but -r output kept SHF_GROUP on their
members, leaving SHF_GROUP sections that no SHT_GROUP lists, which the
gABI forbids and objdump 2.30 rejects. -r output now keeps each such
group with flag word 0, as GNU ld and lld do. Other links still treat
its members as ordinary sections.
CREL tables of non-allocated sections are not decoded when an object
file is read; their relocations are streamed when they are applied.
On ppc64v1, rewrite_opd() also rewrites relocations of non-allocated
sections such as .debug_info, and rels_mut() reinterpreted such a table
as RELA records. Linking a CREL object built with -g failed an
assertion, and a table whose size was a multiple of 24 would be misread.
A CREL table can't be rewritten in its compressed form, so rels_mut()
now decodes it into the side table first, as get_rels() in C++ mold
did. relocation_iter() then returns the rewritten records from there.
This reverts aed5a38d1, which made -e accept a number as an entry
address if no symbol has that name, as GNU ld and lld do. New features
are deferred until after 3.0.0.
This reverts f03b74595, 48428d389 and 3a1b9c6c2, which merged
GNU_PROPERTY_AARCH64_FEATURE_1_AND, emitted BTI PLTs and landing pads
for range extension thunks, and added -z force-bti, pac-plt, gcs,
bti-report and gcs-report.
GCC 16 no longer records -mbranch-protection in .note.gnu.property but
in AArch64 build attributes, which we don't read, so outputs silently
lost the BTI, PAC and GCS bits and -z force-bti warned about every
object file. We'll bring the feature back with build attribute support
after 3.0.0. Until then, ARM64 outputs have no .note.gnu.property, as
in 2.42.1.
These sorts have keys that are unique among the elements, so a stable
sort produces the same order. Using an unstable sort, as C++ mold did,
avoids the stable sort's buffer and signals that ties don't matter.
Two mapping symbols can share an offset, e.g. when `ld -r` places a
section ending with a mapping symbol before one starting with another.
The later one in the symbol table applies, as in the C++ version, which
used stable_sort. An unstable sort could leave BE8 code unswapped.
Default features are turned off because uuid's std feature refers to
optional wasm32-only crates, and Cargo puts those into Cargo.lock even
though nothing builds them. With them off, uuid is the only new package.
When run in an extracted source archive, dist.sh cloned the release tag
as a bare repository into .git to get the commit hash and timestamp.
Cargo refuses to build in a directory whose .git is a bare repository,
and the embedded hash named the tag rather than the files actually
built. A worktree doesn't work either, because its .git file points
outside the build container and the hash is silently omitted. Stop with
an error unless .git is a directory.
Non-allocated strings were merged in a background Rayon task, but Rayon
has no task priorities: a thread waiting for a foreground parallel pass
could pick up a long piece of the merge, and the pass waited up to
200 ms for it. Resolve these sections in compute_section_sizes as before
2316c952a. Within noise on large debug builds (0.94-1.04).
Instruction pairs such as ADRP+ADD on ARM64 refer to the same string
with two consecutive relocations, and each got a dummy symbol of its own.
Reuse the previous relocation's dummy symbol if it refers to the same
section symbol with the same addend. Since both relocations of a pair now
refer to one symbol, ADRP+ADD relaxation applies to them too. On
chrome-release-arm64, reattach_section_pieces 39 -> 30 ms and peak RSS
5.86 -> 5.56 GB.
Range extension thunk creation and removal lay out input sections in a
serial loop that looked up each member through its file, costing a few
dependent cache misses per section. Read sizes and alignments into an
array in parallel first, and publish the offsets in parallel. On
chrome-release-arm64, compute_section_sizes 34 -> 27 ms and
remove_redundant_thunks 25 -> 15 ms; on chrome-debug-arm64, 87 -> 81 ms
and 72 -> 50 ms.
CIE deduplication, which both ICF and .eh_frame layout run, compared each
CIE with every distinct CIE found so far in a serial loop. Compute their
SipHash digests in parallel with a random key and number the classes
through a hash table instead, assuming that digests never collide as ICF
does. FDE offsets are now assigned in parallel too. uniquify_cies 9 -> 2
ms on chrome-release-arm64; .eh_frame layout 15 -> 8 ms on
chrome-debug-x86_64.
set_visibility() is only called on exclusively owned symbols, but it
updated the bits with a compare-and-swap loop. Take &mut self and write
them directly.
merged_resolve_members visited every file on one thread. Fold the
members of each file in parallel and append the per-thread lists in
file order. About 5 ms on chrome-release-arm64.
gather reserved each shard for every recorded occurrence of a name, about
three times the number of distinct names, so the sparse table touched far
more pages than it used. Estimate the distinct names with HyperLogLog
instead. chrome-debug-arm64 peak RSS 26.76 -> 25.91 GB.
Pattern::matches called memchr::memmem::find for the literal after each
star, which builds a searcher on every call. Build them when compiling
the pattern instead. apply_version_script on tensorflow-debug 260 -> 220
ms.
compute_import_export collected updates in parallel and applied them on
one thread because Symbol's attribute bits needed &mut. Make the bits an
AtomicU16 so that file-parallel passes can set them through shared
references, testing a bit before writing it as mark() does.
parse_symbol_version now finds versioned symbols in parallel; it still
applies them in input order because one may read a version that another
has just set. compute_import_export on tensorflow-debug 167 -> 9 ms.
Assemblers emit a call to a static function in another section as a
branch to a section symbol plus an offset, but a range extension thunk
jumps to the start of a symbol. On RELA targets, give such a branch a
symbol at its destination and a zero addend, as we do for references to
section fragments. This fixes Chromium's ARM64 debug build, which
9b9162146 made an error and 2.42.1 linked into calls to wrong addresses.
mold needs Rust 1.95 for std::hint::cold_path and the atomic update
methods, while its dependencies need only 1.85. Declaring it makes Cargo
reject older toolchains with a clear message before building, and lets
the resolver avoid dependency versions that need a newer rustc.
This reverts 6f8dbb64b. mimalloc uses 64-bit atomics, which GCC
implements as calls into libatomic on hosts such as 32-bit PowerPC.
The need comes from linking mimalloc statically into mold, so the
executable is the right place to add libatomic. Our mimalloc_rust fork
is pinned to the commit before its libatomic change, so the fork
differs from upstream only by the bundled mimalloc version.
This reverts bd97ebc90. mimalloc's page map still doesn't cover the
sign-extended high half of the sparc64 address space, but mimalloc maps
its arenas at hinted addresses in the low half, and since v3.5.0 a page
outside the map fails to register instead of corrupting memory. On an
UltraSPARC T5 running Linux, mold with mimalloc 3.5.3 passes the native
test suite.
mimalloc_rust now links libatomic itself on targets without 64-bit
atomics, so cli/build.rs, which did that for 32-bit PowerPC hosts, is
no longer needed.
readelf before binutils 2.39 can't decode version 9 indices and exits
with an error, so gdb-index-dwarf5-types ignores its exit status as
check_index already does. arch-x86_64-eh-frame-hdr-order links without
the C runtime, whose objects may have FDEs of their own.
mold handles .ARM.exidx and other users of SHF_LINK_ORDER directly
rather than implementing the flag in general. This reverts 49129fbd5,
which kept SHF_LINK_ORDER sections alive with their linked sections, and
restores the ARM-specific GC edge to .ARM.exidx. 9aacb62c0 now sets
sh_link only in relocatable output, which needs it to preserve the flag.
GCC emits no pubnames for type units, so the index listed a type only
for the compilation units referring to it by signature. GDB 18 doesn't
follow such a reference when it looks a type up through the index, so
ptype failed for types defined in type units. Read the name of the type
each type unit defines from its DIEs, resolving their section offsets
with input relocations, and add it to the index for the type unit.
A non-PIC object referring to an absolute or undefined weak symbol with
a PC-relative relocation, such as AArch64's ADRP, can't be linked into
a position-independent output, but recompiling it with -fPIC fixes that
by going through the GOT. The error message lost that hint when it was
reworded, which broke abs-error on aarch64_be.
od -w is a GNU extension, so arch-x86_64-eh-frame-hdr-order failed on
FreeBSD. head and grep -q exit before reading all of their input, so
under pipefail the producer may die of SIGPIPE and fail the pipeline,
which happened to mergeable-entsize-order on FreeBSD.
Clang gives the STT_TLS type to the labels of TLSDESC code sequences,
which are in .text. With --emit-relocs they are written to .symtab,
where their values were computed as offsets from the start of the TLS
segment, which underflowed. Write the address of an STT_TLS symbol
outside a TLS section as it is.
Assemblers emit a call to a static function as a relocation against a
section symbol plus an offset. A range extension thunk jumps to the
start of a symbol, so taking such a branch through a thunk landed at
the wrong address. Report it as an error instead.
Relocations in a relocatable output may refer to local symbols such as
RISC-V's .L labels, so removing them with -x, -X or -s left relocations
pointing to symbol 0. -r now ignores -x and -X, and -s strips only debug
info as -S does.
In the extreme code model, the pcalau12i with R_LARCH_TLS_GD_PC_HI20 or
R_LARCH_TLS_LD_PC_HI20 begins a 64-bit sequence whose lu32i.d and lu52i.d
supply the bits beyond ±2 GiB, so the GOT may be farther away than that.
The other *_PC_HI20 relocations were already applied without a range
check, as in GNU ld and lld, but these two rejected such a GOT with an
out-of-range error.
In the extreme code model, lu32i.d and lu52i.d follow pcalau12i by 8 and
12 bytes, and the upper 32 bits they set are relative to the pcalau12i's
page. We used page(P - 8) for both, which is the next page for lu52i.d if
the pcalau12i is the last word of a page. The result was wrong whenever
the borrow reached bit 52; e.g. a pcalau12i at 0x100000ffc materialized
0xfff0000080000000 instead of 0x80000000.
mold crashed at startup on sparc64 Linux, which maps user memory at
sign-extended high addresses. mimalloc v3 registers pages in a map that
covers only 48-bit addresses, so the first registration wrote out of
bounds. Until the bundled mimalloc handles that layout, sparc64 hosts use
the system allocator (https://github.com/rui314/mold/issues/1639).
mold -run looked for mold-wrapper.so in /usr/local/lib/mold unless it was
next to the executable or in ../lib/mold, and install-mold.sh always put
it in $PREFIX/lib/mold, so a package using lib64 or a multiarch libdir
could not make -run find it. The C++ version took the directory from
CMake's libdir. MOLD_LIBDIR now sets it at build time and install time.
util::prefetch assumed SSE on all x86 targets, but i586-unknown-linux-gnu
has none, so mold died with SIGILL on CPUs such as the Pentium, K6,
Vortex86 and Quark. The x86 prefetch is now compiled only when the target
has SSE. Such a target simply ignores the hint, as other targets without
a prefetch instruction do.
Versions of a symbol have the same name, so they tied in the unstable sort
of exported dynamic symbols. Rayon breaks ties with a pseudo-random
generator whose width follows usize, so .dynsym and its hash and version
tables differed between 32-bit and 64-bit hosts. The input order now breaks
the tie. .eh_frame_hdr had the same problem with FDEs for one function,
which are now ordered by their addresses.
On a 32-bit host, linking an output of 2 GiB or more panicked, because
the doubled mapping length wrapped around and sizes of 4 GiB or more were
truncated to usize. Each panic also left a multi-GiB temporary file. We
now check the size before creating the file and report a fatal error.
The limit is 2 GiB, as no Rust object can be larger than isize::MAX bytes.
A range extension thunk jumps to its destination with `br x16`, which
faults under BTI if the destination doesn't begin with a landing pad, as
is the case for a static function that is only called directly. For such
a function, we now put `bti c; b <function>` at the beginning of the first
thunk after the function's section, which is always within direct branch
range, and other thunks jump there instead, as lld's landing pad thunks do.
-z force-bti and -z pac-plt force the BTI and PAC bits of the output's
GNU_PROPERTY_AARCH64_FEATURE_1_AND, and -z pac-plt also makes PLT entries
authenticate .got.plt entries with autia1716 and sets DT_AARCH64_PAC_PLT.
-z gcs=implicit|never|always controls the GCS bit. -z bti-report and
-z gcs-report diagnose input files lacking the features, and as in lld,
-z force-bti and -z gcs=always imply the warning level.
We used to merge only x86 properties in .note.gnu.property, so an ARM64
output lost GNU_PROPERTY_AARCH64_FEATURE_1_AND even if all its inputs were
compiled with -mbranch-protection. We now AND-merge it as GNU ld and lld do.
If the output is BTI-enabled, the PLT header and the PLT entries that may be
reached by an indirect branch start with `bti c` as lld's do, which makes PLT
entries 24 bytes long, and DT_AARCH64_BTI_PLT is set.
R_RISCV_ALIGN has no symbol, and its addend is the size of the NOP
padding that a relaxing link shrinks to realign the code after it.
symidx_addend returned (0, 0) for any symbol not written to the output
symbol table, including the null symbol, so `mold -r` wrote R_RISCV_ALIGN
with addend 0 and a later link could no longer realign the code. We now
keep the addend of a relocation without a symbol, as GNU ld and lld do.
LoongArch's R_LARCH_ALIGN refers to a local symbol and was not affected.
GCC 14 uses R_LARCH_TLS_DESC64_PC_LO20 and R_LARCH_TLS_DESC64_PC_HI12
with -mcmodel=extreme -mtls-dialect=desc to extend the TLSDESC sequence's
GOT access to 64 bits, and we rejected them as unknown. We now apply them
like the other 64-bit PC-relative pairs and don't rewrite the sequence's
LO12 instruction with pcaddi. As GNU ld and lld do, we don't relax the
sequence to IE or LE, except in static executables, which can't use
TLSDESC and therefore get LE.
is_got_load_pair checks that a GOT load's PCREL_LO12_I refers to the
label at its AUIPC by comparing the label's value with the AUIPC's
r_offset. Symbol values are adjusted for removed bytes before relocations
are applied, but r_offsets are not, so a GOT load after a shrunk
instruction was no longer rewritten to AUIPC+ADDI. We now compare with
the label's st_value, as find_paired_reloc does.
R_RISCV_32_PCREL, R_RISCV_PLT32 and R_RISCV_GOT32_PCREL in allocated
sections were truncated to 32 bits without a range check, so a reference
to a symbol or a GOT slot more than 2 GiB away silently got a wrong
value. We now report an out-of-range error for them, as we already do
for R_RISCV_32_PCREL in .eh_frame and as lld does.
The .riscv.attributes parser knew the value types of only the three tags
mold uses, so the value of any other tag was read as a tag. For example,
an unknown tag 40 with value 5 was mistaken for Tag_RISCV_arch. As per
the psABI, an even tag has a ULEB128 value and an odd tag has a
NUL-terminated string, so we now skip other attributes by that rule.
They are still dropped from the output, as lld does for unknown tags.
The output's e_flags were the first input's plus EF_RISCV_RVC, so an
object compiled for the TSO memory model (-march=rv64gc_ztso) lost its
EF_RISCV_TSO bit if it came after an ordinary RVWMO object. RVWMO code
runs correctly on TSO hardware but not vice versa, so the output now has
the bit if any input has it. GNU ld does the same; lld still takes the
bit from the first file.
-Map wrote address zero as "0x0" where C++ mold wrote "0", and
--print-dependencies omitted the blank line that separated the comment
header from the dependency lines. Both outputs are now byte-identical
to those of C++ mold.
With MOLD_JOBS=1 and an empty XDG_RUNTIME_DIR, mold joined "" with
"mold-lock" and created the lock file in the current directory, so
processes in different directories did not serialize. Like other XDG
variables, an empty value now means unset, and mold falls back to
/tmp/mold-lock-$USER.
-rpath-link, -fuse-ld, -O, --sort-section and --max-cache-size are
accepted only for compatibility, and their values are never used. mold
nevertheless required the values to be UTF-8, so a non-UTF-8 directory
given to -rpath-link was a fatal error. Read the values as raw bytes.
The manual said -z separate-code was the default, but mold defaults to
-z noseparate-code and adds no padding between segments unless
--section-order is given, which implies -z separate-loadable-segments.
The manual documents --no-color-diagnostics, but mold rejected it as an
unknown option; accept it as lld does. Conversely, the manual and --help
listed -z nodefs and -z start_stop_visibility, which neither mold nor
GNU ld accepts; the options are -z undefs and -z start-stop-visibility.
--help also said --color-diagnostics means always, while it means auto.
--dynamic-list-data, --lto-pseudo-probe-for-profiling and a bare
--thinlto-index-only take no argument in GNU ld and lld, but mold read
the next command line argument as their value, silently dropping an
input file placed there. GNU ld's --package-metadata takes an optional
argument, so mold now accepts its value only after an equal sign.
Rust's io::Error appends " (os error N)" to the strerror text when it is
displayed, so mold printed "cannot open foo.o: No such file or directory
(os error 2)" where the C++ version printed just the text. Diagnostics
now format I/O errors with a strerror helper that returns the text alone
and falls back to the usual formatting for errors not from the OS.
scan_pcrel, which also handles GOT-relative relocations, told the user
to recompile with -fPIC whenever it couldn't resolve a relocation, even
though code using GOTOFF is PIC already. It now reports the reason: a
symbol at a fixed address, such as an absolute or undefined weak one, in
a position-independent output, or a preemptible symbol defined in the
shared object being created, which -Bsymbolic or hidden visibility fixes.
`-e 0x401000` was looked up only as a symbol name, so the link warned
that the entry symbol was undefined and set e_entry to 0. GNU ld and lld
fall back to parsing the argument as a number (decimal, 0x hex or octal)
when no symbol has that name, and so does mold now.
When a version script named the same symbol in two version nodes, each
exact-name match overwrote the previous one, so the last node won.
GNU ld and lld give the symbol the first node's version (lld warns about
the reassignment, and GNU ld rejects a name that is both local and
global). Apply only the first exact match. Wildcards still let the last
matching node win, as in both linkers.
A version script or --default-symver on a link that exports no dynamic
symbols, such as a static executable, made verdef::construct index the
empty .dynsym and panic. Symbol versions exist only for dynamic symbols,
so write no version sections in that case, as GNU ld does.
Undefined symbols are allowed by default with -shared or with
--unresolved-symbols=ignore-all, and a non-weak hidden one was then
silently resolved to 0. A hidden symbol can't be resolved at runtime,
so report it as an error regardless of those options, as GNU ld and lld
do. Weak undefined hidden symbols still resolve to 0.
--defsym=foo=bar with bar defined by a shared library copied bar's
library-relative value into an absolute foo, so calling foo jumped to a
bogus address. Such a symbol has no link-time address, so report it the
same way as an undefined one. GNU ld and lld reject it as well.
The first file on the command line owned a common symbol, so `int buf[4]`
in one file and `int buf[64]` in another gave a 16-byte buf, and writes
to it clobbered its neighbours. As GNU ld and lld do, let the largest
common own the symbol and align it to the largest alignment among them,
for .common, .tls_common and -r output alike.
--export-dynamic and --dynamic-list export symbols even from an
executable that links against no DSO. Such an executable has no
.dynamic, but the symbols were still added to .dynsym, and .dynstr
lacked its leading empty string, which only the creation of .dynamic
adds, so entry 0 of .dynsym got a symbol's name. Like GNU ld and lld,
emit no dynamic symbol table without .dynamic.
Mergeable sections that share a name, type and flags but differ in
entry size are merged into separate output sections. Their sort keys
tied, so their order depended on which one the parallel conversion of
input files happened to create first. Break the tie with the first
member's file priority and section index, as for output sections.
add_synthetic_symbols defined __start_ and __stop_ symbols for every
allocated chunk with a C identifier name, including the ELF and program
headers named EHDR and PHDR. fix_synthetic_symbols skips headers, so
the symbols were left in .symtab with value 0, and -z
start-stop-visibility=protected exported them. Exclude headers in
start_stop_name, which both functions use.
compute_address_significance marked exported symbols' sections as
address-taken by walking .dynsym, but .dynsym is not populated until
after ICF, so the loop did nothing and --icf=safe gave identical
exported functions of a DSO the same address. Walk each object file's
defined global symbols instead, whose is_exported bits are already set.
DT_INIT and DT_FINI refer to the --init and --fini functions, but only
the entry point was a GC root. If such a function was not exported
because it was hidden or made local by a version script, --gc-sections
discarded its section, DT_INIT and DT_FINI were emitted with value 0,
and the DSO crashed on load. Make them GC roots as GNU ld and lld do.
The $a, $t and $d symbols that mold creates for PLT entries and thunks
were STT_FUNC, copied from the symbols naming the entries, while AAELF
requires mapping symbols to be STT_NOTYPE. push_mapping_symbol now
builds the symbol from a section index and an address itself.
Only a final link converts big-endian ARM code to BE8, so relocatable
output keeps the input's BE32 code, but it was marked EF_ARM_BE8 anyway.
GNU ld rejected such an object as "already in final BE8 format", and
disassemblers decoded its code in the wrong byte order. -r also failed
with "--be32 is not supported" unless --be8 was given, which GCC and
Clang never pass with -r.
A 32-bit Thumb instruction is a pair of halfwords, each in the byte
order of the input's code, but NOP.W was written as a single 32-bit
word. On big-endian targets, that swapped its halfwords, so a Thumb call
to an undefined weak symbol and a TLSDESC call relaxed to local-exec
became garbage in BE8 output. NOP.W is now written a halfword at a time.
Clang assembles ADR and LDR (literal) referring to a symbol in another
section to R_ARM_ALU_PC_G0 and R_ARM_LDR_PC_G0 in ARM code and to
R_ARM_THM_ALU_PREL_11_0 and R_ARM_THM_PC12 in Thumb code, which failed
with "unknown relocation", as did 8- and 16-bit data. The offset's sign
picks ADD or SUB, or the U bit of LDR, and the Thumb forms are relative
to the PC aligned down to 4, producing the same bytes as GNU ld.
TBZ/TBNZ to a label in another section, .2byte/.4byte data referring to
a symbol, and GOT loads compiled with -mcmodel=tiny -fPIC failed with
"unknown relocation". TSTBR14 reaches ±32 KiB and is range-checked like
GNU ld and lld do, without thunks. ABS16 and ABS32 are too narrow for
dynamic relocations, so they must be link-time constants in PIC output.
GOT_LD_PREL19 is an LDR literal that loads the symbol's GOT slot.
Relocations copied by --emit-relocs refer to symbols in .symtab, which
--strip-all omits, so the combination crashed or produced relocation
sections pointing to a nonexistent symbol table. Report an error as
lld does. (GNU ld instead keeps the symbols the relocations need.) -r
sets --emit-relocs internally and keeps accepting --strip-all.
An SHF_LINK_ORDER section's sh_link refers to the section it describes,
but mold always wrote zero. Once passed through -r, such sections lost
their links, so relinking with --gc-sections dropped .ARM.exidx and
__patchable_function_entries, and C++ programs on ARM aborted when
throwing. Set sh_link from the members' linked-to sections. Since it
can name only one section, -r also separates SHF_LINK_ORDER sections
whose linked-to sections are in different output sections, as GNU ld
and lld do.
An SHF_LINK_ORDER section such as __patchable_function_entries holds
metadata for the section that its sh_link refers to, and usually
nothing else refers to it. --gc-sections therefore discarded all of
them except .ARM.exidx, for which there was an ARM-specific edge from
a section to its unwind index. Replace that edge with a generic one
from a section to the SHF_LINK_ORDER sections linked to it, as GNU ld
and lld do.
Allocated fragments of mergeable sections start out dead under
--gc-sections and are revived by the garbage collector, but mold -r
returns before running GC. As a result, `-r --gc-sections` discarded
every .rodata.str* and .rodata.cst* piece and left the relocations
referring to them with symbol index 0. Since mold doesn't collect
garbage for -r, ignore the option there. GNU ld and lld do collect
garbage with -r, taking roots only from -e and -u.
R_SPARC_WDISP16 is used by branch-on-register instructions such as brz,
whose 16-bit word displacement is split into d16hi at bits 21:20 and
d16lo at bits 13:0. We ORed the fields into the upper halfword of the
instruction instead, corrupting its opcode, and limited the displacement
to 64 KiB instead of 128 KiB. Place the fields as GNU ld does.
R_SPARC_OLO10 carries a second addend in the upper 24 bits of r_info,
which is added after the low 10 bits of S + A are taken. RelocSection
built its output records with ElfRel::new(), which zeroes that field, so
`ld [%g1 + %lo(var) + 8]` lost its +8 in -r and --emit-relocs output.
Build each record from a copy of the input one instead.
The 8- and 16-bit GOT and TLS relocations fill the displacement of a
(d8,An,Xn) or (d16,An) operand, which the CPU sign-extends, but we checked
them against unsigned ranges. A GOT offset of 32 KiB or more was thus
silently encoded as a negative displacement, and a local-exec TLS offset,
which is usually negative because the thread pointer is biased by 0x7000,
was rejected. Check them as signed as GNU ld does.
SH4 object files keep relocation addends in the relocated places even
though the relocations are RELA, and RelocSection already follows that
convention. EhFrameRelocSection, however, put adjusted addends only into
r_addend, which SH4 readers ignore, so after `ld -r` every FDE's pc_begin
pointed at the start of its output .text and C++ exceptions terminated.
Write the addends into .eh_frame as we do for REL targets.
This enables the three -r exception tests that were skipped on SH4.
On SH4, r2 carries the address of the buffer for a struct return value.
Our PLT header loaded the link map into r2 and set r0 to 0, which glibc's
resolver accepts, but the resolver then restores r2 to its entry value, so
a lazily bound function returning a struct overwrote the link map. The
header now passes the link map in r0 as GNU ld does, borrowing r2 only
until the jump's delay slot restores it from the stack.
A section folded by ICF has no output section, so with --emit-relocs a
relocation against its section symbol was emitted with symbol 0, and one
in .rela.eh_frame, such as an LSDA pointer, hit an expect(). Symbols
defined in a folded section are not in .symtab either, so references to
them lost their symbol or, in .rela.eh_frame, got a wrong one. Emit both
relative to the leader's output section, as the relocated contents and lld
do.
When relaxing TLSDESC to initial-exec, we write an AUIPC to the
TLSDESC_ADD_LO12's slot and a load to the TLSDESC_CALL's slot, but we
computed their GOT-relative offsets from the TLSDESC_HI20's address. The
two coincide only if the instructions in between are deleted, which is not
the case for objects lacking R_RISCV_RELAX on TLSDESC relocations, such as
clang 18's, so the load read a wrong GOT entry. Use the AUIPC's address,
as lld does.
With --emit-relocs, a TLSDESC_HI20 relaxed to IE or LE was rewritten to
R_NONE as soon as it was applied. The TLSDESC_LOAD_LO12, TLSDESC_ADD_LO12
and TLSDESC_CALL that follow find it with find_paired_reloc(), which
matches by type, so such links failed with "paired relocation is missing".
Rewrite TLSDESC_HI20s after all relocations have been applied. LoongArch is
unaffected because its TLSDESC relocations refer to the symbol directly.
shrink_sections adjusted each symbol by the r_deltas of its own input
section, but a section folded by ICF is never shrunk, so its symbols kept
their pre-relaxation offsets while being resolved against the shrunk
leader. With LoongArch GCC's leading R_LARCH_ALIGN nop, calls to a folded
function landed 4 bytes past the leader's entry. A folded section was
identical to its leader before relaxation, so use the leader's deltas for
both symbol values and st_size.
GCC calls _savegpr0_N, _restgpr0_N, _savefpr_N and similar functions
from -Os prologues and epilogues and expects the linker to provide them.
We synthesized them only for ELFv2, so ELFv1 programs built with -Os
failed to link with undefined symbols. ELFv1 calls them with plain
R_PPC64_REL24 relocations, not through function descriptors, and
expects the same code, so emit the same section in target byte order
and move its instruction table from ppc64v2.rs to the section's module.
R_PPC64_GOT16, _LO, _HI and _HA take the offset of a GOT slot from the
TOC base, but we subtracted the TOC base from the slot's offset within
.got rather than from its address, so the result was off by the address
of .got. GCC addresses data through .toc instead, so only hand-written
assembly using @got was affected.
With -fpic, PPC32 code loads a GOT slot with one instruction whose
R_PPC_GOT16 relocation is a signed 16-bit offset from the GOT, and
R_PPC_GOT_TLSGD16, R_PPC_GOT_TLSLD16 and R_PPC_GOT_TPREL16 are the
same. We truncated these offsets silently, so a program whose GOT grew
past 32 KiB linked but loaded the wrong slots. Report an error instead,
as the psABI requires overflow checks for these fields.
A static executable's startup code applies the R_IRELATIVE relocations
between __rela_iplt_start and __rela_iplt_end, and we placed the start
by counting only the GOT slots of IFUNC symbols. On PPC64, a data
reference to an IFUNC symbol gets its own R_IRELATIVE instead of the
canonical PLT address, so the range began too late and skipped a GOT
slot, and a call through that slot ran the resolver. Count them too.
A global symbol defined only in a member of a discarded COMDAT group has
no owning file, so it has nothing to resolve to. The relocation scan
reports a reference to it and skips the relocation, but --noinhibit-exec
downgraded the error to a warning and the relocation was applied anyway,
which panicked on a missing GOT or TLS entry on most targets. Report the
reference as a fatal error in that case.
The relocation scan skips a relocation to an undefined symbol when that
is an error, because the link fails anyway. --noinhibit-exec reports the
error as a warning and writes the output, so relocations that needed a
GOT, PLT or TLS entry were applied without one, and mold panicked on the
missing index. Such relocations are now scanned as they are with
--warn-unresolved-symbols, so the symbol gets the entries it needs and
resolves to 0, as it does in GNU ld and lld.
Every target crashed on a -fPIC object reading an undefined variable,
an undefined TLS variable, or both.
GNU as emits R_RISCV_ALIGN and R_LARCH_ALIGN for an alignment directive
in an executable NOBITS section. With -r, RelocSection sliced the output
section's bytes at each relocation before calling write_addend, but a
NOBITS section has no bytes in the file, so the slice was out of range.
Since only REL targets and SH4 keep addends in section contents, take
the slice only on those targets.
GAS and Clang emit R_RISCV_64 and R_LARCH_64 for `.quad sym` even in
ELF32 objects. On RV32 and LA32, they are not the word-size absolute
relocation that the output section applies, so they reached
apply_reloc_alloc, which had no case for them and panicked. Handle them
like other non-word-size absolute relocations, as lld does: scan them
with scan_absrel and write S + A as a 64-bit value.
With a plugin that lacks get_symbols_v3, mold re-executes itself with
the same command line. The new process applies -C again, but it was
started in the directory the first one had already changed to, so a
relative -C failed with "chdir failed". Remember the directory before
-C and return to it before re-executing.
Input file names already have --chroot applied. Relative thin archive
members and relative names in linker scripts are resolved against such
a name, but the result was opened with --chroot applied again, so the
link failed. Open these paths as they are. Also write the --repro
archive next to the output in the chroot rather than outside of it.
--repro reopened each input by its recorded name. Response files and
--retain-symbols-file are opened before -C changes the directory, so a
relative name no longer found them and the link failed, and a name with
--chroot applied got the prefix twice. Archive the contents already in
memory instead: no input has been rewritten in place at this point.
An existing output file is reused by writing it in place. When that file
is also a memory-mapped input, as in `mold a.o -o a.o` with an a.o larger
than 32 KiB, its private mapping sees the truncation and the writes while
the input is still being read, so mold crashed with SIGBUS or produced a
broken output. Such a file is now replaced with a fresh one. Inputs are
matched by device and inode, which covers archives and thin members too.
An absolute path in a linker script is prefixed with the sysroot if
the script is in the sysroot. is_in_sysroot returned false for an
empty sysroot, so such a path fell through to the -L search, and an
incompatible file was skipped with a warning. C++ mold treats an empty
sysroot as containing every file and opens the path as is, reporting
an error if it cannot be used. GNU ld and lld don't search -L either.
--dynamic-list and --export-dynamic-symbol-list files were opened by
the driver after parse_args had already changed the directory for -C,
so a relative path was looked up in the new directory. C++ mold reads
them in the option loop, as we already do for --retain-symbols-file.
Open them there too and keep the mapped files in DynamicListSource;
the driver still parses them, since the parser needs a Context.
A panic aborts the process in release builds, skipping the cleanup that
a fatal error or a SIGSEGV/SIGBUS does, so a failed link left its
.<output>.<pid> file behind. Install a panic hook that removes the file
and then calls the default hook, so the panic message, backtrace and
abort stay as they were. The hook runs on the panicking thread, so it
also covers rayon workers and the extern "C" LTO plugin callbacks.
Arguments are passed on as C strings, so a NUL byte in the -plugin path,
a -plugin-opt or the -o name made loading the LTO plugin panic. A
response file is the only way to get a NUL byte into an argument, as
the OS passes arguments as C strings too, so reject one when reading a
response file instead of at each C string conversion.
mimalloc uses 64-bit atomics, which GCC implements as calls into
libatomic on hosts such as 32-bit PowerPC. The mimalloc crate links
libatomic only on ARM, so mold failed to link there, which CI's cargo
check did not catch. Link libatomic when the target lacks 64-bit
atomics, as the C++ version did. It is a link argument rather than a
library because rustc would place the latter before mimalloc, where
--as-needed drops it.
libz-sys declares the length parameter of adler32_combine() as
libc::off_t, which is 32 bits on 32-bit glibc hosts, but Debian and
Ubuntu now compile C code for armhf and powerpc with 64-bit off_t, so
the bundled zlib reads a 64-bit length there. Every zlib-compressed
debug section larger than one 1 MiB shard got a wrong checksum, and
readelf, objcopy and gdb rejected it. Compute the combination in Rust
instead; it is a few lines of modular arithmetic.
apply_reloc_alloc read the 4-byte instruction at every relocation to
extract its register fields, so a 1- or 2-byte data relocation such as
R_SPARC_8 or R_SPARC_DISP16 in the last bytes of a section panicked.
Read the fields only for the relocations that rewrite an instruction.
i386 aligns 64-bit integers to 4 bytes, so Symbol and NameRecord are
smaller there than on other hosts, and the assertions that they are
exactly 48 and 16 bytes failed to compile. Assert upper bounds instead,
which still catch growth on 64-bit hosts, and check the i686 build in
CI.
std has no AtomicU64 or AtomicI64 on hosts such as 32-bit PowerPC,
32-bit MIPS and pre-v6 ARM, so mold did not compile there, whereas the
C++ version fell back to libatomic. Take those two types from
portable-atomic, which uses the std types where they exist and falls
back to locks elsewhere. Also check the 32-bit PowerPC build in CI; the
existing armv7 check did not catch this because armv7 has 64-bit
atomics.
See https://github.com/rui314/mold/discussions/1643#discussioncomment-18603572.
The .tbss layout loop gained a second end-of-list check next to the
existing one, for a case that cannot happen: .tbss is never the last
chunk. DigestMap::new was rewritten with repeat_with, which took five
lines to say what one line did. Restore both.
Follow-up to https://github.com/rui314/mold/pull/1685.
FileRecord::size was size_of::<Self>(), and record sizes were written
both ways. Use size_of everywhere. Unlike the trait method, it is a
const fn, so it also works in the compile-time assertions.
https://github.com/rui314/mold/pull/1683 unified the spelling in the
other direction.
An ignored SIGCHLD is inherited across exec, e.g. from `trap '' CHLD`
in a shell. The child that fork_child creates is then reaped
automatically, so the parent's waitpid fails with ECHILD. The parent
ignored that and read the untouched zero status as a successful exit,
so a link that failed exited with status 0. Report a failure instead.
Also read and write the completion byte through File, which removes
two raw libc calls.
Based on https://github.com/rui314/mold/pull/1682.
Both callers of from_hex already check that the byte is a hex digit,
so the new assertion was redundant. The utsname rewrite traded
MaybeUninit::assume_init for mem::zeroed without removing an unsafe
block, so restore the original code.
The fix it tests is target-independent, and only its last check was
x86-64-specific. Count the dynamic relocations against foo and bar
instead of looking for R_X86_64_64. PowerPC is skipped because GCC puts
the read-only pointers in writable sections there, so nothing is
promoted.
LLVMgold calls get_symbols with a null pointer and a zero count for an
IR file that has no symbols, e.g. one compiled from an empty source
file. slice::from_raw_parts does not accept a null pointer even for a
zero length, so the debug build aborted on its precondition check.
Use an empty slice for a zero count, in add_symbols as well.
Based on https://github.com/rui314/mold/pull/1680.
Building the thread pool fails when threads cannot be created, e.g.
under a low `ulimit -u`. That hit an expect, so mold aborted with a
panic message rather than reporting the error like other system
errors.
Taken from https://github.com/rui314/mold/pull/1678, which also
added checks for malformed inputs that are not included here.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Input files are opened in parallel, and each file is appended to the
file pool when it is opened, so the pool's order changed from run to
run. --dependency-file and --repro list files in that order. Return
the pool sorted by name instead.
The dependency-file tests expected a space after a.o, i.e. that a.o
was not the last dependency. That failed intermittently on Alpine,
where libc.so is not a linker script and no dependency reliably comes
after a.o.
report_out_of_range looked the record up again through the file's
relocation table while apply_reloc_alloc held that table mutably, which
created a shared view of memory behind a live unique reference, and it
walked the CREL iterator from the start to do so. Every caller has the
record at hand, so pass it in. The loop indices that only served the
lookup are gone with it.
Cherry-picked from https://github.com/rui314/mold/pull/1674.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
CieRecord stored a shared slice over its .eh_frame relocation records,
but reattach_fragment_relocations later rewrites those records in place
through the private input mapping, and the FDE and CIE accessors read
through the stale slice afterwards. The record now keeps only the
relocation section index and reads the current table through
ObjectFile::relocations, as decoded CREL tables already did. The raw
relocation view no longer claims a 'static lifetime.
Cherry-picked from https://github.com/rui314/mold/pull/1674.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
mold accepts a multi-letter option with a single dash, but the option
chain is first-match, and some single-letter options were tried before
long options starting with the same letter. "-entry=main" was thus read
as "-e ntry=main", "-filter libf.so" as "-f ilter" followed by an input
file, and "-fix-cortex-a53-843419" left a bogus DT_AUXILIARY entry. GNU
ld and lld read all of these as the long option.
Move each single-letter option past the last long option it shadows.
Conversely, GNU ld and lld read "-execute-only" as "-e xecute-only" and
"-omagic" as "-o magic", because those names take double dashes only.
Spell such names with double dashes in the option table, which
match_option now honors, replacing the rule that all names starting
with "o" need two dashes. Where GNU ld and lld disagree, follow GNU ld.
Thanks to @lu-zero for investigating this issue and proposing a fix in
https://github.com/rui314/mold/pull/1670.
Fixes https://github.com/rui314/mold/issues/1671
Zip the glob NFA's three parallel arrays and fill a bracket range with
one slice fill, merge the two glob module headers left over from two C++
files, check SipHash's output width at compile time and split its input
into words with as_chunks, mark Timer must_use so a dropped one is a
compile-time warning, remove a Windows TODO from the C++ demangler that
never applied to this implementation, and fix a typo in a comment.
The merged glob header no longer lists qt_private_api_tag* as an
example of an Aho-Corasick pattern, since prefix patterns are matched
directly rather than by the Aho-Corasick matcher.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Each output section promoted the imported symbols of its read-only
absolute relocations and classified its relocations in one pass, while
other sections ran in parallel. Whether a writable section's relocation
against the same symbol became a dynamic relocation therefore depended
on which section ran first, so the output changed from run to run. In a
non-PIE that references an imported variable and function from both
.rodata and .data, 37 of 100 links in one measurement emitted two extra
R_X86_64_64 dynamic relocations.
Collect all sections' relocations, promote, and only then classify, so
that the result is the same in every run: the word is resolved at link
time whenever a copy relocation or canonical PLT entry makes its address
known.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The --undefined names were cloned only because get_symbol takes the
whole context mutably, so intern them through the symbol table instead.
The section-word reversal swaps whole words, the .tbss layout loop no
longer indexes past the last chunk, the has_lto_obj predicate replaces
an inline copy of itself, and the --exclude-libs ALL test is hoisted out
of the file loop. The dynamic symbol groups are extended in place, the
IRELATIVE count loses a temporary, and thunk addresses zip symbols with
their offsets.
In ICF, the digest no longer converts to little-endian on the way into
a hash whose key is random per run, the digest map is filled with
repeat_with, and HashMap is imported once.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
apply_reloc_nonalloc on ARM32, PPC32, m68k and SH-4 read the relocation
section of a non-allocated section as an array of ordinary records. A
CREL table of a non-allocated section is not decoded ahead of time, so
for an object with SHT_CREL relocations those bytes are compressed
records: linking a PPC32 or m68k object that Clang 19 built with -g and
-Wa,--crel panicked with a failed length assertion in
records_from_bytes. Use the relocation iterator, which decodes CREL
tables, as the other targets already do.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A relocatable link keeps every COMDAT group's members in output
sections of their own, so an object with many groups of one name, such
as the .debug_macro groups GCC emits with -g3, produces output sections
whose sort keys tie. The tie fell back to the order in which the
parallel creation phase happened to make them, which changed from run
to run: 30 runs of the link in relocatable-compressed-debug-info.sh
produced 13 different outputs. Break the tie with the first member's
file priority and section index, which are in input order.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
C++ mold adds the _savegpr0_* family after the __start_ and __stop_
symbols, and the internal object's symbol order is the order in the
output .symtab. Match it so that ppc64le outputs get the same .symtab,
and hence the same build ID, as C++ mold's.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
C++ mold defines these target-specific segments after PT_GNU_RELRO, but
the port defined them before PT_GNU_STACK. ARM and RISC-V outputs
therefore had their program headers in a different order than C++
mold's for the same input, and a different build ID. Define them where
C++ mold does.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The four places that write a target word through an IS_64 branch
(.got, .got.plt, .relr.dyn and an output section's absolute
relocations) now use the existing Word::<E>::new(value).write(slot).
The padding filler walks its buffer with chunks_exact_mut; thunk
symbols zip their two parallel vectors; relr_offsets concatenates its
shards once; the .ARM.exidx lookup uses find_map instead of position
plus a second match, and text_end borrows the section headers instead
of copying them; .gnu.hash looks each symbol's hash up once;
merged-section resolution takes the timers by destructuring instead of
cloning them; OutputSectionId::new folds its two checks into one; the
.sframe writer slices each FRE block by start and length; WORD_SIZE is
derived from the word type like IS_64; and ElfDyn derives what its
siblings derive.
The doc comment on ChunkHeader's local-symbol fields no longer starts
with a stray "For --gdb-index", and a few comment typos are fixed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
shndx_at_in was identical to shndx_at, is_discarded_comdat re-implemented
is_discarded_comdat_sym, ObjectFile's Display duplicated display_file,
rels_mut looked its table up twice, find_null re-implemented a chunked
position search, the archive member name reader re-implemented cstr_at
and memchr, the file type probe indexed the header table it was
iterating, and the linker script reader spelled slice patterns as length
checks plus indexing.
The loop in is_dwarf32 now compares the end of the next unit header with
the section size instead of subtracting its start from the size, which
wrapped around when a unit's size field pointed past the end of a
malformed section.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The command-line and result types of the per-target link functions were
spelled out in the twenty target crates, the executable and the driver.
Name them Cmdline and LinkResult in the driver and use the names
everywhere else.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The record views in elf.rs relied on a debug assertion, and therefore on
the FileRecord contract alone in release builds, for the alignment that
their pointer casts require. An inline const assertion is evaluated per
instantiation, so a record type that violates the contract no longer
compiles.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Several comments compared the code to C++ mold's arena storage and
chunks, or named functions that do not exist in this implementation
(SharedFile<E>::mark_live_objects, Symbol<E>::get_addr and
open_file_impl); point them at the Rust functions or drop the
comparison. The ICF and thunks modules repeated their description in a
second module-level summary, and the TLS module docs had a stray title
line. drop_mappings did not say that it discards the relocation records
rewritten in place, and the madvise in write_build_id did not say why
dropping written pages of the output loses nothing. Also fix three
typos.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The page-aligned start of the interior range was turned back into a
pointer with an integer cast, which leaves it without the allocation's
provenance. Derive it from the caller's pointer with with_addr instead,
as the rest of the function already reads addresses with addr().
Co-Authored-By: Claude Opus 5.5 <[email protected]>
record_comdat_owner and set_visibility_bits hand-rolled fetch_min and
update. record_comdat_owner keeps its initial load so that a file that
does not lower the priority still leaves the cache line shared. The
packed visibility and flags bytes are no longer pub, since nothing
outside symbol.rs touches them and a direct store would corrupt the
state bits they share.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
x86-64 and i386 defined their own little-endian write helpers with the
same bodies as write_ul16, write_ul32 and write_ul64 in util::endian,
and i386 read REL addends with inline from_le_bytes calls. Call the
shared functions directly, as the other fixed-endian targets do. The
ARM64 and RISC-V instruction accessors now delegate to them, like
LoongArch's already do.
Also drop an unused parameter of relax_tlsdesc_to_ie, a redundant mask
on SPARC relocation types (the record keeps the type in a byte of its
own), two TRAP comments that described C++ target structs instead of
the instruction, and the pub on two PPC64 ELFv1 functions that only
their module uses. SH-4 reports an unknown relocation with error! like
every other target, so that all of them are listed in one run.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The key length was an UnsafeCell whose reads and write needed unsafe
blocks justified by the release/acquire pair of the key pointer. A
relaxed atomic expresses the same protocol, compiles to the same loads
and stores, and takes the probe comparisons out of unsafe code. Also
check the probe invariant at compile time and reject keys longer than
32 bits instead of truncating their lengths.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
driver is a public module, so `pub` made copy_chunks part of the
library's API although only the crate itself calls it; make it
pub(crate). from_hex now asserts its precondition in debug builds
instead of silently producing garbage for a non-hex byte.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Six flags in the ignored-for-compatibility chain were unreachable
because earlier arms already consume them: color-diagnostics,
eh-frame-hdr, fatal-warnings, enable-new-dtags, disable-new-dtags and
warn-once.
returns_etxtbsy zeroed a MaybeUninit<utsname> and then asserted it
initialized after calling uname. utsname consists of byte arrays, so a
zeroed one is already a valid value; start from that and drop the
assume_init step.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The runner starts each script in its own process group, so a script
that read the terminal would stop there until the timeout. Give it
end-of-file instead.
run already returns ExitCode, so report setup errors through it
instead of exiting the process. Also drop the second sort of the
discovered scripts (make_jobs sorts the combined list), the unused
Clone on TestJob and the intermediate vector in the PATH search, and
give the unsafe kill a SAFETY comment like the other unsafe blocks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
R_X86_64_GOTPC32, R_386_GOTPC, R_390_GOTPC and their variants compute
GOT + A - P, the distance to the start of the GOT section rather than
to an entry in it. They were grouped with the GOT-indirect relocations,
so every link that referenced the GOT base got an entry for
_GLOBAL_OFFSET_TABLE_ that nothing ever read, plus a RELATIVE
relocation to fill it in a PIE.
R_ARM_REL32 was in the bucket of relocations that need no scanning,
so a reference to a symbol defined in a DSO was silently resolved as
if the symbol were at address zero. Route it through scan_pcrel like
PREL31 and MOVT_PREL, which gives imported data a copy relocation,
an imported function a canonical PLT entry, and a shared object an
error.
R_X86_64_GOTOFF64 and its counterparts on i386, ARM32, s390x and SH4
compute S + A - GOT, which is a link-time constant only if the symbol
is in the output file. We didn't scan them at all, so a reference to a
symbol defined in a DSO was silently resolved as if the symbol were at
address zero. Route them through scan_pcrel, which already makes the
right decision: a copy relocation for imported data, a canonical PLT
entry for an imported function, and an error in a shared object.
Fixes https://github.com/rui314/mold/issues/1668
The port kept C++ idioms in which a raw pointer or an unchecked index
stood in for a slice, guarded by an assertion. Add SyncUnsafeCell, a
slice-of-cells view for parallel passes, and use it for the file
mapping, the symbol-table allocators, the symbol editor and decoded CREL
tables. Make the fields that parallel passes write atomic, which turns
allocate_aux, the dynsym pass, the gdb_index prefix scan and the COMDAT
signature slots into safe code. Write merged sections and the gdb_index
tables through split slices, and look up files and sections with plain
bounds checks.
This removes 58 unsafe blocks and 16 assertions.
check_duplicate_symbols and check_symbol_types looked every reference
up in the symbol table, a cache miss each in a large link, before checks
on the file's own entry that skip it anyway. An undefined reference is
never a duplicate, and one without a type never mismatches.
Assisted-By: Claude <[email protected]>
R_AARCH64_TLSLE_LDST{8,16,32,64,128}_TPREL_LO12_NC were unhandled, so
objects using them failed to link with "unknown relocation". The immediate
is the local-exec TLS offset scaled by the access size.
addr_with decoded the tagged origin word into its five-variant enum up
front and matched it twice, which left the enum spilled to the stack on
every relocation. It only needs to know whether the origin is a fragment
or an input section, and each is a mask and a compare.
Assisted-By: Claude <[email protected]>
The check read every referenced symbol's ELF entry from its owning file,
a second cache miss per reference, only to get the type the symbol
already caches.
Assisted-By: Claude <[email protected]>
Sizing .rela.dyn walks every GOT symbol, a cache miss each, and the
section is sized three times on the way to the final layout. The walk
was the only serial one of that size left before the copy.
Assisted-By: Claude <[email protected]>
scan_abs_relocations already collects every R_X86_64_64 in an allocated
section, and apply_abs_rels applies it. Resolving the symbol again here
only to fall through an empty match arm is wasted work.
Assisted-By: Claude <[email protected]>
Six of the fifty relocation kinds need G, but every relocation paid for
its aux-table lookup and bounds check. Move the computation into a
closure, as the other targets already do.
Assisted-By: Claude <[email protected]>
A section's address and the GOT base do not change while its relocations
are applied, but the section offset is atomic and the context carries
interior mutability, so the compiler must reload both after every store
into the output buffer. Read them before the loop instead.
Assisted-By: Claude <[email protected]>
InputSection stores its contents as a pointer and sh_size. For a
compressed section, sh_size is the Chdr's uncompressed size while the
pointer led to the compressed bytes, so reading contents() before
uncompress() made a slice far longer than its buffer. Only a debug_assert
guarded this, and a crafted SHF_COMPRESSED .eh_frame reached it through
the .eh_frame parser.
Store a null pointer for compressed and NOBITS sections instead, so that
a nonnull pointer always has sh_size bytes behind it, and pass the
compressed bytes to uncompress() from the file's section header. The gABI
forbids compressing allocated sections, so reject them with an error.
Parallel writes into an output section went through OutputBuffer, a raw
pointer whose ranges were computed by the caller and only debug-asserted.
For ARM32 BE8 byte swapping those ranges came from mapping-symbol values
in the input file, so a corrupted symbol wrote outside of its section.
Replace OutputBuffer with for_each_member, which splits the buffer at
member offsets with rayon::iter::split and split_at_mut and hands each
member its own slice. Thunks get their slices by splitting the buffer
after the member loop, and the ARM32 swap indexes mapping-symbol ranges
within the member's slice. No unsafe code remains in either file.
Word-size absolute relocations were applied after the copy pass through
a raw pointer into the output section, with only a debug_assert on the
offset. A corrupted r_offset therefore wrote past the section in release
builds instead of failing.
Apply them in the per-member copy loop instead, indexing the member's
slice so that an out-of-range r_offset panics. abs_rels is built in
member order, so each member's relocations are one run of the vector;
AbsRel now records the member index and write_to finds the run with
partition_point. This removes write_word and its SAFETY assumption that
input relocations do not overlap.
The 80k-section tests take about 2 seconds alone but 13 to 14 seconds
on every target when the suite runs 128 jobs at once, so any other load
on the machine pushed some of them past 30 seconds. The limit exists to
catch hangs, and 60 seconds still does that promptly.
Layout carried only the file format: byte order, word size and the
record forms that differ between ELF32 and ELF64. No type implemented it
without also implementing Target, so the split earned its keep only in
RISC-V and LoongArch, where the record types depend on a const generic
and could not be chosen inside one Target impl.
Those two targets now take their record types from Class<IS_64>, a
helper private to the target module that maps the const parameter to
the ELF32 or ELF64 forms. ARM32 and SH4 lose their per-byte-order impls
and read section header words through u64::from like the other 32-bit
targets.
What the trait enumerates are targets, not architectures: Ppc64V1 and
Ppc64V2 share an architecture and differ in ABI, and Arm64 and Arm64Be
differ only in byte order. The architecture-level notion already has its
own name, Family. The rest of the tree already said target, in
Arm64Target, TargetTraits, emulation_to_target and the comments, so the
trait was the one place the vocabulary broke. The workspace directory
arch/ keeps its name because target/ at the repository root is Cargo's
build directory.
common.inc sets pipefail, and grep -q exits as soon as it finds a match.
If the producer still has output to write, it dies of SIGPIPE, the
pipeline fails, and a test that gates on the match silently skips or
fails. Plain grep reads its whole input, so the pipeline's status is
grep's own.
Clippy's match_same_arms lint reports 15 pairs. Most sit in relocation
tables that follow the relocation number order, where merging would
break the order, so only the two that are not in such tables change.
Apply clippy's redundant_pub_crate suggestion. An item in a private
module is crate-visible at most, so `pub(crate)` on it says nothing
that `pub` does not.
Apply clippy's ptr_as_ptr, borrow_as_ptr, ref_as_ptr and
ptr_cast_constness suggestions. `ptr.cast()` and `ptr::from_ref()`
cannot silently change the pointee type when a declaration elsewhere
changes, and `&raw mut x` passes a pointer to C without creating an
intermediate reference.
rewrite_opd looks up the symbol of a function descriptor by its offset
with a binary search. Aliases share a descriptor, so the search could
land on any of them, and a section-relative reference to a static
function then went through one of its global or weak aliases: it became
preemptible and resolved to whichever definition won. Use the lower
bound instead; the stable sort keeps the local symbol first.
`dist.sh arm` failed in the networked fetch step: cargo could not read
its git database for the mimalloc dependency ("Value too large for
defined data type"). ext4 hands 64-bit directory offsets to the QEMU
process, the armhf glibc reports EOVERFLOW when they don't fit a 32-bit
off_t, and the libgit2 built into the 32-bit cargo uses that interface.
Only the git dependency is affected; everything else goes through Rust's
std, which uses the 64-bit calls.
Keep cargo's cache on a tmpfs inside the networked container and have it
`cargo vendor --locked` into target/dist-vendor-$arch together with the
printed config. The offline build reads the vendored sources through
`--config` and no longer needs a cargo home at all.
Tests: ./dist.sh x86_64
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_0141irCwtPmPMZa9s7pCN3Cz
Two struct-size assertions hold only with 8-byte pointers: SymbolAux ends
in a Vec and PendingComdatSignature starts with a fat pointer, so they
are 48 and 16 bytes on a 32-bit host and rustc rejected the crate for
armv7. Gate them on target_pointer_width = "64"; the other five size
assertions have no pointer-sized fields and pass on both widths.
CI now runs cargo check for armv7-unknown-linux-gnueabihf over the whole
workspace so this cannot regress unnoticed.
Tests: cargo check --release --target armv7-unknown-linux-gnueabihf --workspace --all-targets
Tests: cargo test --release (pass=5681 skip=654 fail=0)
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_0141irCwtPmPMZa9s7pCN3Cz
The bullseye targets (aarch64, arm, ppc64le, s390x) left apt on the live
deb.debian.org mirrors. Those mirrors dropped the architectures Debian LTS
does not support when bullseye entered LTS, so `dist.sh ppc64le` and
`dist.sh s390x` fail in `apt-get install` with "Unable to locate package
build-essential". The pinned snapshot the images ship commented out in
sources.list still has them, and using it also gives these builds the
reproducibility the header comment already claims.
Make the snapshot switch the default apt_setup; riscv64 (deb822 sources)
and loongarch64 (live sid) keep their own.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_0141irCwtPmPMZa9s7pCN3Cz
Retain the original versioned reference when following undefined symbols
in shared libraries. This prevents an unversioned archive definition from
causing unnecessary extraction while preserving symbol interposition.
Skip the retained undefined references in the secondary symbol type check
and port both upstream regression tests unchanged.
Validation: full workspace tests passed with 34 unit tests and 5,684
integration tests across 14 target configurations; 651 skipped, 0 failed.
Four additional targets lacked a compiler or QEMU. The archive regression
fails before the fix, and both new tests pass on all available targets.
Upstream-Commit: 294e22a86a
Retain the original versioned reference when following undefined
symbols in shared li6braries. This prevents an unversioned archive
definition from causing unnecessary extraction while preserving symbol
interposition.
Fixes https://github.com/rui314/mold/issues/1657
Append 70,000 empty sections to GCC-generated LTO assembly for a tiny
program. Preserve the extended section numbering checks for slim and
fat LTO objects and verify that the fat object actually invokes LTO.
Validation: this test passes on all 14 available target configurations.
The observed LoongArch run dropped from 19 seconds to 4.4 seconds; the
14-target run completed in 5.7 seconds.
Replace 1,000 grep invocations with one awk process while checking all
500 DWARF32 and 500 DWARF64 names. This avoids timeouts spent repeatedly
spawning grep during parallel test runs.
Validation: the updated test passes on all 14 available target
configurations. The check accepts a complete dump and rejects missing
names and empty input.
Remove the NOBITS debug-link placeholder and remap section indices so
symbols and section headers still refer to the correct sections. Port
the upstream regression test unchanged.
Validation: separate-debug-file tests across 14 target configurations:
29 passed, 13 skipped, 0 failed. The regression fails before the fix and
passes afterward.
Upstream-Commit: f4edcbd37e
Remove the malformed NOBITS debug link and adjust section indices so
symbols and section headers still refer to the correct sections.
Add a regression test for following the debug link and preserving
symbols in later sections.
Fixes https://github.com/rui314/mold/issues/1656
The relocatable output lays out all leader CIEs before the FDE blocks
of every file, but EhFrameRelocSection::copy_buf emitted the
relocations per file: a file's CIE relocations followed by its FDE
relocations. When a file after the first has a CIE with a relocation,
which is any object with a personality routine, that relocation was
written after the previous files' FDE relocations, so .rela.eh_frame
was not sorted by offset.
Consumers pair CIEs and FDEs with a contiguous run of relocations in a
single scan over .eh_frame, and parse_ehframe is one of them. On the
unsorted section the personality relocation was attached to an earlier
file's FDE with a negative in-record offset, so the final link wrote the
value into the preceding FDE and left the CIE's personality pointer
zero. libgcc treats an encoded zero as "no personality", so functions
under that CIE lost their cleanups and catch handlers: an exception
thrown inside their try block escaped uncaught. Found by dev-util/perf,
whose build partially links a Rust workload object after C objects.
Emit the CIE relocations of every file first and only then the FDE
relocations, which matches the layout, and add a test that links a C
object before a C++ object with `-r`, checks that the relocations are
sorted, and catches an exception through the result.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_0141irCwtPmPMZa9s7pCN3Cz
The relocatable output lays out all leader CIEs before the FDE blocks
of every file, but the relocations for them were emitted per file: a
file's CIE relocations followed by its FDE relocations. When a file
after the first has a CIE with a relocation, which is any object with
a personality routine, that relocation was written after the previous
files' FDE relocations, so .rela.eh_frame was not sorted by offset.
Consumers pair CIEs and FDEs with a contiguous run of relocations in a
single scan over .eh_frame, mold included. On the unsorted section the
personality relocation was attached to an earlier file's FDE with a
negative in-record offset: we panicked on the slice index, and C++ mold
silently wrote the value into the preceding FDE and left the personality
pointer zero, which libgcc treats as "no personality", dropping that
function's cleanups and catch handlers. Found by dev-util/perf in the
Gentoo campaign; a Rust workload object partially linked after C objects
triggered it.
Emit the CIE relocations of every file first and only then the FDE
relocations, which matches the layout. The C++ implementation received
the same change.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_0141irCwtPmPMZa9s7pCN3Cz
Move shell cases and the runner directly under tests/, and remove the
standalone symbol output chunk test files and their references.
Make shell scripts executable and invoke them directly, honoring their
shebangs and execute permissions.
Context already uses E through its input files and output sections, so it
does not need a PhantomData field.
Validation: workspace check across all targets.
Reserve ReaderJob for command-line inputs. Store deferred scripts and LTO
files as tuples with required mapped-file references, and remove unused
archive metadata from the command-line job. Move reader contexts into the
script queue while preserving command-line sorting.
Validation: workspace check; 28 script, archive and LTO tests passed
(two 32-bit prerequisite skips); clang, Chrome and TensorFlow debug
outputs match the baseline.
Replace the manual registration flag with std::sync::Once. Keep counter
updates atomic and skip registration when statistics are disabled.
Validation: workspace check; clang, Chrome and TensorFlow debug links
with matching counter values under --stats.
Remove PhantomData<()> from LoongArchTarget. Its const parameter needs no
marker field, and the target remains a zero-sized type.
Validation: workspace check, including both LoongArch targets.
Replace the separate shuffle mode and seed fields with ShuffleSections.
Keep the last explicit seed while parsing, so a later bare shuffle option
reuses it even after --reverse-sections. Extend the existing seed test to
cover that option sequence.
Validation: workspace check; shuffle and reverse integration tests;
the new option sequence also passed before the implementation change.
Replace SectionOrderKind and its separate fields with a SectionOrder
enum. Keep names or numbers in their variants and retain the original
token only for address diagnostics.
Validation: workspace check and existing section-order integration tests.
Allocate character tables only for bracket expressions. Literal tokens
carry their bytes, and star and question tokens carry no data. Match the
variants directly when parsing patterns and building the NFA.
Validation: workspace check; four glob unit tests; 34 glob and version
script integration tests; clang, Chrome and TensorFlow debug benchmarks.
Replace BuildIdKind and the separate value/hash_size fields with one
BuildId enum. Each parsed option sets a complete variant, and note
emission reads its payload.
Validation: workspace check; existing build-id test; 10 checks covering
absent IDs, fixed byte values and option overrides.
Remove the redundant struct and constructors. Keep the iterator opaque
and retain always-inline dispatch for ordinary and deferred CREL records.
Validation: workspace check; 15 relocation and GDB-index tests; three
large debug links with byte-for-byte identical output.
Replace Arm32InputSectionExtra with u32, using u32::MAX for an absent
.ARM.exidx section. Add InputSectionExtra::empty so ARM32 starts at the
sentinel instead of u32's default zero.
Implement InputSectionExtra directly for Box<[RelocDelta]> and use it on
RISC-V and LoongArch. Remove the wrapper that only forwarded access to
its boxed slice.
Replace the section display wrapper and both file-name display wrappers
with formatting closures. Share the file-name formatter between input
parsing and merged-section diagnostics.
Use Rayon's recursive splitter and split_at_mut to give each task a
separate slice of the preallocated edge array. Preserve edge order and
the per-vertex count checks while removing EdgeBuffer and its unsafe
Sync implementation.
Use owned vectors for parallel thunk symbol collection, removing
ThunkSymbolBins and its unsafe Sync implementation. Atomic symbol marking
still deduplicates entries, and the final sort preserves output order.
Collect files and scripts in per-worker vectors, locking only to append
completed results. Consume the bins after the scoped jobs join and restore
command line order as before.
Keep one cache per Rayon worker and lock it once per input file. Remove the
custom UnsafeCell wrapper and its unsafe Sync implementation while
preserving cache reuse across jobs.
Report a parse error instead of silently defining address zero when a
decimal --defsym value exceeds u64::MAX, as for hexadecimal values.
Test the largest valid decimal and hexadecimal values, zero, and values
beyond the limit. The new regression fails with the original Rust linker;
all seven defsym integration tests pass with this fix.
Keep generated names as bytes and replace each non-ASCII-alphanumeric byte
with an underscore, matching C++ mold for valid and invalid UTF-8 input.
Add a regression that links references to ordinary start/stop symbols.
It passes on all 14 available target configurations and with C++ mold,
and fails with the original Rust linker.
Keep linker-script assignments, --defsym definitions and targets, retained
names, and forced undefined symbols as byte vectors through lookup and GC.
Accept raw command-line symbol names and trim retain-file entries like C++ mold.
Add a regression with distinct invalid UTF-8 and replacement-character names,
raw alias names, archive extraction, and GC roots. It passes with the Rust and
C++ linkers and fails with the original Rust linker. Existing defsym and
retain-symbols tests also pass.
Store the temporary winning file priority in an actual AtomicU32 instead of
casting an immutable u32 field to an atomic reference. Keep ordinary
symbol-index access behind getters and an exclusive setter.
Add a concurrent ownership regression verified under Miri with both Stacked
Borrows and Tree Borrows.
Store files in Vec<Box<T>> and live entries as u32 pool indices. Resolve
access through the pool so indexed access and live iteration do not rely
on cached pointers across Box borrows.
Use disjoint pool slices for mutable and parallel iteration, preserving
stable file indices while letting Box and Vec manage ownership.
Use the abort panic strategy for release builds to reduce generated
code. A panic now terminates the process without unwinding the stack.
An isolated five-run comparison across three workloads reduced .text
size by 12.7% on the local x86-64 machine and 13.6% on Fedora AArch64.
The measured link-time differences were small relative to run-to-run
variation.
Validated release builds for all 20 targets and successful links for
all three workloads on both machines.
Avoid clock reads, getrusage calls and shared record locks during ordinary
links. Explicit --perf keeps the existing timing output. Cover nested timers,
background handles and repeated stops in a unit test.
Match C++ output growth by passing the old length and appended size to
fallocate. Revisiting the already-written prefix flushes dirty extents on btrfs
and adds about a second to Chromium debug links on Fedora. The GDB-index suite
passes across available targets, and the change is neutral on ext4.
Inspect ELF symbol type and definition with readelf instead of requiring nm to
label foo as text. PPC64 ELFv1 function symbols name descriptors in .opd and
correctly appear as data to nm. This lets the existing path and reproduction
tests run across the full target suite.
Initialize reachability when creating object, LTO, and shared inputs so
mandatory objects outrank unextracted archive members during initial resolution.
Process every --undefined-glob match, including multiple matches in reachable
files, so GC retains each matching definition. Port both upstream regressions
verbatim.
Upstream-Commit: af7d056853
Mark dead fragments, copy relocations, and dead input sections as cold paths,
matching the upstream branch hints while retaining ICF leader resolution and
discarded-section diagnostics.
Upstream-Commit: 5bdd5e6284
Collect strong undefined references during the existing import/export traversal
and apply binding changes together with the import flags. Remove the later full
object-symbol scan, preserving weak binding when every undefined reference is
weak.
Upstream-Commit: a4bcbb45ae
Remember up to 32 merged-section keys and IDs in each conversion worker,
avoiding the shared lock on cache hits. Copy immutable keys rather than
retaining pointers into the growable section vector, and discard the cache when
conversion finishes.
Upstream-Commit: 4a83b5d856
Record shared-library names and default-version aliases directly into worker
bins during parsing. Reserve both symbol vectors first so their recorded
slots stay stable until the existing parallel gather fills them. FileList
retains the backing files through SONAME deduplication. This removes the
temporary name arrays and the later registration pass.
Upstream-Commit: 20eb664a7e
Load flag bytes before setting bits, skipping the read-modify-write when all
requested bits are already present. Apply the shared helper to symbol and
input-section setters; keep fetch_or for operations that need the previous
value.
Upstream-Commit: c72d19dcca
Initialize reachability when creating object, LTO, and shared inputs so
mandatory objects outrank unextracted archive members during the first
symbol resolution pass.
Process all --undefined-glob matches regardless of file reachability so
--gc-sections retains every matching definition. Cover archive ordering,
directly linked objects, and multiple matches in one object.
MergedSection::get_instance took a shared lock for every mergeable
input section, though nearly all of them resolve to the same few
output sections. Each worker now remembers up to 32 sections it has
already resolved, keyed on the same name, flags, type and entry size,
and checks them before taking the lock. The cache lives only for the
conversion pass, during which those fields are immutable.
Geometric-mean link time over the 21-program suite fell 0.87% at 32
threads on a Threadripper 7980X and 3.88% at 20 threads on an M1
Ultra, where create_merged_sections on clang-release went from 24 ms
to 10 ms.
Shared libraries inserted their symbol names into the symbol map one
at a time on the parsing thread, leaving a serial tail behind large
DSOs. Their unversioned names now go through the same per-thread bins
as object-file symbols, so gather_symbols creates the canonical
Symbols in parallel across hash shards. The symbol vector is reserved
up front, so the recorded slots stay valid until the gather fills them.
Geometric-mean link time over the 21-program suite fell 1.41% at 32
threads on a Threadripper 7980X, almost all of it from blender-release
(0.177 s to 0.142 s), which links against many shared libraries.
Relocation scanning requests GOT, PLT and TLS entries for the same
symbols repeatedly, and each request took exclusive ownership of the
symbol's cache line even when the bits were already set. Atomic's
operator|= now loads first and skips the read-modify-write when no
bit would change. fetch_or stays unconditional for callers that need
it.
Measured before the check moved from a symbol-specific helper into
the wrapper: geometric-mean link time over the 21-program suite fell
1.95% at 32 threads on a Threadripper 7980X and 2.71% on an M1 Ultra.
Use OsString for arguments and PathBuf for filesystem names so valid Unix paths
survive parsing and reach the filesystem unchanged. Preserve their bytes in
loader metadata, LTO callbacks, reproduction archives and dependency rules as
well.
Add a regression using compiler-generated objects with non-UTF-8 bytes and
spaces in their names. Cover direct arguments, nested response files, library
search, linker scripts, thin archives and reproduction replay.
Store compact chunk IDs in symbol origins instead of pointers with erased
lifetimes and target types. Resolve each ID through an append-only context
registry, preserving the 48-byte symbol layout and removing the raw pointer
dereference.
Add runtime tests for context moves, chunk reordering and 32-bit layouts, plus
compile-fail tests for dangling and incorrectly typed references. The unit
tests, API tests, doctests and 15 related native linker tests pass.
Keep SONAME bytes intact during library deduplication, dependency checks and
dynamic string and version-table output. Distinct non-UTF-8 names must retain
distinct library identities.
Add two versioned libraries whose names previously collapsed under lossy UTF-8
decoding. Verify both names in .dynstr and run the linked program. All 43 native
SONAME, dependency and version-script tests pass.
Make archive member paths absolute without resolving symlinks, matching the
names retained in response.txt. Reopen each input to copy its original contents
as before.
Add a regression that extracts and replays archives for a relative symlink and
an absolute path through a symlinked directory. Both native reproduction tests
pass.
Use the reader context to determine the target of a nested script before
accepting it. This lets an incompatible script beside its parent be skipped in
favor of a compatible script in a later library directory.
Add an x86-64 regression with competing i386 and x86-64 nested scripts. It fails
before the fix and passes in C++ mold; all 15 native linker-script tests pass.
Stop decompression when the requested output buffer is full, allowing the
twelve-byte DWARF classification probe to read a prefix of a larger frame.
Preserve the premature-end diagnostic for short output.
Add a test with verified Zstd-compressed DWARF32 and DWARF64 objects and check
their output order. The new test fails before the fix and passes in C++ mold.
All seven native Zstd and debug-sorting tests pass.
Recognize global:foo and local:* in the shared version-command parser so VERSION
blocks behave like standalone version scripts. Keep colons intact in input
filenames.
Add a regression covering local and global visibility and colon-containing input
names. Discover the new test/ directory alongside the existing shell suite. All
32 native version-script tests pass.
Record how all 27 upstream commits are represented in Rust, including
existing equivalents and C++-only changes. Describe the Rust-specific
optimizations and the completed build, integration, unit-test and static
checks.
Document the 21-workload release benchmark, build settings and noise
rejection criteria. Include raw samples supporting the measured 6.1%
improvement over the original Rust build and the remaining 2.9%
geometric-mean gap to C++.
Inline addr_with and the small auxiliary-index accessors used during
relocation and symbol-table writes. Move rare discarded-section and
.eh_frame diagnostics into a cold, non-inlined helper while retaining
ICF leader handling in the common address path.
Read the auxiliary record once when checking for PLT entries. These
changes reduce call and lookup overhead without changing computed
addresses or diagnostic behavior.
Force AhoCorasick::walk into its caller because it runs once per input
byte while matching symbol names. Keeping the state index and current
priority in the caller's registers removes per-byte call overhead. The
transition logic and highest-priority match rules stay unchanged.
Use mark_reachable for shared-library edges so repeatedly visiting an
already reachable DSO does not keep writing its reachability flag. This
avoids unnecessary cache-line traffic during parallel traversal.
Allocate child-work buffers only when a batch discovers new sections,
and transfer full buffers with mem::take. Exported symbols can already
make most sections roots, leaving many batches with no children; those
batches no longer need a heap allocation.
Feed ICF digest words directly to SipHash when its byte buffer is empty,
avoiding temporary byte arrays and buffering work during propagation.
Keep the byte-update fallback for partial buffers, preserve the native
digest byte order and wrapping length counter, and inline the hot update
and finalization paths.
Compare word updates with byte updates across partial prefixes, scalar
tails and length wraparound to verify that the digest is unchanged.
Replace the eight Option<u32> fields in SymbolAux with four-byte indices
using u32::MAX for absence. This removes separate discriminants and
reduces each auxiliary record from 96 to 64 bytes. Use u32::MAX - 1 for
the provisional dynamic-symbol index so it remains distinct from the
missing-entry sentinel.
Update table insertion and lookup helpers, assert the record size, and
test absent indices, index zero, the provisional value and reuse.
Upstream changes temporary-output-name construction to avoid a GCC
-Wrestrict warning from C++ string concatenation. Rust already builds
these names through formatting and has no matching expression to fix.
This empty commit records that no Rust change is needed.
Upstream-Commit: dad2695f3e
Allocate decoded_crel for all section indices on the first CREL section.
This avoids repeatedly growing and copying the table during section
parsing. Files without CREL allocate no table.
Upstream-Commit: ff0cb501b0
Parsing already omits losing COMDAT members and marks retained sections
alive, so skip redundant state updates for newly parsed files. Keep the
member walk when reusing parsed files, when LTO can change ownership, or
with --gdb-index.
Upstream-Commit: 91252d8aa5
Upstream increases per-thread arena blocks from 64 KiB to 1 MiB to
reduce global reservations, retaining the local-allocation cutoff at one
quarter of a block. Rust uses vectors and mimalloc rather than that
arena, so there is no corresponding block-size setting to change. Record
the C++ change with an empty commit.
Upstream-Commit: 4a11b7849d
Populate name-length caches during symbol registration. Scan global
names up to '@' or NUL, then scan the version suffix only when present,
retaining the raw length before version normalization or --wrap. This
removes a separate cache-population pass for both regular and LTO
objects.
Upstream-Commit: 53715c07ee
Mark the .dynsym, .plt and .plt.got add_symbol functions inline so the
compiler can expand their calls in the serial table-allocation loop.
This exposes the small insertion helpers to optimization while
preserving table insertion order and index assignment.
Upstream-Commit: 41d2f376c3
Reuse the version-suffix flag set during global-symbol registration
instead of scanning COMDAT signature names for '@' again. Handle a bare
trailing '@' separately because it does not set the flag.
Upstream-Commit: 3a58944ff9
Check dynamic-symbol version conflicts in parallel, using existing
symbol-map locking and synchronized diagnostics. Keep the error
checkpoint after all checks finish.
Upstream-Commit: ec7c803207
Create fragment records as section-symbol relocations are reattached,
collecting only the records each file needs in relocation order. Reserve
exactly their final slots in the central symbol vector, then construct
symbols and append their IDs to each file in parallel. This removes the
preliminary relocation-counting pass and unused dummy symbols. Retain
the cheap CREL-header capacity bound to avoid moving existing symbols.
Upstream-Commit: f9bdf4b4de
Rust already runs the synthesized-symbol, object-file and shared-file
symbol-table writers concurrently. Their symbol, string and
extended-section-index ranges are disjoint, so independent writers need
no shared append point. This empty commit records that the upstream
parallelization is already present.
Upstream-Commit: e9de4def18
Keep each dynamic symbol's name and string offset after sizing .dynstr,
reusing cached GNU hash sort keys when available. Both output writers
can then use independent parallel loops without repeating prefix scans
or name lookups. The cache costs 24 bytes per dynamic symbol.
Upstream-Commit: 18d173697c
Rust already caches exact u32 symbol name lengths while keeping Symbol
at 48 bytes. Its name accessors use that cache without rescanning long
names, so there is no corresponding layout or lookup change to apply.
This empty commit records the upstream equivalence.
Upstream-Commit: fa1e8edc39
Rust already uses flat hashbrown tables for symbol-map shards. Compare
cached hashes before strings and fold higher hash bits into bucket
placement so fixed shard-selection bits do not concentrate keys in the
same buckets. Preserve entropy in the high-bit lookup tags and keep
symbol storage stable while reducing lookup overhead.
Upstream-Commit: 841880212f
Resolve non-allocated merged sections and assign their offsets in a
background Rayon task while symbol processing, GC and relocation
scanning continue. Give the task owned metadata and immutable input
bytes, create .comment before starting it, and publish the result before
section sorting and sizing. Keep allocated-fragment layout after GC, and
reassign offsets when debug sections are reordered.
Upstream-Commit: f585fee6ca
Fill .gnu.version slots in parallel after allocating the array. Each
dynamic symbol owns a distinct slot, so these writes are independent.
Upstream-Commit: 95137b235e
Rust already collects newly marked GC roots in local Rayon worker
vectors and combines them before graph traversal. This avoids contention
on a shared concurrent vector, matching the upstream change. No source
change is needed; this empty commit records that equivalence.
Upstream-Commit: 95dbca6708
Give consecutive version-script patterns with the same version index one
priority. Stop matching when no higher-priority result is possible, and
check simple substring patterns before glob patterns.
Upstream-Commit: f8cb3daf00
Compute fragment hashes and cardinality estimates during the string or
record scan, avoiding a second traversal of the input contents and
fragment offsets.
Upstream-Commit: 0f582b0a23
Debug relocations often visit consecutive merged strings. Cache the last
symbol and mergeable-section metadata in each relocation stream, so
input sections do not share mutable lookup state. Try the next fragment
before binary search and prefetch upcoming entries.
Upstream-Commit: 2c402f707f
Process four unsigned bytes per iteration and factor the DJB recurrence
into pairs to shorten its dependency chain. Hash the remaining bytes
with the scalar loop.
Upstream-Commit: 046a644906
Add a stable partition helper that evaluates predicates and scatters
elements into disjoint output ranges in parallel. Use it for the
local/global and import/export partitions in sort_dynsyms, preserving
the relative order within both groups. Evaluate predicates sequentially
for small ranges.
Upstream-Commit: 8566358143
Prefetch symbol records ahead of their auxiliary records to hide the two
dependent pointer loads in the serial table-allocation loop.
Upstream-Commit: 9887339f72
Keep dynamic symbols grouped by their owning input file so auxiliary
records can be initialized and indexed in parallel without globally
sorting symbol IDs. File ownership makes writes independent. Preserve
first-use order within each file, suppress repeated IDs, and retain
existing auxiliary data before the serial table-allocation loop.
Upstream-Commit: 0706198355
Cache GNU hash bucket indices and name views before sorting dynamic
symbols. This avoids repeated pointer chasing, name-length lookups and
bucket calculations in the comparison function.
Upstream-Commit: cbcf79c0ef
Compute string offsets with parallel scans and write dynamic symbols and
strings in parallel. This removes serial work on large dynamic symbol
tables while preserving output order.
Upstream-Commit: 6060657622
Allocate decoded_crel for all section indices on the first CREL section.
This avoids repeatedly growing and copying the table during section
parsing. Files without CREL allocate no table.
Parsing already omits losing COMDAT members and marks retained sections
alive, so skip redundant state updates for newly parsed files. Keep the
member walk when reusing parsed files, when LTO can change ownership, or
with --gdb-index.
Populate name-length caches during symbol registration. Scan global
names up to '@' or NUL, then scan the version suffix only when present,
retaining the raw length before version normalization or --wrap. This
removes a separate cache-population pass for both regular and LTO
objects.
Reuse the version-suffix flag set during global-symbol registration
instead of scanning COMDAT signature names for '@' again. Handle a bare
trailing '@' separately because it does not set the flag.
Check dynamic-symbol version conflicts in parallel, using existing
symbol-map locking and synchronized diagnostics. Keep the error
checkpoint after all checks finish.
Create fragment symbols in the arena as section-symbol relocations are
reattached. Collect pointers locally and reserve the file's symbol
vector once before appending them in relocation order, eliminating the
preliminary counting pass.
Run synthesized-symbol, object-file and shared-file symbol-table writers
concurrently. They write disjoint symbol, string and
extended-section-index ranges; initialize section symbols before
starting these writers.
Keep each dynamic symbol's name and string offset after sizing .dynstr,
reusing cached GNU hash sort keys when available. Both output writers
can then use independent parallel loops without repeating prefix scans
or name lookups. The cache costs 24 bytes per dynamic symbol.
Store a signed 32-bit name length in Symbol's existing padding to avoid
rescanning long names. Saturate at INT32_MAX and scan the suffix only
for saturated lengths.
Replace per-key unordered_map nodes with linear-probing slots containing
cached hashes and pointers to arena entries. This reduces lookup
overhead while keeping symbol addresses stable.
Resolve non-allocated merged sections and assign their offsets in a
low-priority TBB task while symbol processing, GC and relocation
scanning continue. Create .comment before starting the task and join
before section sorting and sizing consume the results. Keep
allocated-fragment layout after GC, and reassign offsets if debug
sections are reordered.
Collect newly marked root sections in per-thread vectors, then flatten
them before graph traversal. This avoids contention on the shared
concurrent vector.
Give consecutive version-script patterns with the same version index one
priority. Stop matching when no higher-priority result is possible, and
check simple substring patterns before glob patterns.
Compute fragment hashes and cardinality estimates during the string or
record scan, avoiding a second traversal of the input contents and
fragment offsets.
Debug relocations often visit consecutive merged strings. Cache the last
symbol and mergeable section per thread, keyed by input section and
symbol index. Try the next fragment before binary search and prefetch
upcoming entries.
Process four unsigned bytes per iteration and factor the DJB recurrence
into pairs to shorten its dependency chain. Hash the remaining bytes
with the scalar loop.
Add a stable partition helper that evaluates predicates and scatters
elements in parallel. Use it for the local/global and import/export
partitions in sort_dynsyms, keeping the standard algorithm for small
ranges.
Initialize SymbolAux in the per-file collection pass, where symbol
ownership makes the writes independent. This reduces work in the serial
allocation loop.
Cache GNU hash bucket indices and name views before sorting dynamic
symbols. This avoids repeated pointer chasing, name-length lookups and
bucket calculations in the comparison function.
Compute string offsets with parallel scans and write dynamic symbols and
strings in parallel. This removes serial work on large dynamic symbol
tables while preserving output order.
Deferred LTO jobs already own archive names, but claiming each member
allocated a second copy. Move the name into the object and use the object's
archive name when setting as_needed.
Validation: all 27 native LTO tests pass. Workspace Clippy passes with
warnings as errors. All 545 upstream test files match C++ mold exactly.
Remove five Rust-only shell tests so the test cases and common support file
match the C++ suite exactly. The Cargo runner still provides test discovery and
execution.
Validation: compare every tracked test file and Git file mode against C++ mold
at 92eb17a07c60a88b3a1a8c9cef014ac84eaf0cf9; no missing, extra or modified test
cases.
Port C++ mold 92eb17a07c. Record IR files from
direct inputs, regular and thin archives, and linker scripts, then claim them
serially after sorting by their command line positions.
Remove the redundant claim mutex and once-only initialization guard. Require
mutable context access when claiming an IR file, and retain the simple
loaded-state check used by the serial plugin lifecycle.
Copy the upstream regression test unchanged. It reproduces an undefined
__cxx_global_var_init before the fix. All 27 native LTO tests pass afterward.
Port C++ mold 450832db6b. Apply the
internal-object binding rule after handling input-local symbols and relocatable
output, preserving global binding for exported synthetic symbols.
Copy the upstream symtab-binding test unchanged. It fails before the fix and
passes afterward. Native symbol-table, synthesized-symbol, start/stop, export
and relocatable-output checks: 24 passed, 1 skipped.
Port the test from C++ mold d3cf4a23e0 unchanged.
The Rust symbol and section arenas were already replaced with vectors, so there
is no large writable arena reservation to change.
Validation: the native rlimit-data test passes with the current allocator.
Port C++ mold 32626eba5d. Ignore empty SFrame
input before reading its header, as required for startup objects assembled by
GNU assembler 2.45.
Copy the upstream regression test unchanged. It failed with a corrupted-section
diagnostic before the fix and passes afterward. Native SFrame tests: 1 passed, 2
skipped because the assembler lacks SFrame v3 support.
The parallel file reader handed IR files to the LTO plugin in
whatever order it found them. LLVM's LTO assumes that the module
holding the prevailing copy of a COMDAT group arrives before the
modules holding the other copies, as it does with a serial linker;
otherwise the losing copies' internal symbols, such as
__cxx_global_var_init.N, become undefined references in the LTO
result. The reader now only records IR files and claims them after
all input files have been found, sorted by command line position.
Claims were already serialized, so no parallelism is lost, and the
serializing mutex and once_flag in lto-unix.cc are no longer needed.
Fixes https://github.com/rui314/mold/issues/1649
Co-Authored-By: Claude Fable 5.1 <[email protected]>
35f4a334 changed is_local() to derive a symbol's binding in .symtab from
its visibility. Linker-synthesized symbols never have their visibility
merged from the internal object's ELF symbols, so they went from
STB_LOCAL to STB_GLOBAL by accident. That adds _DYNAMIC, __bss_start,
_end and the like to the set of global symbols, which confuses ABI
checkers that read .symtab.
Restore the old rule for the internal object's symbols: they are local
unless we export them. The exported ones, such as __global_pointer$ on
RISC-V or __start_*/__stop_* under -z start-stop-visibility=protected,
stay global as before.
Reported by Yao Zi in https://github.com/rui314/mold/pull/1650
Fixes https://github.com/rui314/mold/issues/1651
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Keep Endian, byte order markers, and integer wrappers together in
util::endian. Add fixed-endian read/write helpers and use them from
Endian and architecture code.
Use one macro to provide new, get, and set consistently for all five
standard integer wrappers.
Make create_phdr's define and append helpers accept &ElfShdr directly.
Reuse section headers at their call sites and use short osec bindings
for optional sections.
Validation:
- cargo build --locked --workspace
- cargo test --locked --workspace: 5,487 integration tests passed,
637 skipped, no failures; all 14 library tests passed. Four cross
targets lacked a compiler or QEMU.
Rename the module directory and update its declaration, imports, and
call sites to use crate::chunks.
Validation:
- cargo check --locked --workspace
- cargo test --locked --workspace: 5,487 integration tests passed,
637 skipped, no failures; all library tests passed. Four cross targets
were unavailable because their compiler or QEMU was missing.
Give each section a sibling module containing its type and functions,
and update callers to use the direct paths. Remove the misc and version
grouping modules and redundant nested wrappers.
Validation:
- cargo check --locked --workspace
- cargo test --locked --workspace: 5,488 integration tests passed,
636 skipped, no failures; all library tests passed. Four cross targets
were unavailable because their compiler or QEMU was missing.
ConcurrentMap exposed lookup and traversal while insertion was still
possible. Move those entry points to FrozenMap and keep the construction
API limited to allocation, insertion, prefetch, and freezing.
Freeze the GDB name map before collecting sorted entry references.
Extend the existing map test to verify lookup by the original entry IDs
after freezing, including the associated keys and values.
ConcurrentMap::prefetch only issued a cache hint on x86-64. Move the hint
into a shared utility and use stable inline assembly for x86, ARM32/64,
RISC-V32/64, PowerPC32/64, s390x, and LoongArch64.
Assume SSE on both x86 variants. Encode RISC-V PREFETCH.R as its ORI hint
so it remains a no-op without Zicbop. Keep unsupported architectures as
no-ops and preserve the map's empty-table guard.
Move the Linux and Android MADV_HUGEPAGE calls into shared utilities,
with no-op implementations on other targets. Reuse them for output
mappings, concurrent-map storage, and symbol allocations, keeping the
symbol advice limited to complete pages within the allocation.
Extract platform-specific allocation and deallocation helpers for the
concurrent map, and guard page-alignment rounding against overflow.
The 8 GiB arena was mapped read-write, so on hosts with overcommit disabled
(vm.overcommit_memory=2) Linux ignored MAP_NORESERVE and charged the whole
reservation against the commit limit, and fork() charged it again for the
child. Even a hello-world link then failed with "fork: Cannot allocate
memory". Reserve the range with PROT_NONE instead and make it writable in
64 MiB chunks as the bump pointer advances, as the Windows code already does
with MEM_COMMIT. Only the chunks in use are charged, by mmap and by fork()
alike, and transparent huge pages still back them.
Under RLIMIT_DATA, which counts writable private mappings the same way, a
trivial link now needs 0.5 GiB instead of more than 8 GiB.
Fixes https://github.com/rui314/mold/issues/1644
Co-Authored-By: Claude Fable 5.1 <[email protected]>
InputSectionId stored an already packed integer while FragmentRef kept
its component indices. Store a private ObjId and zero-based u32 index in
InputSectionId, with direct field access in file() and index(). Confine
the packed representation to raw() and from_raw().
Keep the eight-byte size and the existing Origin bit layout. Use the
reserved u32::MAX index for NONE and apply the index-plus-one encoding
at the packing boundary so its payload remains zero. Extend the existing
round-trip test to cover zero indices, maximum supported indices, and
the placeholder value.
Origin was assembling and splitting FragmentRef's two indices itself,
while InputSectionId already supplied raw and from_raw methods. Add the
same encoding interface to FragmentRef and use it from Origin::new and
Origin::get.
Keep FragmentRef's typed fields and preserve the existing packed layout.
Origin still checks the 30-bit merged-section limit before shifting the
payload to make room for its tag.
ObjectFile section access and output-section construction already know
the owning file ID, but building a section ID reloads it from each input
section. Construct IDs from the known owner and the decoded dense index,
and pass the owner once into regular_ids_mut.
Move paired ID/reference lookup to ObjectFile so it resolves the ELF
section index once. Reuse the validated dense-index accessors for shared
and mutable section access, avoiding redundant owner loads and checks.
InputSectionIds are assigned when a section is inserted into a file's
dense vector, and their file indices refer to the stable file pool.
Resolving these IDs currently repeats bounds checks for both indices on
hot section-access paths.
Add internal unchecked accessors for the pool and dense section vector,
and use them in Context::input_section. Document their insertion-based
preconditions and retain debug assertions on both bounds. Keep the
unsafe operations within these accessors and the ID resolution method.
The early fragment estimate scans ordinary relocation tables before
archive extraction, including members that may never be selected. That
adds work to large links such as ClickHouse just to predict capacity.
Restrict the early estimate to cheap CREL header counts from direct
inputs and already reachable archive members. Leave ordinary relocation
counting to the existing preparation of selected files after COMDAT
selection. This retains the early reservation needed by CREL-heavy links
while avoiding an extra scan of ordinary relocation records.
Rewriting mergeable-section relocations can append millions of dummy
symbols after global symbols have been constructed. Reserving only local
symbols allows this later append to grow the vector and copy the already
populated table, which is particularly costly for CREL-heavy links.
Estimate fragment demand before gathering globals, using relocation
counts from CREL headers and inspecting ordinary relocations that refer
to mergeable section symbols. Include that estimate in the initial
reservation and compute it across files in parallel. Reapply huge-page
advice only when vector capacity actually changes.
The ClickHouse benchmark contains C++ symbol names longer than the u16
limit, causing valid input to panic in Symbol::new. Restore u32 storage
for the cached name length and retain the checked conversion into that
representation.
Change the boundary test to construct a 65536-byte name successfully and
verify that Symbol::name returns all of its bytes. This removes the
restriction introduced by narrowing the length field.
Replace obsolete arena terminology with the actual storage and lifetime
rules: symbols and section members live in vectors, input files belong
to a stable pool, and parallel construction reserves disjoint ranges
before exposing writable slots.
Remove stale ArenaPtr distance restrictions and describe background
debug-info work in terms of tasks and concurrency. These are comment
updates; allocation, ownership, and linker behavior are unchanged.
Reserve an upper bound for local symbols when gathering globals so that
appending locals does not immediately move a populated symbol vector.
Request transparent huge pages for whole pages inside that allocation,
leaving partial boundary pages that may contain allocator metadata
alone.
Let per-file section vectors grow with the inputs actually inserted;
reserving for every ELF header wastes space on relocation and metadata
headers that never become InputSections. Validate the packed file-index
limit when inserting files into the pool, and use debug assertions for
the redundant checks when encoding and decoding validated section IDs.
Section lookups were constructing an InputSectionId and decoding it
again even when the caller only needed a reference. Factor the ELF
section table decoding into input_index and use the resulting dense
index directly for shared, mutable, and paired ID/reference access.
Keep regular entries and entries with merge metadata on the same lookup
path. Extend the Origin round-trip test to cover the full supported
32-bit section-index range.
Have InputSectionExtra expose r_deltas_mut, returning a mutable
reference to the owned delta array. InputSection::set_r_deltas can then
perform the assignment itself while retaining its nonempty-array
assertion.
RISC-V and LoongArch return their embedded field. Targets without
relaxation metadata retain the unreachable guard for attempts to mutate
it, keeping the target-specific implementation limited to field access.
Rename MergeableSection to MergeInfo and update the section-table flags
and accessors to describe its role as metadata for an ordinary input
section. Update merging, symbol attachment, and debug-info callers to
use the new names.
Preserve the existing ownership model: eligible SHF_MERGE inputs remain
in their dense slots and are marked dead while their contents are
represented by fragments. Store merge metadata only for those inputs,
without adding a merge-info field to every InputSection.
Logical section IDs and directly owned target state allow input sections
to move when their storage grows. Store InputSection values in each
SectionList's dense vector and resolve IDs by indexing that vector. Keep
merged inputs in their slots so existing references remain valid.
Remove SectionArena, its thread-local allocation blocks, and the manual
section destruction code. Vector ownership now supplies allocation and
destruction, and the shared virtual-memory reservation module can be
deleted because neither symbols nor input sections use it anymore.
The self-relative pointer to InputSectionExtras prevents input sections
from moving and requires a separate arena allocation for target state.
Make InputSection generic over Arch and embed an associated
InputSectionExtra value directly in each section.
ARM32 stores its exception-index section link, and RISC-V and LoongArch
own their relaxation-delta arrays. Other targets use a zero-sized extra
type. Propagate the target type through section owners and callers, and
remove the self-relative pointer encoding and extras allocator.
Arena-relative section IDs tie callers to the location of an allocation.
Represent InputSectionId as an object-file ID and a dense index within
that file, packed into 64 bits. Reserve two bits for Origin's tag and
keep zero available as the placeholder value.
Add a dense per-file slot table that still resolves to the existing
arena during this transition. Update symbol origins, output-section
members, relaxation, and layout lookups to resolve the logical ID
through its owner. This allows section storage to change independently
of the IDs used throughout the linker.
Symbols are already referenced through SymbolId, so their addresses only
need to remain stable while a construction or mutation phase borrows
them. Replace the 8 GiB SymbolArena reservation with Vec<Symbol> and let
the vector own and release the initialized symbols.
Keep the existing parallel construction scheme: reserve capacity before
publishing writable slots, assign disjoint ranges to workers, and set
the initialized length after they finish. Remove the custom virtual
memory allocation, slice access, and destruction code for symbols.
Use u16 for Symbol::name_len, reducing the width of the cached length
field while keeping the existing thin pointer to the name bytes. This
representation imposes a maximum symbol name length of 65535 bytes.
Check the conversion in Symbol::new so that longer names produce an
explicit diagnostic. Add a regression test for a 65536-byte name to
exercise that boundary.
Let the compiler decide whether to inline private and generic helpers that do
not correspond to explicitly inline C++ code. Keep forced inlining on the hot
ELF and relocation paths identified by the C++ implementation and perf
profiles.
Firefox-debug performance remained neutral in both single-threaded and
32-thread runs. The full test suite and release clippy checks pass.
Merge duplicated C++ and Rust comment blocks into single explanations that
describe the Rust implementation while retaining the original rationale.
Remove stale C++ type references and restore the missing rustdoc summary for
sorted concurrent-map entries.
Remove helpers, accessors, and constants that are unreachable from the mold
command and its tests. This includes wrapper methods whose only callers were
other dead wrappers, unused ELF integer operations, and unused concurrent-map
APIs.
Keep the remaining internal len methods crate-visible so Clippy does not treat
them as incomplete public collection APIs.
Keep Origin eight bytes and use its low two bits to distinguish input
sections, output chunks, merged-section fragments, and symbols. Remove the
separate origin-kind bits from Symbol’s visibility byte.
This follows C++ mold’s tagged-pointer representation without the auxiliary
tables and per-variant codecs required by the discarded 32-bit experiment.
The experiment reduced Symbol from 48 to 40 bytes, but it required a
manual codec for every ChunkId variant, a separate namespace for allocated
merged sections, and context-dependent origin decoding. It added 434 net
lines.
It also made the previous Rust linker about 2% slower on Firefox debug and
10% slower on TensorFlow debug, even though it remained faster than C++ mold.
Prefer the simpler 64-bit origin until the memory saving justifies those
costs.
Use unary low-bit tags so fragment origins have a 31-bit payload. Pack
allocated merged-section IDs and fragment bucket IDs into that payload,
and use compact encodings for input sections, chunks, and symbols.
Also compact the remaining symbol metadata so that Symbol shrinks from 48
to 40 bytes.
Split the output-section and symbol-table borrows so
gather_thunk_addresses can record addresses while traversing thunks. This
matches the C++ loop and avoids a temporary allocation for each section.
Replace the callback-based internal parser with an Iterator built using
std::iter::from_fn. Keep the eager Vec results and member creation order
unchanged.
Return the original io::Error from the shared file-opening path instead of
discarding it and reading errno later. Keep optional file lookup semantics by
mapping NotFound to None, and remove errno_string.
Rewrite comments copied verbatim from C++ so they describe the Rust
implementation and terminology. Remove notes that have no Rust counterpart,
and update source-file references to their Rust names.
Keep C++ comparisons when they document an intentional implementation
difference or a remaining performance gap.
GNU assembler 2.45 emits a zero-size .sframe section for an input file
that needs no unwind info, such as glibc's Scrt1.o on Ubuntu 25.10.
parse_sframe() read a header past the end of such a section and reported
it as corrupted.
Fixes https://github.com/rui314/mold/issues/1646
Co-Authored-By: Claude Fable 5 <[email protected]>
Mold performs one link per process, so keep diagnostic settings, error state,
and output serialization in a process-wide singleton. This removes
diagnostic plumbing from contexts and helper APIs while preserving the mutex
that keeps messages from worker threads from interleaving.
Port mold dist.sh to build Rust release binaries in old distro
containers. Download a pinned official Rust toolchain as gzip, fetch
Cargo.lock dependencies before disabling the network, and normalize
package timestamps.
Allow install-mold.sh to consume an external Cargo target directory.
path_clean("../../a/b") returned "a/b": the ParentDir branch popped a
trailing ".." from the output instead of appending another one, so any
path with two or more leading ".." components collapsed into a wrong
path. C++ mold uses std::filesystem::path::lexically_normal, which
keeps leading ".." components intact.
Library search paths are cleaned with this function, so mold failed
with "library not found" whenever a Makefile referenced an in-tree
library as -L../../<dir>, which is a very common pattern. Found by the
Gentoo all-package campaign within its first hour: sys-fs/lvm2,
sys-fs/xfsprogs, sci-astronomy/montage, dev-libs/OpenNI2,
dev-erlang/fast_yaml, and app-backup/spideroak-bin all failed with
mold-rust and built with mold-c++.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0118ERbwNepcG1Z9gh5cm6Ut
Benchmark record, 2026-09-01 (unrelated to this change; this tree is
what was measured, built as 6becb4da before this message was amended):
C++ reference: mold at e322a3cc, i.e. the translation base fbaaa75f
plus 15 fixes that do not touch hot paths.
Protocol: ~/bench/tab4-bench.sh as-is — run 1 warms the page cache,
median of runs 2-6 reported; all runs of one linker complete before
the other linker starts; sync + dirty-writeback wait between runs;
--no-fork; default thread count (min(ncores, 32) = 32 on this
128-core machine); frequency boost off, performance governor,
THP madvise, idle machine.
Workload C++ (s) Rust (s) Rust vs C++
blender-debug 0.94 0.82 -12.8%
blender-release 0.20 0.17 -15.0%
chrome-debug-arm64 1.88 1.97 +4.8%
chrome-debug-x86_64 1.75 1.80 +2.9%
chrome-release-arm64 0.77 0.79 +2.6%
chrome-release-x86_64 0.66 0.69 +4.5%
clang-debug 1.49 1.28 -14.1%
clang-release 0.11 0.10 -9.1%
clickhouse-debug 1.14 1.09 -4.4%
clickhouse-release 0.43 0.43 +0.0%
firefox-debug-arm64 1.07 0.98 -8.4%
firefox-debug 0.87 0.87 +0.0%
firefox-release 0.21 0.20 -4.8%
godot-debug 0.51 0.47 -7.8%
godot-release 0.08 0.08 +0.0%
libmergedlo.so-debug 0.46 0.47 +2.2%
libmergedlo.so-release 0.20 0.18 -10.0%
pytorch-debug 0.93 0.84 -9.7%
pytorch-release 0.15 0.14 -6.7%
tensorflow-debug 3.16 2.89 -8.5%
tensorflow-release 0.69 0.63 -8.7%
Geomean -5.1% (Rust faster); 13 faster, 3 even, 5 slower. No workload
regressed against the import-time table in the initial commit; the
open item remains chrome's +2-5%.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0118ERbwNepcG1Z9gh5cm6Ut
Apply relocation deltas, adjust symbols and recompute executable output
section layouts in parallel, matching the three file and chunk parallel
loops in C++ mold.
Process object files in parallel as C++ mold does. Use the existing
disjoint-range output-buffer abstraction so each worker can reverse its
input sections in place.
Process object files in parallel as C++ mold does. Serialize symbol access
with the existing byte-sized symbol lock and rewrite each file's relocation
records in place.
Use stock zlib's low-level API to reproduce the C++ Z_BLOCK, deflatePending,
optional deflatePrime and Z_SYNC_FLUSH sequence. This avoids the old
libbacktrace boundary bug and preserves C++ mold's compressed bytes.
Install the Unix handlers with SA_SIGINFO and track the mapped output range
as it is created, extended or moved. Report a possible full disk only when
SIGBUS or SIGSEGV identifies an address in that range.
Port the C++ Counter facility and collect the same parser, section, symbol,
ICF, relocation, COMDAT, thunk and merged-section statistics. Keep counter
updates dormant unless --stats is requested.
get_api_version handed the plugin the buffer of a stack-local
std::string, which dies when the function returns. Make it static.
Co-Authored-By: Claude Fable 5 <[email protected]>
The thin archive reader skipped a __.SYMDEF symbol table without
advancing the read position, so it kept parsing the same member header
forever. Skip past the member contents like the other symbol table
branch does.
Co-Authored-By: Claude Fable 5 <[email protected]>
R_RISCV_TLSDESC, R_RISCV_GOT32_PCREL, R_SPARC_SIZE32 and R_SPARC_JMP_IREL
were printed as "unknown (n)" in diagnostics.
Co-Authored-By: Claude Fable 5 <[email protected]>
parse_defsym_value returned a default-constructed variant on a bad hex
string such as 0xz, which holds a null Symbol pointer. The null pointer
was then dereferenced when marking GC roots, crashing the linker instead
of printing a diagnostic.
Co-Authored-By: Claude Fable 5 <[email protected]>
reattach_section_pieces rewrote every symbol whose local definition is in
a mergeable section, including global symbols that lost resolution to a
definition in another file. The loser then redirected the symbol to its
own section piece. Since the pass runs on all files in parallel, it was
also a data race on the shared Symbol.
Co-Authored-By: Claude Fable 5 <[email protected]>
R_390_12, R_390_PC12DBL, R_390_PLT12DBL, R_390_GOT12 and R_390_GOTPLT12
OR'd their 12-bit value into the instruction through a little-endian
16-bit accessor. s390x is big-endian, so the value landed in the wrong
bytes, corrupting both the displacement and the base register field.
Co-Authored-By: Claude Fable 5 <[email protected]>
create_phdrs walks a filtered, address-sorted copy of ctx.chunks, but the
loop that merges adjacent note sections into one PT_NOTE examined
ctx.chunks with an index into that copy. The two vectors differ e.g. when
the ELF and program headers are not memory-mapped, in which case every
note section got its own PT_NOTE segment.
Co-Authored-By: Claude Fable 5 <[email protected]>
When a merged section is laid out in two regions, 32-bit-addressable
fragments followed by the rest, the loop that rebases per-shard offsets to
section-global ones started at shard 1. That is fine for the 32-bit region,
whose shard 0 needs no adjustment, but it left shard 0 of the 64-bit region
unadjusted, so its fragments overlapped the beginning of the section. Start
at shard 0; the 32-bit fragments of that shard just get zero added.
Co-Authored-By: Claude Fable 5 <[email protected]>
The alloc relocation path incremented the local pointer loc instead of the
byte it points to, so the relocated byte was left untouched. The result was
wrong whenever an 8-bit label difference spanned code whose size changed
during relaxation. The non-alloc path already did this correctly.
Co-Authored-By: Claude Fable 5 <[email protected]>
The non-alloc relocation paths of i386, ARM64, s390x, SPARC64 and PPC64V1
passed the relocation value and the relocation index to check_range in the
wrong order. The intended range check was thus never performed, and if the
relocation index happened to fall outside the value range, the error path
indexed the relocation table with the relocation value, reading out of
bounds.
Co-Authored-By: Claude Fable 5 <[email protected]>
Use the same KERN_PROC_PATHNAME sysctl as C++ mold when locating the
current executable on FreeBSD, where /proc may not be mounted. Run one
shell test at a time under sanitizers so large cases do not overlap,
while leaving two CPUs available to the linker under TSan.
Use Cargo per-target features in native and platform jobs so they do not
rebuild all 20 linker instantiations. Tests that require an i386 linker
already skip when that target is disabled. Keep test-all-targets using
defaults for complete QEMU coverage.
If an object file has SHN_LORESERVE (0xff00) or more sections, e_shnum
is zero and the actual number of sections is stored in the first section
header's sh_size field. is_gcc_lto_obj() used the raw e_shnum, so it
scanned no sections and misclassified such LTO objects as regular
object files.
Fixes https://github.com/rui314/mold/issues/1642
Co-Authored-By: Claude Fable 5 <[email protected]>
Give ThreadSanitizer extra time for the many-output-sections test.
Disable incremental artifacts and debug info in the disk-constrained FreeBSD VM.
Also suppress the non-x86 unused prefetch parameter warning.
Add the four platform jobs from C++ mold and make the Rust implementation
compile on each native platform.
Use the corresponding Windows virtual-memory, output-file, subprocess, LTO
and process-timer facilities. Keep the Unix paths for macOS and FreeBSD while
applying platform-specific system calls only where C++ mold does.
Tested with cargo build --release, cargo test and cargo clippy --release
--all-targets. The MinGW executable also links against dlfcn-win32.
The all-target CI job exhausted its runner disk because it retained the
output of every completed shell test. Remove each work directory after a
passing or skipped test while preserving logs and all failed-test files for
diagnosis.
The multi-architecture build generated incremental and debug artifacts on
an ephemeral runner, leaving too little room for the test outputs and
causing the one-hour timeout. Disable both for this job.
In Ubuntu 24.04 Podman, the clean build used 693 MB and the completed test
tree used 7.4 GB. All targets passed with pass=6782 skip=895 fail=0.
ARM BE8 instructions are little-endian, but data embedded in PLT and
thunk entries uses target byte order. Match C++ mold's ul32 instruction
templates and U32<E> literal stores.
CI-equivalent Ubuntu 24.04 Podman:
all targets: pass=6782 skip=895 fail=0
armeb: pass=368 skip=53 fail=0
All workspace crates are built before testing, but the target crates and CLI
binary contain no unit tests. Select the library and integration test targets
so Cargo does not exhaust the runner disk linking empty harnesses.
musl uses an unsigned type for statfs.f_type while libc exposes
TMPFS_MAGIC as a signed value. Convert the constant to the field's type so
the comparison builds with both musl and glibc.
Match C++ mold by attempting fallocate when fstatfs fails.
The job-level environment did not populate the container configuration, so
the Rust toolchain action still lost the system directories when it extended
GITHUB_PATH. Set PATH through container.env instead.
The Tumbleweed image does not declare PATH in its container
configuration. GitHub's implicit default disappears when rust-toolchain
adds Cargo's bin directory through GITHUB_PATH. Define the normal system
path explicitly for the distro matrix so Bash remains available.
Run install-test-deps.sh with /bin/sh so minimal container images can
invoke it. Expand the Debian cross-toolchain package list without Bash
brace expansion, install Bash on Tumbleweed, and remove the Alpine
bootstrap step from CI.
The Alpine base image does not include Bash, so install it before
running install-test-deps.sh. Keep the dependency script Bash-based so it
can use brace expansion.
Run the complete multi-architecture suite and native tests across the
same Linux distribution matrix as C++ mold. Install the extra toolchains
used by the upstream tests and fail CI if any expected target is
unavailable.
Build and test ASan and TSan configurations with nightly Rust. Use the
system allocator for sanitizer builds and instrument the two C adapters as
well. Limit TSan to two CPUs to avoid false reports from Rayon while still
exercising parallel execution.
Make install-test-deps.sh portable across the distribution containers.
Tested with cargo test: pass=5360 skip=636 fail=0. ASan passed 477
native tests with 9 skips, and TSan passed 454 native tests with 32 skips.
Match C++ mold's relaxation behavior by updating input relocation records
at the point where instructions are rewritten. Store decoded CREL records
in typed mutable side tables so emitted relocation sections observe the
same updates, and remove the relocation type reconstruction hooks.
Use I386, X86_64, M68k, and Sparc64 as representative layouts when
classifying GCC LTO objects, matching C++ mold. Use real target layouts for
the compile-time record assertions and remove Elf32Le, Elf64Le, Elf32Be, and
Elf64Be.
Also remove the obsolete Record trait. Direct file records use
FileRecord, and Record had no implementations or callers.
Checked with cargo check --workspace. The test suite was not rerun as
requested.
Rename ProgramHeaderRecord and CompressionHeaderRecord to the shorter
PhdrRecord and ChdrRecord names. Expand the four plain layout definitions
directly and remove the plain_layout macro.
Infer the ELF class from the layout word size and whether relocations
carry addends from RelRecord. This removes the duplicated IS_64 and IS_RELA
declarations from every target.
Tests: cargo test (pass=5360 skip=636 fail=0)
Lint: cargo clippy --release --all-targets
Keep input section headers and symbols in their endian-aware on-file
representation instead of decoding them into target-independent mirrors.
Carry that representation through output chunks and expose fields through
record traits, so reads and writes remain target-endian.
Remove SymbolEntry, SectionHeader, SymTable, ShdrTable, and RecordLayout
now that no converted record mirrors remain. Move the SFrame constants to
elf/consts.rs.
On the 21-workload, 32-thread benchmark, the geometric-mean time changed
by -1.1% versus 801fc. Excluding tensorflow-debug's unusually large
improvement, it was effectively flat at -0.1%.
Tests: cargo test (pass=5360 skip=636 fail=0)
Lint: cargo clippy --release --all-targets
Represent uniform ELF records with public target-endian fields and select
class-dependent symbol, program-header, and compression-header layouts
through Layout. Convert differing layouts as complete records instead of
adding field-by-field forwarding accessors.
Remove the raw st_info API and the old offset-based record readers.
Tests: cargo test
Tests: cargo clippy --release --all-targets
Define byte-backed integer and relocation types that mirror the C++ ElfRel
specializations, including the SPARC64 type-data field.
Select each relocation record through Layout and view input and output
tables as ordinary typed slices. This removes the host-native conversion
wrappers and the synthetic r_info representation.
Preserve target byte order independently of the host.
Tests: cargo test
Tests: cargo clippy --release --all-targets
Define relocation constants and diagnostic-name functions for each
architecture from a single macro invocation. Keep each macro input to the
relocation name and value, and use stringify! to produce the diagnostic
spelling.
Remove the separate relnames module so relocation names and values can no
longer drift apart.
Validation passes with cargo build --release and cargo clippy --release
--all-targets. Cargo test reports 5360 passed, 636 skipped, and no failures.
Replace the callback-only relocation traversal API with streaming Rust
iterators. Make CrelReader and RelocationIter implement Iterator while
preserving deferred CREL decoding without allocation.
Use ordinary for loops in the architecture backends and gdb-index reader.
Keep a specialized fold implementation so Iterator::for_each can select the
ordinary or CREL representation once.
The full test suite passes with 5360 cases passed, 636 skipped, and no
failures. Clippy is clean.
Five-run firefox-debug medians improved from 0.798 s to 0.765 s at 32
threads and from 11.135 s to 11.002 s with one thread. The C++ reference
at fbaaa75f measured 0.792 s and 11.999 s, respectively.
Configure Rayon's process-global pool after target detection and process
forking. Register the main thread as worker zero so --threads=N limits the
linker to N active threads.
Run the linker directly instead of wrapping its entire body in
ThreadPool::install. This follows mold's command-oriented execution model
and its process-wide TBB parallelism limit.
Rename Rust modules to match their C++ source counterparts where
practical. Use archive_file, cmdline, error, output_chunks,
shrink_sections, and perf while keeping mapped_file, output_file, lto,
and subprocess unchanged.
Remove filename-only comments added during the port. They were not copied
documentation and became redundant once the filenames were aligned.
Update all module paths and target entry points for the renamed modules.
Carry over mold's distribution-specific dependency installer as
install-test-deps.sh. Drop CMake packages that the Rust build no longer uses.
Keep test setup out of the user build instructions. Explain the stable Rust
and native C compiler requirements directly in README.
Copy the current README, manual, logos, and license from mold. Adapt the
build and installation instructions for the Rust implementation.
Add install-mold.sh to install the binary, preload library, manual, license,
and linker symlinks under a configurable PREFIX with DESTDIR support.
Move the two remaining C source files from csrc/ to c/ and update the
build script and LTO documentation to use the new paths. The files are
otherwise byte-identical.
Tests: cargo build --release
Tests: cargo test (pass=5360 skip=636 fail=0)
Tests: cargo clippy --release --all-targets
Tests: cargo fmt --all -- --check
Replace the local allocator wrapper and vendored mimalloc sources with the
rui314/mimalloc_rust repository pinned at:
cc58b72775b8b5bc89ee37752af61fcb4f7c2909
The dependency contains mimalloc 3.5.0 and its overlapping upstream files
are byte-identical to the deleted copy. Update the workspace, allocator
import, lockfile, and README.
Tests: cargo build --release
Tests: cargo test (pass=5360 skip=636 fail=0)
Tests: cargo clippy --release --all-targets
Tests: cargo fmt --all -- --check
Register a custom integration-test target for mold-cli and move the Rust
runner into the mold-tests workspace crate. Keep all 537 shell-test files
byte-for-byte unchanged under tests/cases, run every available target and
QEMU job by default, and stage Cargo-built test artifacts in the active
profile directory.
Remove run-tests.sh and make every workspace crate a default member so
plain cargo test is the complete test entry point.
Use explicit wrapping arithmetic for unsigned C++ operations exposed by
the debug-profile test suite.
Tested with:
cargo test
cargo build --release
cargo clippy --release --all-targets
cargo fmt --all -- --check
Full shell-suite result: 5,996 scheduled, pass=5360, skip=636, fail=0.
Treat every C++ implementation comment at mold commit
fbaaa75f9f as a first-class porting
asset. Restore all 2,179 non-namespace comment occurrences at their
Rust counterparts, including file-level explanations, ABI notes,
examples, TODOs, original typos, inline instruction annotations, and
repeated comments. Add provenance where one Rust implementation
consolidates multiple C++ implementations. Closing namespace labels
are the sole comments without a Rust construct.
Verify that all 537 upstream test files remain byte-for-byte identical
to the pinned C++ tree.
The audit also exposed a few implementation-fidelity gaps: reopen
input mappings before writing a repro archive, reset discarded-COMDAT
symbol indices after owner selection, mirror C++ GOT
dynamic-relocation accounting branches, and reserve version-section
output storage as C++ does. These are faithful translations, not new
algorithms.
Validated with cargo build --release, ./run-tests.sh --all (pass=5360
skip=636 fail=0), cargo clippy --release --all-targets, cargo fmt
--check, and git diff --check.
Mirror C++ ConcurrentMap sorted-entry handling: reserve each shard vector, use unstable sorting, retain direct stable entry references, and test occupancy through the atomic key pointer alone.
Store direct stable NameEntry references in indexed-name records so the table writer does not translate a bucket ID for every public-name occurrence.
Reference C++ mold: fbaaa75f9f
Validation:
- cargo build --release
- ./run-tests.sh --all: pass=5360 skip=636 fail=0
- cargo clippy --release --all-targets
32-thread benchmark, median of five, with sync before and sync plus sleep 1 after every linker invocation:
- chrome-debug-arm64: C++ 1.736 s, Rust 1.834 s (+5.65%)
- chrome-debug-arm64 --no-gdb-index: C++ 1.576 s, Rust 1.592 s (+1.02%)
- chrome-release-arm64: C++ 0.715 s, Rust 0.738 s (+3.22%)
The remaining Chrome debug difference is isolated to TBB low-priority arena behavior; Rayon has no corresponding task-priority feature.
mold demangles with the platform's __cxa_demangle, and FreeBSD's libcxxrt
spells `operator delete[]` as `operator delete []`, so the quoted name in
the test never matched there. Test literal matching of brackets with
`S::operator[](int)` instead, which every demangler spells the same way.
Co-Authored-By: Claude Fable 5 <[email protected]>
So far -z rewrite-endbr was implemented only for x86-64, where it
rewrites unnecessary ENDBR64 landing pads (Intel CET) with NOPs for
functions whose addresses are not taken. arm64 has an equivalent
control-flow-integrity feature, BTI, whose landing pad is `bti c`.
This commit generalizes rewrite_endbr into a template shared by both
targets: the landing pad and NOP byte patterns are selected per-arch
(ENDBR64/NOP on x86-64, `bti c`/`nop` on arm64). The whole-program
analysis that preserves landing pads for address-taken functions is
unchanged.
Unlike x86-64, arm64 needs range extension thunks, and a thunk reaches
its target with an indirect branch (`br x16`). A function that is only
called directly can therefore still be entered indirectly once its call
is routed through a thunk, so we must keep its landing pad. We preserve
the landing pad of every thunked symbol; the thunk symbol set has
already been narrowed to the ones that actually need a thunk by
remove_redundant_thunks().
Note that on arm64 with recent glibc, `main`'s landing pad is
legitimately removed: the CRT passes a local __wrap_main stub (whose
landing pad is kept) to __libc_start_main and reaches main through a
direct branch, which does not require a landing pad.
Also note that code built with -mbranch-protection=standard (BTI +
pointer authentication) starts functions with `paciasp`, which doubles
as a landing pad, so there is no separate `bti c` to rewrite and this
option has no effect on such code.
This was locally tested on an Apple M2 machine running Asahi Linux.
Fixes#1552.
The relocation has a 16-bit displacement from the TOC pointer, but the
ELFv2 backend applied it without a range check and silently truncated
displacements to .toc entries beyond 32 KiB, producing a load from the
wrong address. The ELFv1 backend already checked it; do the same here.
Co-Authored-By: Claude Fable 5 <[email protected]>
FDEs refer to their functions with 32-bit PC-relative relocations, and
.eh_frame_hdr stores 32-bit offsets, but mold applied them without any
range check. A function more than 2 GiB away from .eh_frame, which
happens with a huge text section or with -Ttext or --section-start
placing code far from the default layout, silently got a wrapped-around
FDE that the unwinder cannot use, while GNU ld and lld reject such
links. Those relocations are now range-checked like all others, and so
are the .eh_frame_hdr entries.
The aarch64 long thunk test placed code with unwind tables 4 GiB away
from .eh_frame and relied on the bogus FDEs going unnoticed. It now
compiles the far functions without unwind tables and keeps the far
section fixed while swapping the functions between the sections, so
calls in both directions are still covered.
Co-Authored-By: Claude Fable 5 <[email protected]>
GNU ld takes a quoted name in a version script or a dynamic list as a
literal string, while an unquoted one is a glob pattern. mold dropped
the quotes and treated everything as a pattern, so `"operator
delete[](void*)"` in an `extern "C++"` block never matched anything
because `[]` parsed as an empty character class. Glob metacharacters in
a quoted name are now escaped before the name is handed to the matcher.
Co-Authored-By: Claude Fable 5 <[email protected]>
The tokenizer keeps colons within a token for the C++ scope operator,
so `local:*;` lexed as a single token that the parser then took for a
global symbol pattern named `local:*`, and nothing was localized. The
label is now split off such a token.
Co-Authored-By: Claude Fable 5 <[email protected]>
mold took the instruction set of a branch target from the LSB of the
symbol value alone, but only STT_FUNC symbols encode the instruction
set that way. A Thumb BL to a plain label in hand-written assembly,
which is NOTYPE and has an even address, was rewritten to BLX and
crashed with SIGILL, and a Thumb B.W to it went through a mode-switching
thunk. Like GNU ld, we now switch the instruction set only for function
symbols and for PLT entries, which are always ARM code.
Co-Authored-By: Claude Fable 5 <[email protected]>
With -Os, GCC calls linker-synthesized routines not only to save and
restore general-purpose registers but also floating-point and vector
registers. mold only provided the former, so such links failed with
undefined symbol errors. The new routines use the same layout and
encodings as GNU ld; the section that holds them is renamed to
.save_restore_regs accordingly.
Co-Authored-By: Claude Fable 5 <[email protected]>
With -r, common symbols are not converted to .bss, but to_output_esym()
had no case for them and wrote them out as absolute symbols whose value
was their alignment, so linking the -r output again resolved them to a
bogus fixed address. They are now emitted as SHN_COMMON symbols with the
alignment in st_value, as in the input files.
Co-Authored-By: Claude Fable 5 <[email protected]>
The addend of R_RISCV_ALIGN is the number of NOP bytes the assembler
emitted, which is the requested alignment minus the minimum instruction
size, so the alignment is the smallest power of two greater than the
addend rather than the addend rounded up to a power of two. `.balign 4`
in RVC code (addend 2) and `.balign 8` in non-RVC code (addend 4) were
therefore treated as 2- and 4-byte alignments; in a static link that
left glibc's __sigsetjmp at a 2 mod 4 address.
Co-Authored-By: Claude Fable 5 <[email protected]>
An unresolved weak function resolves to address 0. In a huge executable,
a call site can be more than 2 GiB away from it, so a call relocation
such as R_X86_64_PLT32 cannot encode the displacement and mold reported
an out-of-range error. Such a call is never executed because the program
has to null-check the function first, so we now skip the range check for
call relocations against such symbols and write the truncated value.
This covers x86-64, RISC-V, s390x, SPARC64 and LoongArch; ARM already
rewrites such calls to NOPs, and PowerPC calls go through thunks. Data
references keep their range checks because their results feed the null
check.
This fixes linking Godot's sanitizer build, whose .text ends 70 MB past
the 2 GiB mark.
Fixes https://github.com/rui314/mold/issues/1640
Co-Authored-By: Claude Fable 5 <[email protected]>
Without .sdata, we defined RISC-V's __global_pointer$ as absolute
address 0. Since crt1.o initializes gp with a PC-relative
`lla gp, __global_pointer$`, such an executable could not be linked
more than 2 GiB away from address 0, for example with -Ttext-segment.
We now define the symbol at the beginning of the image, like GNU ld
which defines it relative to .data or .bss in that case. The exact
value doesn't matter because we don't relax gp-relative accesses.
Co-Authored-By: Claude Fable 5 <[email protected]>
reattach_section_pieces() traversed every relocation of every alloc
section once just to count the references to mergeable sections, so
that it could allocate their symbols at once, and then traversed them
again to create the symbols. The count needs only ELF data, so it is
now taken while the sections are parsed, where a CREL relocation table
has just been decoded and is still in cache.
Co-Authored-By: Claude Fable 5 <[email protected]>
The tokenizer tested each byte against six whitespace characters and
three quoting characters in turn. A table makes the scan a single
load and compare per byte, and halves the time to tokenize Chromium's
1.9 MiB response file.
Co-Authored-By: Claude Fable 5 <[email protected]>
The loop looked up every undefined symbol in the --wrap set even when
no --wrap option was given, hashing 5.1 million long C++ names on a
Chromium link, and fetched the thread's ShardedMap bin through TBB's
thread-local lookup once per symbol rather than once per file.
Co-Authored-By: Claude Fable 5 <[email protected]>
create_output_sections() copied each output section's members from its
per-file vectors into the final array on one thread and then freed the
vectors one by one, which took about 10 ms on a Chromium link. Copy
and free per file in parallel instead, computing the alignment in the
same pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
The pass took each symbol's lock once per undefined reference, 5.1
million times on a Chromium link, although nearly every reference is
to a defined symbol, which the pass leaves alone. Popular symbols are
referenced from thousands of files, so their locks bounced between
cores. Find the references to symbols without a definition first,
which needs no lock because the pass does not change defined symbols,
and take the locks only for those.
Chromium release on a Threadripper 7980X at 32 threads: the pass goes
from 21 ms to 2 ms. Output files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
When a symbol with hidden visibility resolved to a DSO, symbol
resolution started over from scratch for all symbols with that DSO
definition excluded, and Chromium has three such symbols. Resolution is
independent per symbol, so only those symbols are resolved again, and
if one of them moves to an archive member that is not reachable yet,
liveness is propagated from that member alone. The final resolution
pass after section parsing recomputes everything else as before.
Chromium release on a Threadripper 7980X at 32 threads: resolve_symbols
goes from 143 ms to 115 ms. Output files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
Sorting the 44,683 files of a Chromium link into command line order
took 7 ms on one thread at the end of the input reading phase.
Co-Authored-By: Claude Fable 5 <[email protected]>
scan_abs_relocations() gathered each member's absolute relocations into
a vector of its own and then flattened the vectors on one thread. For
.data.rel.ro of a Chromium link that was 147,000 vectors holding 1.1
million entries, and the flattening and freeing took about 10 ms of
serial time. Count the entries per member first and write them to
their final positions in parallel instead.
Co-Authored-By: Claude Fable 5 <[email protected]>
The job that read a thin archive opened all of its members before
feeding them to the reader loop, so a large archive's members were
opened one after another on one thread. Chromium's libblink_core.a
has 2,424. The archive job now only lists the member paths, and each
member's job opens its file, in parallel with the others.
Co-Authored-By: Claude Fable 5 <[email protected]>
Each MergeableSection added itself to its MergedSection's member list
under a mutex from the parallel conversion loop. An output has only a
few merged sections, so on a Chromium link, 32 threads took the same
mutex 150,000 times and spent most of the pass in futex calls. Collect
the members from the files after the loop instead.
Chromium release on a Threadripper 7980X at 32 threads:
create_merged_sections goes from 55 ms to 34 ms and its system time
from 0.74 s to 0.07 s. Output files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
mmap(2) takes the process's address space lock, so with tens of
thousands of input files, the calls serialize at a few microseconds
each no matter how many threads make them. Linking Chromium maps 44,683
files, and that serialization alone bounds the input reading phase at
about 220 ms, more than a third of the link. Files up to 32 KiB, which
are 64% of Chromium's inputs by count but 16% by size, are now read
into malloc'ed buffers instead; read(2) takes no such lock, and larger
files are still mmap'ed so that their pages are shared with the page
cache rather than copied.
Chromium release on a Threadripper 7980X at 32 threads: the input
reading phase goes from 228 ms to 108 ms and the whole link from 0.69 s
to 0.56 s, with peak RSS unchanged. Output files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
The tokenizer pushed every byte through an ostringstream and copied
each token twice. Tokens without quotes or backslashes, which is nearly
all of them, are now substrings of the mapped file, and whitespace is
tested inline instead of through isspace().
Expanding Chromium's 1.9 MiB response file of 19,000 arguments took
33 ms of single-threaded startup on a Threadripper 7980X; it now takes
5 ms. Output files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
scan_abs_relocations() classified an output section's absolute
relocations and copy_buf() applied them in serial loops, although
write_dynrels() already processes them in parallel shards. A PIE
Chromium has 1.1 million of them in .data.rel.ro alone, so each loop
ran for about 30 ms on one thread while the others waited.
Chromium release on a Threadripper 7980X at 32 threads: scan_relocations
goes from 84 ms to 50 ms, copy from 89 ms to 75 ms, and the whole link
from 0.90 s to 0.86 s. Output files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
Most relocations refer to a section that another thread has already
visited, yet visit() always did an atomic fetch_or, so popular sections
bounced between cores on every reference. Check the bit with a plain
load first, as Atomic::test_and_set() does; it is never cleared during
marking, so a stale positive answer is still correct.
Chromium release on a Threadripper 7980X at 32 threads: the mark phase
goes from 65 ms to 25 ms, the whole link from 0.96 s to 0.92 s. Output
files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
GDB 10 aborts on GCC's DWARF 5 type units even when the executable
contains no .gdb_index section, which broke CI on Debian 11. Probe GDB
with an index-less executable first and skip the test if it cannot
print a type from it.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CKdZWtcaghcU7m5zoZtMfN
Emit version 9 indexes when type units are present so GDB 12 falls
back instead of interpreting DWARF 5 type-unit offsets as
.debug_types. Keep version 7 for indexes without type units.
Allow the regression test to run with older readelf versions and skip
it under TSAN, where relocatable links hit known oneTBB races.
With -gdwarf-5 -fdebug-types-section, GCC moves each type into its own
type unit and emits it as a one-section COMDAT .debug_info contribution,
so an object file contains several .debug_info sections. We assumed one
.debug_info section per file containing nothing but compilation units:
type units were missing from the index, and a pubnames offset could be
interpreted relative to the wrong contribution.
Read every live .debug_info contribution instead, classifying units by
their header's unit_type. Type units are collected into the .gdb_index
type-unit table along with their signatures, and each pubnames set is
matched to its contribution through the relocation on the set header's
debug_info_offset field. COMDAT group selection has already discarded
duplicate type units by the time we read them, so exactly one copy of
each type unit is indexed.
Fixes https://github.com/rui314/mold/issues/1634
Co-Authored-By: Claude Fable 5 <[email protected]>
If an IR file claims a COMDAT group, the LTO result provides the group's
contents. However, post-LTO COMDAT group selection started from a clean
slate, so an archive member extracted only after LTO could win a claimed
group, resurrecting a copy that collides with the LTO result.
Remember the claim on the signature symbol, and let only LTO-generated
files win such a group after LTO. LLVM keeps claimed groups in its
output; GCC emits their contents without a group, so no file wins.
Fixes https://github.com/rui314/mold/issues/1637
Co-Authored-By: Claude Fable 5 <[email protected]>
shrink_section() fills r_deltas once, and the later passes only read it,
so it does not need to be a growable vector afterwards. Build the deltas
in a local vector and copy them into the arena. This shrinks InputSection
by eight bytes on RISC-V and LoongArch and makes the class trivially
destructible on every target, so InputSection objects can live in the
arena without an owner that runs their destructors.
Co-Authored-By: Claude Fable 5 <[email protected]>
Several InputSection booleans were stored separately even though one
atomic byte is sufficient. Pack liveness, GC visitation, address
significance, decompression, and ICF-removal state into flags, and add
relaxed fetch_or/fetch_and operations for updating them.
Also remove fde_end by marking the final FDE in each section range in
existing FdeRecord padding. This reduces InputSection<X86_64> from 56 to
48 bytes without increasing FdeRecord or Symbol.
The output offset, dense ICF index, and arena-relative ICF leader are
needed at disjoint points in the link. Keeping separate members wastes
space and requires a separate eligibility flag.
Store them in a union, using icf_idx == -1 to represent ineligibility.
Process the gathered ICF sections directly and retain icf_leader only for
eliminated sections after ICF, allowing leaders to restore their offsets.
Counting distinct ICF classes costs about as much as a digest
propagation round, but it was done after every round. Since refinement
only splits classes, convergence does not need to be checked that often.
Run three propagation rounds between class counts and move digest-buffer
swapping into propagate(). This reduces counting overhead while preserving
the final partition.
When an ODR violation leaves a relocation referring to a discarded COMDAT
section, ObjectFile::scan_relocations() reports an error, but
scan_relocations() continues with the absolute-relocation pass. That pass
assumes symbols are resolved and may dereference a null Symbol::file,
causing mold to crash instead of exiting with the diagnostic.
Add a checkpoint between the two relocation passes so mold exits before
inconsistent symbol state reaches the absolute-relocation scan.
Fixes https://github.com/rui314/mold/issues/1635
create_comdat_group_sections assumed that a relocation section in a
comdat group is always SHT_RELA or SHT_REL. If an input file uses the
CREL format, the relocation sections are SHT_CREL instead, so we took
the branch for ordinary member sections and dereferenced a null
pointer, as mold does not create an InputSection for a relocation
section. mold -r had been crashing on such files since CREL input
support was added in mold 2.38.0.
Recognize SHT_CREL the same way as SHT_RELA/SHT_REL so that the
member's relocation section is propagated to the output group.
Co-Authored-By: Claude Fable 5 <[email protected]>
Commit 35f4a334 changed Symbol::is_local() to check visibility and
version-script localization, but get_output_sym_idx() also relies on it
to decide whether a symbol's slot offset is relative to the local or the
global partition of .symtab. An input-local symbol has default
visibility and no version, so is_local() returned false for it, and a
relocation emitted by --emit-relocs against a named local symbol
referred to an arbitrary global symbol.
This went unnoticed on most targets because GAS usually redirects
relocations against local symbols to section symbols, which take a
different code path. On LoongArch and RISC-V, the assembler keeps them
as-is because linker relaxation may change section-relative offsets, so
relocations against symbols like .LC0 were visibly wrong.
Fixes https://github.com/rui314/mold/issues/1631
Compilers may emit a meaningless FDE covering a zero-length address
range, e.g. for a function fragment that hot/cold splitting has
entirely optimized away. Such an FDE begins at the same address as the
FDE of the next real function, giving the binary search table in
.eh_frame_hdr duplicate keys. If a search lands on the empty FDE
instead of the real one, the unwinder fails to find an existing
exception handler and aborts the process.
We now write a tombstone value to .eh_frame_hdr for such an FDE. The
tombstone compares greater than any real address, so a binary search
never returns it. The FDE itself is harmless because an empty address
range doesn't match any program counter.
AArch64 range extension thunks are small and unlikely to benefit
measurably from aligning every entry to a 16-byte boundary. Padding
every entry with brk instructions wastes meaningful space when an
output contains many thunks.
Keep each thunk group 8-byte aligned, but pack its entries tightly so
the ARM64 short and long forms occupy 12 and 24 bytes instead of 16
and 32 bytes.
A weak TLS variable is usually guarded by a separate weak non-TLS
symbol indicating whether the TLS definition is available. A typical
use looks like this:
extern char foo_used __attribute__((weak));
extern __thread int foo __attribute__((weak));
if (&foo_used)
foo = 1;
If foo remains undefined, its access is unreachable and there is no
meaningful value to apply to its TLS relocations. Leave those
relocations unapplied.
This prevents AArch64 from rejecting an otherwise unreachable TLS
access when an R_AARCH64_TLSLE_ADD_TPREL_HI12 relocation is out of range.
https://github.com/rui314/mold/issues/1626
SH4 raises a slot-illegal exception when a PC-relative load is placed
in a branch delay slot. Our PLT entries did exactly that:
jmp @r0
mov.l 2f, r1
QEMU now checks for such illegal delay-slot loads as of
https://github.com/qemu/qemu/commit/b754cb2dcde26a7bc8a9d17bb6900a0ac0dd38e2,
matching real hardware and the SH-4 ISA. Older QEMU versions accepted
this sequence because they did not enforce the restriction, so upgrading
QEMU caused dynamically linked mold output to trap when entering a PLT
slot.
Move the relocation-index load before the jump and put a nop in the
delay slot. Grow each PLT entry from 16 to 20 bytes to make room while
keeping the two literal values 4-byte aligned.
`exit` in an awk script that follows readelf in a pipeline closes the
pipe as soon as the first match is found. If readelf has not finished
writing its output by then, it is killed by SIGPIPE, which pipefail
turns into a test failure. The race is timing-dependent and was
observed roughly once per thousand runs on a loaded machine.
Co-Authored-By: Claude Fable 5 <[email protected]>
This rewrites lib/gentoo-test.sh to make a full-distro run practical
and its results easier to trust:
- Build each package with mold first, and run the GNU ld control build
only when the mold build fails. Results are classified into success/,
failure/ (builds with GNU ld but not with mold, i.e. a mold bug) and
broken/ (doesn't build with either linker, so not our fault).
- Run test suites only for the package under test by scoping
FEATURES=test through package.env. Applying it globally drags in the
test dependencies of the entire dependency graph and makes a large
part of the tree unresolvable.
- Let emerge apply the configuration changes it would otherwise just
suggest (--autounmask-continue). An ephemeral container has no
hand-maintained package.use, so without this, thousands of packages
are unbuildable out of the box.
- Bring the container image to a consistent ~amd64 state with a world
update at image build time. A stable stage3 base with testing
packages on top otherwise accumulates unsolvable dependency
conflicts, such as perl disagreeing with its installed virtuals.
- Share work between containers through host bind mounts: source
tarballs are downloaded once into a shared DISTDIR, and binary
package pools let each dependency be built once instead of once per
container that needs it. The mold and GNU ld runs use separate pools
so that a control build can never install mold-linked binaries.
- Serialize the container image build with flock. Parallel invocations
used to race to build the same image, one per job slot.
- Work around a deadlock in portage's binary package extraction with
lib/gentoo-gpkg-workaround.py: tar_stream_reader.close() waits for
the external decompressor to exit without draining its stdout, which
hangs when the undrained residue exceeds the pipe buffer.
- Refuse to run if dist/mold doesn't embed a git commit hash, since
test results are keyed by it.
Co-Authored-By: Claude Fable 5 <[email protected]>
MergeableSection::resolve_contents inserts every section fragment into
the parent's ConcurrentMap. The table is typically much larger than
the cache, so each insertion stalls on a cache miss for its first
probe. Since split_contents has already computed all fragment hashes,
prefetch the bucket eight insertions ahead.
On firefox-debug at 32 threads, this cuts the CPU time of
compute_section_sizes, which performs the insertions, by about 15%
(2.7 to 2.3 CPU-seconds) and the pass's wall time from 0.11 s to
0.10 s; the end-to-end change is within noise. All 5809 tests pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
get_sorted_entries splits the bucket array into NUM_SHARDS pieces, and
MergedSection::compute_section_size assigns fragment offsets with one
task per shard, so the shard count caps that pass's parallelism at 16
threads even when more are available. Raise it to 64.
Probing is confined to a shard, and a probe that visits MAX_RETRY
occupied slots aborts, so MIN_NBUCKETS grows with the shard count to
keep at least MAX_RETRY buckets per shard in the smallest maps.
Since fragment offsets are assigned shard by shard, the layout of
merged sections changes slightly (inter-shard alignment padding).
Outputs remain bit-reproducible for a given linker version, which is
the guarantee mold makes.
On firefox-debug at 32 threads, compute_section_sizes goes from
0.13 s to 0.11 s; the end-to-end change is within noise. All 5809
tests pass.
Co-Authored-By: Claude Fable 5 <[email protected]>
RelDynSection::update_shdr runs several times per link because the
size of .relr.dyn feeds back into section layout: set_osec_offsets
calls it until the size converges, and compute_section_headers and
the final pre-copy refresh add more calls. Each call walked all
chunks serially and ended with a serial sort of every RELR offset.
On a Firefox debug link (~360k GOT slots, ~1M relative relocations),
this accounted for most of a ~0.2 s single-threaded stretch between
layout and output copy that is plainly visible in a CPU utilization
trace.
Three changes:
- GotSection::get_num_dynrels no longer materializes the full GOT
entry array just to count relocations. Computing each entry's
value involves a symbol address lookup, which dominated the cost.
The counting logic mirrors get_got_entries' relocation choices.
This is most of the win: the GOT was the longest pole on every
call even with the collection parallelized.
- Collect per-chunk dynamic relocation counts and RELR offsets in
parallel; chunks are independent of one another.
- Process chunks in address order when concatenating their RELR
offsets. Each chunk emits its offsets in ascending order and
chunks occupy disjoint address ranges, so the concatenation is
sorted by construction and the global sort is gone.
The output is byte-identical on firefox-debug against the unpatched
linker at the same commit. All 5809 tests pass. An interleaved A/B
on firefox-debug shows the 32-thread link going from 1.18 s to
about 1.05 s (medians of five).
Co-Authored-By: Claude Fable 5 <[email protected]>
RELR needs only the addresses to which relative relocations apply. Even after
the parent commit, mold creates complete REL/RELA records, partitions them in
RelDynSection, and then discards every field except r_offset.
Have each output chunk return a compact vector of its RELR offsets. Aggregate
and sort those vectors in RelDynSection, then encode them directly. Each chunk
also records how many of its dynamic relocations moved to RELR, so it can write
the remaining records straight to its final output slice.
Android+RELR keeps a temporary array for the remaining records because the
size of their APS2 encoding affects layout.
Seven interleaved 32-thread runs with --no-fork gave these median wall times:
Parent This commit
Firefox release 0.472518 s 0.396796 s (-16.03%)
Chromium debug 2.081396 s 2.044275 s ( -1.78%)
Compared with 64b92151, the accumulated improvements are 17.79% for Firefox
release and 1.68% for Chromium debug.
An earlier seven-run A/B test found no consistent cost from returning each
vector instead of appending through an output parameter:
Firefox release: append 0.396 s, return 0.390 s
Chromium debug: append 2.005 s, return 2.028 s
The differences are less than 2% and point in opposite directions.
All 471 native x86-64 tests pass, with nine expected skips. A release build for
all supported targets also passes.
RelDynSection currently asks each producer for a vector and appends it to a
second vector. OutputSection also builds one vector per shard and flattens
them. Large links therefore allocate, copy, and destroy relocation records
multiple times.
Have each producer report its exact relocation count on Chunk. RelDynSection
allocates one destination array and gives each producer a pointer to its exact
slice. OutputSection records per-shard prefix counts so shards can fill
disjoint slices in parallel.
Recompute counts on every layout pass because synthetic GOT relocation counts
can change before layout converges.
Seven interleaved 32-thread runs against 64b92151, with --no-fork, gave these
median wall times:
Firefox release: 0.482679 s -> 0.472518 s (-2.11%)
Chromium debug: 2.079169 s -> 2.081396 s (+0.11%)
All 471 native x86-64 tests pass, with nine expected skips. Release builds for
x86-64 and PPC64V1 also pass.
After fix_synthetic_symbols, we re-ran set_osec_offsets
unconditionally. That pass is needed only under
--pack-dyn-relocs=android, whose packed relocation format encodes
addends in variable-length SLEB128, so fixing synthetic-symbol
values can change the encoded size of .rela.dyn. No other mode
emits anything whose size depends on addends; in particular,
RELR's size depends only on relocation locations, which are fixed
by this point. Make the pass conditional on the Android mode.
On the Firefox release benchmark with 32 threads and --no-fork,
this shortens the link from 0.44 s to 0.41 s. Outputs are
byte-for-byte identical before and after for both a RELR workload
(Firefox) and a RELA workload (Chromium release).
Co-Authored-By: Claude Fable 5 <[email protected]>
The scalar glob matcher is faster for ordinary version scripts, but its
work grows linearly with the number of patterns. Compile sets of 64 or more
complex patterns into one bit-parallel NFA. Each state occupies one bit, so
a machine word advances 64 states at once. Keep the existing scalar and
Aho-Corasick paths unchanged for smaller and simpler pattern sets.
On 200,000 synthetic 60-byte symbol names, the standalone matcher changes
as follows:
128 patterns: 1.00 s -> 0.197 s (5.1x faster)
256 patterns: 1.98 s -> 0.380 s (5.2x faster)
A normal TensorFlow debug link remains on the scalar path. A sanity check
shows apply_version_script changing from 0.177 s to 0.175 s.
The general glob matcher represents all patterns as one NFA and updates
several bit vectors for every byte of every symbol. Its cost therefore
depends on the total pattern length even when a version script contains only
a few complex patterns.
Compile each complex pattern into literal, star, question, and bracket
tokens. Match without recursion, remember the most recent star, and use
string_view::find for literal runs following a star. This replaces MultiGlob
and removes the Bitvector class.
Also store non-root Aho-Corasick edges as sibling lists. Most trie nodes have
one child, so this reduces a node from 1,040 to 24 bytes on a 64-bit host. A
dense root table keeps the common transition fast. Construct failure links
breadth-first, as required by the algorithm.
With symbol-table traversal held constant, the median of five 32-thread
TensorFlow debug links changes as follows:
apply_version_script wall time: 0.503 s -> 0.179 s (-64%)
apply_version_script CPU time: 14.137 s -> 5.467 s (-61%)
total wall time: 3.626 s -> 3.285 s (-9%)
Version script wildcard matching currently visits each object file and scans
its global symbol slots. A symbol is processed only when that file provided
the prevailing definition, so many entries are skipped. More importantly, an
entire file is one task, which leaves workers idle when large object files
reach the end of the pass.
Iterate over the symbol table instead. Its 64 shards visit every symbol once
and distribute the work more evenly. Rename ShardedMap::for_each to
parallel_for_each to make the scheduling behavior explicit.
With the same glob matcher and 32 threads, the median of five TensorFlow
debug links changes as follows:
apply_version_script wall time: 0.699 s -> 0.503 s (-28%)
apply_version_script CPU time: 13.456 s -> 14.137 s (+5%)
total wall time: 3.800 s -> 3.626 s (-5%)
A defined dynamic symbol at VER_NDX_GLOBAL with the VERSYM_HIDDEN bit set
is a legacy compat alias. GNU as emits one for `.symver foo, foo@VER`
without the `remove` modifier, so that binaries linked before the symbol
was versioned keep resolving. It must not satisfy a new unversioned
reference to `foo`; that has to bind to the default version `foo@@VER`.
SharedFile::parse cleared VERSYM_HIDDEN before classifying the symbol, so
such an alias took the "unversioned symbol" path and claimed the plain
name. With both it and the `foo@@VER` definition claiming that name,
resolve_symbols keeps whichever comes first in .dynsym, which is the
alias. No verneed entry is then emitted, and glibc refuses to hand out a
non-base version for an unversioned lookup, so the alias wins at runtime
too and the caller silently gets the old implementation.
lksctp-tools' libsctp.so.1 exports sctp_connectx this way. Linking with
mold selects the pre-SCTP_SOCKOPT_CONNECTX3 ABI, which reports success
but never fills in the caller's association ID.
Skip such symbols while parsing. They cannot be referred to by any name,
as there is no version string to form `foo@VER` from. ld.bfd reports an
undefined reference when one is the only definition, which is what
skipping produces.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Each provisional thunk owns independent symbol and offset vectors. Filter
unneeded entries and recompute their offsets concurrently instead of
shrinking the thunks one at a time.
On the Chromium Arm64 debug link, seven interleaved A/B runs reduce the
remove_redundant_thunks median from 0.080 to 0.073 seconds and median
wall time from 1.924 to 1.897 seconds. The output is byte-identical.
Symbol order does not affect thunk placement, so defer deterministic sorting
until all thunks for an output section have been constructed. Sort independent
thunk vectors concurrently instead of sorting each one in the serial batch
loop.
On the Chromium Arm64 debug link, nine interleaved A/B runs reduce median wall
time from 1.98 to 1.97 seconds. The compute_section_sizes median falls from
0.116 to 0.095 seconds. The output is byte-identical.
Initialize input-section offsets in parallel. Section offsets and end offsets
are monotonic, so use lower_bound to find A and C instead of walking over
millions of sections a second and third time.
On the Chromium Arm64 debug link, five interleaved A/B runs reduce median wall
time from 2.01 to 1.95 seconds. create_range_extension_thunks falls from about
0.17 to 0.11 seconds. The output is byte-identical.
Collect newly discovered thunk targets in per-worker vectors and concatenate
them after each batch. The existing atomic flag still removes duplicates, and
the final sort preserves deterministic output without serializing every vector
insertion through one mutex.
On the Chromium Arm64 debug link, five interleaved A/B runs reduce median wall
time from 2.19 to 2.03 seconds. The compute_section_sizes phase falls from
0.341 to 0.181 seconds, and its system CPU time falls from 5.146 to 0.620
seconds. The output is byte-identical.
Allocate MergedSection objects in the linker arena and store each fragment's
parent section as a 32-bit arena index. This shrinks SectionFragment from 24
to 16 bytes and its ConcurrentMap bucket from 64 to 32 bytes.
Chromium's x86-64 debug link contains about 2.17 million merge-map buckets.
Across five interleaved 32-thread runs, median wall time changes from 1.91 to
1.90 seconds and median peak RSS falls from 24,870,408 to 24,808,228 KiB, a
reduction of 62,180 KiB.
Build the symbol table and constant pool in a low-priority task after
debug-info offsets are fixed. The final output path then reads relocated
address ranges and copies the prepared tables.
The stages have different dependencies and useful parallelism. Reading
input name data needs only input sections and can use all available
workers. Building the table needs final .debug_info offsets but stops
scaling near 12 workers, so run it in a separate bounded arena. Reuse one
task group for both stages.
Keep the standard serial open-addressed hash construction. This preserves
deterministic output without the prototype atomic claim table and repeated
parallel probe rounds.
Against an unrestricted table arena on the same tree, the bounded arena
reduced the 10-run median from 1.935 s to 1.905 s and user CPU from
43.510 s to 42.865 s.
Chromium debug link, 32 threads, 10 interleaved run medians:
wall: 1.975 s -> 1.900 s (-3.8%)
user: 43.125 s -> 42.600 s (-1.2%)
system: 9.740 s -> 9.770 s (+0.3%)
peak RSS: 24,817,662 -> 24,889,408 KiB (+70 MiB)
Use the ELF section header for the stored input size and keep only the
data pointer in InputSection. Decompressed sections use sh_size because
their replacement buffer has the uncompressed size.
This reduces InputSection<X86_64> from 88 to 80 bytes. Relaxed RISC-V
and LoongArch sections continue to use the original section-header size
when copying their input bytes.
Chromium debug link, 32 threads, 10 interleaved run medians:
wall: 1.960 s -> 1.950 s (-0.5%)
user: 42.565 s -> 42.560 s
system: 9.775 s -> 9.615 s (-1.6%)
peak RSS: 24,834,922 -> 24,816,480 KiB (-18 MiB)
Parse each reachable object and record its COMDAT candidates in one
parallel traversal. Reuse the signature name while parsing instead of
recovering it from the selected Symbol for the GCC debug-macro check.
Chromium debug link, 32 threads, 10 interleaved run medians:
wall: 1.965 s -> 1.960 s (-0.3%)
user: 43.160 s -> 42.615 s (-1.3%)
system: 9.820 s -> 9.845 s (+0.3%)
Reserve a sparse address-space arena for symbols and input files. Objects in
that arena can refer to each other with four-byte self-relative pointers,
which halves the size of symbol-reference arrays and reduces Symbol from 48
to 40 bytes. On 32-bit hosts, reserve 256 MiB instead of 8 GiB; this places a
practical limit on linker metadata without affecting 32-bit targets linked on
a 64-bit host.
Keep the standard per-shard hash tables, but allocate their Symbol values in
stable arena blocks. Store global names beside those values, derive local
names from their owner files, and represent each COMDAT group in eight bytes.
Context still destroys file objects normally so their separately allocated
data is released before the arena is unmapped.
On the Chromium x86-64 debug benchmark with 32 threads, median wall time over
eight runs falls from 1.990 to 1.970 seconds. Median user time changes from
42.880 to 43.015 seconds, while system time falls from 9.965 to 9.890 seconds.
Median peak RSS falls from 25,625,100 to 24,841,684 KiB, a reduction of
783,416 KiB.
The generated output was byte-identical.
Chromium has 41.96 million ELF section-table slots but only 5.86 million
regular input sections. Store a 32-bit value per slot instead of an
InputSection pointer. Zero denotes an absent section, the high bit denotes a
mergeable section, and the remaining bits index stable per-file pools.
InputSectionTable owns both pools and presents the same indexed and range
interfaces used by linker passes. Moving a section to the mergeable pool is
now explicit instead of relying on a constructor that clears a pointer passed
by reference.
Chromium debug link, 32 threads, eight interleaved run medians:
wall: 2.005 s -> 2.000 s (-0.2%)
user: 43.595 s -> 43.375 s (-0.5%)
system: 10.160 s -> 9.865 s (-2.9%)
peak RSS: 26,145,360 -> 25,598,862 KiB (-534 MiB)
All section contents matched apart from .comment and the fast build-ID note.
GDB index type vectors contain fixed-width 32-bit records. Sort lists with at
least 256 records using four byte-wise radix passes, while retaining comparison
sorting for smaller lists. Reuse worker-local scratch storage so the linear
path does not allocate for every name.
Chromium debug link, 32 threads, eight interleaved run medians:
wall: 1.965 s -> 1.955 s (-0.5%)
user: 44.720 s -> 43.890 s (-1.9%)
system: 9.940 s -> 10.185 s (+2.5%)
The complete 4.86 GiB output was byte-identical.
Chromium debug has 18.39 million local symbols, and 9.71 million belong to
discarded COMDAT sections. After COMDAT elimination, allocate Symbol objects
only for retained local symbols and map every discarded local symbol to one
global zero-valued placeholder. The ELF gABI requires local symbols in a
discarded COMDAT member to be discarded and forbids references to them from
outside the group, so the placeholder does not preserve per-symbol metadata.
Chromium debug link, 32 threads, six interleaved run medians:
wall: 2.095 s -> 2.060 s (-1.7%)
user: 48.095 s -> 46.735 s (-2.8%)
system: 10.110 s -> 10.280 s (+1.7%)
peak RSS: 26,946,944 -> 26,241,790 KiB (-0.67 GiB)
Chromium sections matched apart from version and fast build-ID metadata.
Chromium debug contains 93.8 million relocations in nonallocated sections.
Its 230 MiB CREL input expands to about 2.1 GiB of ElfRel records. Decode
and apply those relocations in one pass instead of retaining the expanded
arrays. Keep exact arrays for allocated sections, whose relocations are
consumed later.
for_each_reloc calls a target-specific callback for every relocation. GCC
15 does not inline these large callbacks on its own, leaving a function call
for each ordinary REL or RELA relocation. Mark the callbacks ALWAYS_INLINE;
this has no effect on CREL's representation but avoids that regression.
Chromium debug link, 32 threads, six-run medians:
wall: 2.215 s -> 2.115 s (-4.5%)
user: 50.145 s -> 48.415 s (-3.5%)
system: 11.065 s -> 10.215 s (-7.7%)
peak RSS: 27.96 GiB -> 25.64 GiB (-2.32 GiB)
On an Asahi machine, adding ALWAYS_INLINE to the streaming implementation
changes the native clang-debug-arm64 link at 16 threads as follows:
wall: 2.36 s -> 1.85 s (-21.6%)
user: 35.3 s -> 27.1 s (-23.2%)
Chromium output matched apart from the fast build-ID payload. The two Asahi
outputs were byte-identical.
Compilation-unit headers and GNU pubnames do not depend on relocated
debug contents. Read them from per-file input sections and build the name
map in a low-priority TBB arena while foreground passes continue. Address
ranges and final serialization still wait until debug relocations are applied.
Chromium debug link, 32 threads, six-run medians:
wall: 2.255 s -> 2.175 s (-3.5%)
user: 49.670 s -> 49.485 s (-0.4%)
system: 11.460 s -> 11.260 s (-1.7%)
peak RSS: 27.95 GiB -> 27.98 GiB (+36 MiB)
The generated .gdb_index was byte-identical.
Most regular COMDAT signatures already name a global symbol. Reuse that
Symbol as the group identity instead of interning 14.8 million signatures in
a second map. Local, section, and versioned signatures use the same symbol
table through a name-based fallback.
Symbol resolution has been cleared when COMDAT owners are selected, so
sym_idx is temporarily available. Record the lowest file priority there, mark
each group whose file has that priority as the owner, and restore sym_idx with
one symbol-map traversal before normal resolution resumes. This keeps
ownership on the signature itself and needs no dense group IDs or auxiliary
side arrays.
On the Chromium x86-64 debug benchmark with 32 threads, median wall time over
eight runs falls from 2.310 to 2.250 seconds. Median user time falls from
49.520 to 47.390 seconds, while system time changes from 11.720 to 11.935
seconds. Median peak RSS falls from 29,485,754 to 29,103,198 KiB, a reduction
of 382,556 KiB.
The generated output was byte-identical.
ELF symbols already contain their string-table offsets. Cache only a one-byte
NameLen instead of retaining a 16-byte string_view for each input symbol.
Longer lengths use logarithmic lower bounds, so lookups scan only the uncached
suffix. Chromium has 37.8 million input symbols, making the per-symbol
reduction substantial.
On the Chromium x86-64 debug benchmark with 32 threads, median wall time over
eight runs changes from 2.320 to 2.315 seconds. Median user and system time
fall from 49.800 and 11.625 seconds to 49.645 and 11.580 seconds. Median peak
RSS falls from 30,132,552 to 29,503,344 KiB, a reduction of 629,208 KiB.
The generated output was byte-identical.
Store short name lengths exactly in one byte and use the remaining values as
logarithmic lower bounds. Looking up a long name then scans only its suffix.
Copy a long version-prefix name into the string pool when its original storage
is not NUL-terminated. This reduces Symbol<X86_64> from 56 to 48 bytes.
On the Chromium x86-64 debug benchmark with 32 threads, median wall time over
eight runs falls from 2.355 to 2.340 seconds. Median user time falls from
50.475 to 49.875 seconds, while system time is unchanged at about 11.85
seconds. Median peak RSS falls from 30,330,582 to 30,146,730 KiB, a reduction
of 183,852 KiB.
The generated output was byte-identical.
An input section's object and section header already identify its name in
the section-header string table. Store only the 16-bit length in InputSection
and recover the pointer from that existing metadata. This reduces
InputSection<X86_64> from 104 to 88 bytes.
On the Chromium x86-64 debug benchmark with 32 threads, median wall time over
eight runs changes from 2.365 to 2.360 seconds. Median user and system time
fall from 50.940 and 12.005 seconds to 50.495 and 11.700 seconds. Median peak
RSS falls from 30,470,228 to 30,335,754 KiB, a reduction of 134,474 KiB.
The generated output was byte-identical.
A CREL header gives the relocation count before decoding starts. Add
ExactArray, which owns an uninitialized array of a fixed size and provides
indexed access, and fill it directly. Existing relocation consumers receive a
span over the array.
This avoids vector capacity slack and initialization that the decoder
immediately overwrites.
On the Chromium x86-64 debug benchmark with 32 threads, median wall time over
eight runs falls from 2.400 to 2.370 seconds. Median user and system time fall
from 51.655 and 12.115 seconds to 50.855 and 11.965 seconds. Median peak RSS
falls from 31,006,116 to 30,470,208 KiB, a reduction of 535,908 KiB.
The generated output was byte-identical.
Chromium contributes 35.8 million GDB name records. Keep each temporary
record at 16 bytes, replace its parsed form with its indexed form after
interning, and deduplicate names within each compilation unit with a small
linear hash table instead of sorting them.
Compute each string hash and length in one scan, assign constant-pool ranges
with a parallel prefix sum, and record type-vector slots while interning. Keep
the slot-claiming protocol shared by both ConcurrentMap insertion interfaces.
On the Chromium x86-64 debug benchmark with 32 threads, median wall time
over eight runs falls from 2.550 to 2.390 seconds. Median user and system time
fall from 54.735 and 12.390 seconds to 51.275 and 12.095 seconds. Median peak
RSS falls from 32,086,034 to 30,999,878 KiB, a reduction of 1,086,156 KiB.
The generated output was byte-identical.
The cardinality estimators used shared atomic registers even though their
callers already partition work across threads. Give each worker a local sketch
and merge the registers after the parallel scan. Apply the same scheme to GDB
index names and mergeable-section fragments.
Chromium debug link, 32 threads, five interleaved run medians:
wall: 2.60 s -> 2.57 s (-1.2%)
user: 56.22 s -> 55.95 s (-0.5%)
The generated .gdb_index was byte-identical.
Section construction repeatedly recovered names from .shstrtab even when the
caller had just looked up the same name. Pass known names into constructors and
defer other lookups until a name is actually needed.
Chromium debug link, 32 threads, five interleaved run medians:
wall: 2.61 s -> 2.60 s (-0.4%)
user: 57.38 s -> 56.22 s (-2.0%)
The generated output was byte-identical.
Duplicate COMDAT groups account for millions of sections in large debug
links. Read the symbol table and group membership first, choose the winning
groups after archive extraction, and only then construct input sections and
local symbols for retained members.
Let ShardedMap report whether gather created each symbol so new symbols can
be initialized in the same pass. An empty section table means that a file has
not reached section construction, avoiding a separate parsing-state flag.
Chromium debug link, 32 threads, five interleaved run medians:
wall: 2.77 s -> 2.61 s (-5.8%)
peak RSS: 33.2 GiB -> 30.9 GiB (-2.3 GiB)
The generated output was byte-identical.
Canonical PLTs and copy relocations are alternative ways to give an
imported symbol an address in the output. Which one is needed follows
from the symbol type, so recording separate scan flags duplicates
information.
Use one CANONICAL action and NEEDS_CANONICAL flag, then choose the table
to populate when dynamic symbol entries are assigned.
We choose between the two PPC64 ABIs by endianness because ELFv1 is
normally big-endian and ELFv2 little-endian. However, that
correspondence is a convention rather than a rule. In particular, musl
uses ELFv2 on big-endian too, so linking for example Alpine ppc64
objects silently produced an ELFv1-style executable with a synthesized
.opd whose startup code crashes immediately.
We support only the usual combinations, so make get_elf_type() treat the
others as unrecognizable, just like an ELF file for an unsupported
machine. Also give unrecognizable ELF files a proper error message; we
used to print "incompatible file type: ppc64 is expected but got " with
a trailing empty string.
Fixes https://github.com/rui314/mold/issues/1498
Co-Authored-By: Claude Fable 5 <[email protected]>
We computed the binding of a symbol in the static symbol table from its
dynamic import/export status, so in an ordinary executable, every global
symbol was emitted as STB_LOCAL. That misleads all symbol table
consumers; nm prints "t main" instead of "T main", and gdb had to work
around it for a feature that depends on symbol bindings.
Only symbols hidden by visibility or localized by a version script
should be demoted. GNU ld keeps hidden symbols STB_GLOBAL with hidden
visibility while lld converts them to STB_LOCAL; we follow lld, which
also matches our previous output for such symbols.
One wrinkle: claim_unresolved_symbols() assigned version index 0
(VER_NDX_LOCAL) to imported symbols so that their .gnu.version entries
would be 0, which would make the new is_local() misclassify them as
version-script-localized. Leave ver_idx unassigned for imported symbols
and have the .gnu.version writer emit 0 for unversioned undefined
symbols directly, so that VER_NDX_LOCAL on a symbol means only
"localized by a version script".
Fixes https://github.com/rui314/mold/issues/906
Co-Authored-By: Claude Fable 5 <[email protected]>
A default-versioned symbol `foo@@VER` can be referred to not only as
`foo` but also as `foo@VER`, so defining both `foo@@VER` and `foo@VER`
is a duplicate definition of the versioned name. We didn't diagnose it
and silently emitted a dynamic symbol table containing two definitions
of `foo@VER`, leaving it up to the dynamic loader which one a versioned
reference binds to. GNU ld and lld reject this; now we do too.
We check for duplicates by scanning the dynamic symbol table once it's
populated, because the problem manifests if and only if both symbols
are exported, and the exported set is much smaller than the set of all
symbols. GNU ld additionally reports the error for non-exported
symbols; lld doesn't, and we behave like lld.
Fixes https://github.com/rui314/mold/issues/828
Co-Authored-By: Claude Fable 5 <[email protected]>
We ignored defined dynamic symbols whose .gnu.version entry is 0
(VER_NDX_LOCAL) on the assumption that such symbols are bound locally.
That assumption is wrong; a symbol that's actually local to a DSO
doesn't appear in .dynsym at all, so such an entry is ill-formed. GNU
ld briefly emitted version index 0 for unversioned defined symbols
(https://sourceware.org/bugzilla/show_bug.cgi?id=33577), and linking
against a DSO created by such ld made us report confusing "undefined
symbol" errors for symbols that plainly exist in the DSO.
lld rejects such files as corrupted. Follow suit and report an error
pointing at the offending file and symbol instead of silently ignoring
it.
Fixes https://github.com/rui314/mold/issues/1534
Co-Authored-By: Claude Fable 5 <[email protected]>
We silently accepted a shared object file given on the command line even
in the static mode and simply ignored it, producing an executable whose
references to the shared object's symbols remain unresolved. Such a
binary crashes at runtime. GNU ld and lld reject this at link time, and
so should we.
Fixes https://github.com/rui314/mold/issues/1616
Co-Authored-By: Claude Fable 5 <[email protected]>
We used to read files under .git directly to avoid a build-time dependency
on git, but the on-disk repository format is not stable. The hash lookup
broke for repositories using the new reftable format, and it never worked
in git worktrees, where .git is a file rather than a directory.
Just ask git instead. Building from a git checkout now requires git, which
is not much to ask; building from a source tarball still works and simply
omits the hash, as before.
Fixes https://github.com/rui314/mold/issues/1621
Co-Authored-By: Claude Fable 5 <[email protected]>
After LTO, we reset the reachability of archive members and redo symbol
resolution to re-derive which members are actually needed, because the
LTO-generated code may no longer use some of the members that were
extracted to satisfy references from IR files. However, the results of
the pre-LTO COMDAT group selection survived into the second resolution:
ComdatGroup::owner kept the first winner's priority, and the losers'
member sections remained dead.
Therefore, if a group's winner was an archive member that is not
re-extracted after LTO, no remaining file could provide the group's
sections, and we reported a false "refers to a discarded COMDAT section"
error for a perfectly valid program.
Reset the COMDAT group state in do_lto() so that the post-LTO symbol
resolution selects groups from a clean slate among the files that
actually remain in the link.
Fixes https://github.com/rui314/mold/issues/1565
Co-Authored-By: Claude Fable 5 <[email protected]>
Like the previous commit did for comdat groups, symbol interning no
longer goes through a single concurrent hash table. Object file
parsing only records each global symbol name along with which slot of
the file's symbol vector it belongs to; gather_symbols() then
deduplicates the names into Symbol objects and fills in the files'
symbol pointers, processing each hash shard on a single thread with
no synchronization. Besides avoiding concurrent hash table
insertions, this hashes each name only once and no longer constructs
a temporary Symbol per lookup, which the concurrent map's insert
interface required. Linker-synthesized symbols, shared libraries and
linker scripts keep interning through a locked slow path in
get_symbol().
Linking a Chromium debug build on a Threadripper 7980X saves ~12
CPU-seconds (83 -> 71) and about 0.25 seconds of wall time at 32
threads. Output files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
We used to intern comdat group signatures into a single concurrent
hash table as object files are parsed. The concurrent map is costly
under 32-way insertion, and every file redundantly hashed the
signature strings, which are long mangled names for C++ programs.
Signatures are now handled by a new ShardedMap class: parsing only
appends them to thread-local buffers binned by hash, and once all
files have been read, each bin is deduplicated by a single thread
with no synchronization, filling in the files' ComdatGroupRefs. A
signature is hashed once, when it's recorded. Deduplication runs in
resolve_symbols() right before comdat elimination rather than at the
end of read_input_files() so that files parsed later, such as ELF
files produced by the LTO plugin, are covered when we redo symbol
resolution. LTO comdats otherwise take a mutex-protected slow path.
Linking a Chromium debug build on a Threadripper 7980X is about 4%
faster (3.22s -> 3.06s at 32 threads), saving ~4 CPU-seconds. Output
files are bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
We used to read input files one at a time in the command line order,
with a single loop opening and mmap'ing each file and walking archive
files. That loop, not parsing, is the bottleneck of the read phase;
for a Chromium debug build with ~45,000 input files, it takes ~1.3
seconds, about 40% of the total link time.
We now open and read files with a parallel for-each loop with the
feeder pattern; an archive file feeds one job per member. The command
line parser records the reader state (--as-needed etc.) and a
hierarchical position for each input file argument, and files are
sorted back into the command line order afterwards to assign
priorities. Linker scripts, which can mutate the linker context, are
parsed after the parallel loop, one at a time in the command line
order. LTO claims are serialized because the plugin interface has
shared state.
Linking Chromium debug on a Threadripper 7980X is ~2% faster at 32
threads and ~14% faster at 64 threads, where the serial loop
previously made additional threads a net loss. Output files are
bit-identical to before.
Co-Authored-By: Claude Fable 5 <[email protected]>
mremap, which we used to grow the output file mapping for
.gdb_index, is Linux-specific, so commit 9b7c5267 broke the build on
the BSDs and macOS. Instead of growing a mapping that was created
with the exact file size, we now map the output file with twice as
much address space as its size upfront. The mapping beyond the end
of the file may not be touched, but growing the file with ftruncate
makes the tail accessible, so extend() no longer needs any mmap
call, and the output buffer never moves. Reserving address space
does not consume any physical memory; pages materialize only when
they are first touched.
This also fixes the madvise(MADV_HUGEPAGE) call in extend(), which
had always failed with EINVAL because its address was not
page-aligned. That went unnoticed because an mremap'ed mapping
inherits the huge-page flag from the original mapping.
Co-Authored-By: Claude Fable 5 <[email protected]>
A large link creates tens of millions of InputSection objects, one
heap allocation each, and the profiler attributed 5% of all cycles
to their construction. Allocate them from a per-file std::deque
instead and refer to them through plain pointers. Section death is
represented by a null entry as before; the object itself stays in
the pool until the file is destroyed.
This removes the InputSection constructor from the profile and
cuts the allocation count by 4x, but the end-to-end link time is
unchanged within noise on a Chromium debug build; the constructor
cost was evidently not on the critical path. The change still pays
its way by making `sections` a plain pointer vector, which drops
the unique_ptr indirection and the .get() noise throughout.
Co-Authored-By: Claude Fable 5 <[email protected]>
Compilers emit numerous temporary local symbols such as .L.str.42
for unnamed program elements. They are compiler-internal: nothing
references them through the symbol table, and DWARF refers to their
data by section offset. Nevertheless we kept them in the output by
default, following GNU ld. A Chromium debug build contains 2.26
million of them, and they added ~180 MiB of .symtab and .strtab to
the 4.9 GiB output.
Discard them by default, as lld does. The new --discard-none option
restores the previous behavior. -r and --emit-relocs continue to
keep all local symbols, since emitted relocations may refer to
them, and RISC-V and LoongArch no longer need their special case,
as the default now covers them.
Link time is unchanged within noise; the benefit is a ~4% smaller
output for large debug builds and a symbol table that no longer
resolves addresses to compiler-internal labels.
Co-Authored-By: Claude Fable 5 <[email protected]>
Previously, data appended to the output file, such as .gdb_index,
was staged in a heap buffer or a separate memory mapping. We can
instead just grow the existing mapping with mremap(MREMAP_MAYMOVE).
Consumers of the output buffer derive pointers from ctx.buf, so the
mapping is free to move, and since filesize now includes appended
data, the build ID and the .gnu_debuglink CRC naturally cover the
entire file.
Co-Authored-By: Claude Fable 5 <[email protected]>
When a live section refers to a __start_<name> or __stop_<name>
symbol, every section named <name> must be kept alive. The garbage
collector marked those sections one by one in the thread that
found the reference. A single reference can fan out to an enormous
number of sections: ClickHouse is built with XRay instrumentation,
which emits one xray_instr_map section per function, and the XRay
runtime references __start_xray_instr_map, so one thread marked
and visited a couple hundred thousand sections while the rest of
the machine waited for the mark phase to finish.
This was a regression introduced by 384527fd: before it,
C-identifier-named sections were unconditional GC roots and
entered the root set through a parallel loop. That commit
correctly made their retention conditional on a reference to
their marker symbols but expanded the fanout serially. Keep the
precise semantics and spread the fanout over threads with a
nested parallel loop, storing the sections of each name as a
vector instead of entries of a multimap.
This shortens ClickHouse's debug link from 1.98s to 1.45s at the
default 32 threads; its gc pass drops from 0.56s to 0.09s.
Co-Authored-By: Claude Fable 5 <[email protected]>
ConcurrentMap allocated its hash table with MAP_POPULATE, which
zeroes every page of the mapping inside the mmap system call, on
one thread. The string-merge map for a Clang debug build spans
several gigabytes, so every other thread stalled for ~170ms at the
barrier between splitting and inserting strings while one thread
pre-faulted the table.
Map the table lazily and advise MADV_HUGEPAGE instead. Populating
the table on demand from concurrent inserts spreads the fault cost
over all threads, huge pages make each fault populate 2MiB rather
than a base page, and they also cut the TLB miss rate of the
random probes into the table. Explicit parallel pre-faulting was
measured to add nothing on top of this. This shortens a Clang
debug link from 2.13s to 1.85s and a Chromium debug link from
3.38s to 3.27s at the default 32 threads.
Co-Authored-By: Claude Fable 5 <[email protected]>
.gdb_index was constructed in a malloc'ed buffer that close()
appended to the output with a single buffered write, copying the
entire section (534 MiB for a Chromium debug build) through the
page cache on one thread, into disk blocks that ext4 had to
allocate inside the write path.
Add OutputFile::extend, which returns a buffer for data appended
past the mapped image. A memory-mapped output file grows with
ftruncate, preallocates the new disk blocks, and memory-maps the
appended range, so write_gdb_index constructs the section directly
in the output file, in parallel. The range gets its own mapping
because growing the existing one with mremap could move it,
invalidating pointers into it held by the caller. Output files
that cannot be mapped (special files such as stdout, the separate
debug file, and Windows) keep buffering appended data on the heap,
now private to each implementation; the shared buf2 field is gone.
The fallocate calls are now guarded by a configure-time
HAVE_FALLOCATE check rather than #ifdef __linux__.
This shortens a Chromium debug link from 3.31s to 3.18s and makes
close_file effectively free.
Co-Authored-By: Claude Fable 5 <[email protected]>
RelDynSection::update_shdr collected dynamic relocations by walking
all chunks serially, and a single output section such as
.data.rel.ro can account for most of an output's dynamic
relocations (~1.5 million for a Chromium debug build, mostly
R_RELATIVE for vtables). The walk computes each relocation's offset
and addend with Symbol::get_addr, which made it a ~0.1s
single-threaded stretch between parallel passes.
Process an output section's absolute relocations in fixed-size
parallel shards and flatten the per-shard results, so the resulting
section contents are identical to the serial version. This shortens
a Chromium debug link from 3.58s to 3.49s at the default 32
threads.
Co-Authored-By: Claude Fable 5 <[email protected]>
parse_nonpositional_args tested each command line argument against
every option name, and each test allocated a few heap strings in
add_dashes. A command line for a large program consists almost
entirely of input file paths (a Chromium debug link passes ~19,000
paths but fewer than 100 options), so we were spending several
hundred milliseconds of serial time on failed option matches before
opening the first input file.
An argument that does not start with '-' cannot match any option,
so such arguments now go straight to the output list. This shortens
a Chromium debug link from 3.98s to 3.59s on a 64-core machine
because option processing delayed the start of the parallel parsing
phase.
Co-Authored-By: Claude Fable 5 <[email protected]>
SH-4 object files store relocation addends in the relocated places
rather than in r_addend, and mold assumed that dynamic relocations
work the same way, emitting them with zero r_addend and the addend
written to the relocated place. The glibc dynamic linker, however,
follows the usual RELA convention: it reads the addend of a symbolic
dynamic relocation only from r_addend, and it reads the relocated
place only for R_SH_RELATIVE relocations whose r_addend is zero. Any
symbolic dynamic relocation with a non-zero addend thus lost its
addend at load time.
A typical victim is the vtable pointer of a typeinfo object in a
position-independent executable. It must point into libstdc++'s
vtable (e.g. _ZTVN10__cxxabiv117__class_type_infoE + 8), so its
dynamic relocation carries a non-zero addend. With the addend
dropped, virtual calls on such a typeinfo dispatch to wrong vtable
slots, which crashes the C++ exception machinery as soon as a
personality routine tries to match a catch clause. The icf-lsda test
caught this on sh4aeb because its CI toolchain creates PIEs by
default; the little-endian sh4 toolchain defaults to
position-dependent executables, in which the vtable is copy-relocated
and no dynamic addend is involved, so the same test passed there.
Remove the ElfRel specializations that zeroed addends for symbolic
relocations; the generic RELA record types have the identical layout
and keep addends. Relocation records emitted for -r or --emit-relocs
keep following the object-file convention, since their consumers
expect addends in the relocated places.
Co-Authored-By: Claude Fable 5 <[email protected]>
Creating a .gdb_index was slower than it should have been for three
reasons. read_pubnames_cu looked up compunits with a linear scan,
which made it quadratic in the number of compunits; the compunit
vector is sorted by offset, so a binary search suffices. gdb_hash
was computed for every name occurrence even though the same name
typically appears in many compunits; we now compute it only when a
name is inserted into the map for the first time. Finally, nametypes
were sorted with a stable sort, but their ordering is total, so a
plain sort produces the same result without the extra buffer.
On a Chromium debug build, this makes the write_gdb_index pass 2x
faster with --threads=1 (10.2s to 5.2s) and 28% faster at the
default 32 threads (0.69s to 0.50s). The output is bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
Symbols are matched against version script patterns with an
Aho-Corasick automaton (for single-substring patterns) or an
NFA-based glob matcher, both of which scan the entire symbol name on
every query. But almost all patterns in real version scripts are a
match-all ("local: *;"), an exact name, or a name with one trailing
or leading "*", all of which can be matched with a plain string
comparison.
Handle such patterns directly and reserve the automatons for the
ones that actually need scanning, such as "*foo*" or "a*b".
Co-Authored-By: Claude Fable 5 <[email protected]>
The `README` gives users commands to copy, but `install-build-deps.sh`
is not ran as root, even though the CMake install step is indicated to
run with `sudo`. FreeBSD seems not to come with `sudo` by default, so
modifying the install script might be unwise. Simply place it in front
of the script invocation itself in the `README`.
The affinity partitioner replayed each round's chunk-to-thread
placement so that a thread would re-read the digests it wrote in the
previous round from its own cache. That no longer pays off: after the
recent ICF rework, the propagation loop streams over the digest and
edge arrays and is bound by memory bandwidth rather than cache reuse.
Linking Chromium on a Threadripper 7980X (64 cores, 8 CCDs), the
propagate timer is unchanged at 16 and 128 threads and loses only
about 2 ms (4%) at 32 threads, which is invisible in the total link
time. In exchange, the default auto partitioner retains full work
stealing, which balances better against the skewed per-section edge
counts, and the partitioner no longer needs to be threaded through
two function signatures.
Co-Authored-By: Claude Fable 5 <[email protected]>
compute_digest hashes an eligible relocation target without identity,
relying on a graph edge to keep different targets distinguishable
through the propagation rounds. But gather_edges only created edges
for a section's own relocations, not for the relocations in its FDEs,
so the reference from an FDE to its LSDA never became an edge. As a
result, two functions with identical machine code whose exception
tables catch different types were considered identical, and folding
them changes which exceptions the survivor catches. The bug is as old
as mold's ICF and also affects --icf=safe, since such functions are
usually not address-taken.
Create edges for FDE relocations, mirroring exactly what
compute_digest hashes. The included test, two byte-identical
functions catching different exception types, aborts with an uncaught
exception without this fix.
On 32-bit ARM the equivalent bug still exists through a different
mechanism: exception tables are .ARM.extab sections tied to functions
via .ARM.exidx rather than FDE references, which ICF does not yet
track. The test skips ARM for now.
A Chromium debug build is unaffected (same rounds, same number of
folded sections) since Chromium builds with -fno-exceptions.
Co-Authored-By: Claude Fable 5 <[email protected]>
gather_sections and gather_edges each kept a per-element count array
next to a starting-index array and combined the two at the end.
Counting directly into the index array shifted by one and running the
prefix sum in place removes the count arrays, and the extra entry at
the end means edge_indices[i + 1] is valid for every vertex, so the
propagation loop no longer needs to special-case the last vertex to
find where its edge list ends. The empty-input early return in
gather_edges falls out as well.
No behavior change.
Co-Authored-By: Claude Fable 5 <[email protected]>
A propagation round used to compute a vertex's next digest by hashing
its initial digest together with its neighbors' current digests,
which required keeping three digest arrays: the initial one and two
that alternate between current and next.
Hash the vertex's current digest instead of its initial one. This is
the textbook formulation of iterated partition refinement, and it
distinguishes exactly the same vertex pairs in every round: a current
digest determines the initial digest injectively (up to hash
collisions, which the whole algorithm already assumes away), so the
per-round partitions, the number of rounds and the final equivalence
classes are all unchanged.
The initial digest array disappears, saving 16 bytes per vertex, and
the current/next arrays are now two plainly named vectors that swap
after each round instead of an indexed triple with a flip-flop slot
index.
Co-Authored-By: Claude Fable 5 <[email protected]>
The propagation rounds used to record which vertices' digests had
stopped changing in a `converged` bitmap and skip rehashing them in
later rounds. That was worthwhile when rounds were expensive, but by
now a round is a cheap, uniform data-parallel pass, and the
bookkeeping costs as much as it saves: reading and comparing two
digests per vertex per round just to skip a short hash computation.
Rehash every vertex on every round instead. The convergence check was
the last user of Digest's equality operator, so that is gone too.
For a Chromium release build, the whole ICF pass got slightly faster
(0.166s to 0.152s wall, 4.7s to 4.3s of CPU time).
Co-Authored-By: Claude Fable 5 <[email protected]>
Leaf sections, those with no outgoing references, used to be merged
upfront so that they don't have to participate in the propagation
rounds as graph vertices. That was worth the code when the rounds and
the convergence check were expensive, but both have since become
cheap, and the propagation machinery handles leaves naturally: a
leaf's digest never changes, so it is marked as converged after the
second round and costs almost nothing from then on.
This deletes merge_leaf_nodes, is_leaf and the icf_leaf flag, and
moves eligibility classification to gather_sections. The leader
branch in hash_symbol is gone too because no section has a leader
anymore by the time digests are computed.
Also tidy up the neighborhood: the counting pass of gather_edges
skipped symbols that point to a section fragment before asking for
their input section, but a symbol's origin is a tagged pointer that
is either a fragment or an input section, so get_input_section()
already returns nullptr for fragment symbols. Dropping the redundant
check makes the counting pass and the filling pass identical in
shape.
For a Chromium release build, the ICF pass takes the same time as
before; the propagation rounds absorb roughly what merge_leaf_nodes
used to cost. The graph gets about 30% more vertices, which costs
some transient memory.
Co-Authored-By: Claude Fable 5 <[email protected]>
A digest used to be std::array<uint8_t, 16>, so DigestMap had to
memcpy the bytes into 64-bit words before deriving the slot index and
tag from them. Defining Digest as a struct of two u64 members gives
the words names and makes the copies disappear.
icf.cc is compiled once per target, so the struct is defined in every
one of those translation units. The definitions are all identical, so
this is fine under the one-definition rule, just as the old type
alias was.
No behavior change.
Co-Authored-By: Claude Fable 5 <[email protected]>
Leaf sections used to be merged with a dedicated hash map keyed by
raw section contents, with its own hasher and equality functor. That
equality was subtly weaker than the one the propagation rounds use:
it compared FDE bodies but ignored which CIE each FDE points to, so
two identical functions whose CIEs recorded different personality
routines could be merged incorrectly. It also ignored section flags.
Instead, compute the same digest for leaves that compute_digest
produces for graph sections and merge leaves through DigestMap, the
same structure the propagation rounds use. One definition of section
equality now covers both paths, which fixes the personality bug, and
LeafHasher, LeafEq and the last digest-keyed use of tbb's hash map in
this file disappear.
The trade-off is that leaf merging now relies on 128-bit digest
uniqueness instead of comparing contents exactly, but that is the
same assumption the rest of ICF is already built on.
For a Chromium debug build linked with --icf=all, merge_leaf_nodes
went from 0.07s to 0.10s because a leaf's contents are now hashed by
SipHash twice, once to insert and once to look up. 152 of 3.6M folded
sections are no longer merged; those are the sections the weaker
equality was conflating.
Co-Authored-By: Claude Fable 5 <[email protected]>
To group sections at the end of ICF, we used to build a concurrent
hash map from digest to the lowest-priority section holding it, which
duplicated most of what the digest-counting hash set already does.
That map was also slow enough to construct and destruct that we
deferred freeing it until exit.
This patch generalizes the counting set into DigestMap by adding a
section pointer to each slot. The counting rounds now elect the
lowest-priority section for each digest as a side effect, so by the
time the propagation loop converges, the final round has already
elected the leader of every equivalence class, and the grouping step
reduces to a single lookup pass. The digest-keyed tbb map and its
std::hash specialization disappear, and so does ctx.on_exit, whose
only purpose was to defer freeing that map.
For a Chromium debug build linked with --icf=all, the grouping pass
went from 0.23s to 0.01s and the propagation loop got 0.11s slower
for maintaining the leaders, making the whole ICF pass about 10%
faster (0.89s to 0.79s). The ICF fold decisions are unchanged.
Co-Authored-By: Claude Fable 5 <[email protected]>
To detect convergence of the ICF propagation rounds, we count the
number of distinct digests and stop when it ceases to increase. We
previously gated the count on a cheaper necessary-looking condition:
as long as the number of changed digests per round was still moving,
we assumed the graph could not have converged and skipped counting.
That assumption is wrong. A changing digest does not imply a changing
partition, because sections in the same equivalence class change
their digests in lockstep. For example, in a chain of sections that
are all distinct from the start, the partition is final after the
first round, but the changed-digest count keeps moving until the
propagation depth reaches the chain length, so the gate would delay
termination by the depth of the chain.
Since counting became cheap, we can simply count on every round and
stop as soon as the count repeats, which is the earliest round at
which convergence is provable. propagate() no longer needs to count
changed digests and report them to the caller.
For a Chromium debug build linked with --icf=all, this reduces the
number of propagation rounds from 40 to 28. The 12 saved rounds are
ones in which most sections had already converged, so the wall clock
time is unchanged. The output file is bit-identical.
Co-Authored-By: Claude Fable 5 <[email protected]>
To detect convergence of the ICF propagation rounds, we count the
number of distinct digests and stop when it ceases to increase. We
previously obtained the count by copying all digests and running a
parallel sort, which was expensive enough that we ran it only once
per 10 propagation rounds, overshooting the convergence point by up
to 9 rounds plus a full confirmation batch.
This patch replaces the sort with a concurrent open-addressing hash
set. Since the number of distinct digests cannot exceed the number of
digests, the table is allocated once at full size and never grows.
Slots are stamped with the round number, so the table needs no
clearing between rounds; and a slot match effectively compares an
entire 128-bit digest, so the count is exact under the same
hash-collision assumption the algorithm is already built on. Counting
is now cheap enough to run on every round that passes the existing
changed-digest gate, so the loop exits at the earliest round at which
convergence is provable, and the batching heuristic is gone.
For a Chromium debug build linked with --icf=all, the propagation
loop got 2.2x faster (0.73s to 0.34s), the number of propagation
rounds dropped from 59 to 40, and the whole ICF pass went from 1.29s
to 0.90s. The output file is unchanged.
Co-Authored-By: Claude Fable 5 <[email protected]>
This changes the bundled mimalloc.
mimalloc calls madvise(MADV_HUGEPAGE) on suitable memory regions only
when the caller also allows large OS pages, i.e. hugetlbfs pages.
That condition makes sense for hugetlbfs, whose pages are physically
committed at map time and thus shouldn't back lazily-committed
segments, but transparent huge pages have no such cost because their
pages materialize on fault. Since both were gated on the same flag,
huge mimalloc objects, which are allocated through a code path that
never sets the flag, were never eligible for THP under the madvise
THP mode that Linux distributions ship by default. mold's largest
data structures, such as the symbol and relocation arrays, are
exactly such objects; several gigabytes of them were backed by 4 KiB
pages.
This patch makes the THP advise depend only on the allow_thp option
(enabled by default except on Android) and leaves the hugetlbfs gate
as is. mimalloc v3 effectively behaves this way already; only the v2
line has this problem.
Linking a Chromium debug build gets about 2% faster in the default
fork mode (3.37s to 3.29s) and 15% faster with --no-fork (4.05s to
3.43s) on our machine, as the heap becomes almost fully backed by
huge pages, which also makes the process exit much cheaper.
Co-Authored-By: Claude Fable 5 <[email protected]>
The kernel reclaims an exiting process's address space in a single
thread, which takes an appreciable amount of time for a process that
maps tens of gigabytes of input files. We now drop the page table
entries for input file mappings with madvise(MADV_DONTNEED) on all
threads before exiting. File contents stay in the page cache, so warm
rebuilds are unaffected.
This makes linking a Chromium debug build with --no-fork about 11%
faster (4.55s to 4.05s) on our machine. Without --no-fork, the
teardown happens in the detached child process, which now releases
its memory sooner.
Co-Authored-By: Claude Fable 5 <[email protected]>
The PR_SET_THP_DISABLE prctl flag is inherited across fork and exec,
so a linker spawned from a process that has disabled transparent huge
pages for itself also runs without them, no matter what the system
THP setting says. mold allocates on the order of 10 GiB of heap for a
large link, and backing it with 4 KiB pages instead of 2 MiB pages
makes linking nearly twice as slow (6.2s vs 3.4s for a Chromium debug
build on our machine), so clear the inherited flag at startup.
Co-Authored-By: Claude Fable 5 <[email protected]>
lld does this by default. See: https://crbug.com/529102889
If a DSO is only referenced by sections that are garbage-collected, the
DSO's DT_NEEDED entry will be omitted.
These changes broke cross builds for ppc64le and s390x. I couldn't fix
it easily. Every fix attempt also takes a long time because of slow
qemu-user emulation.
I'll revert CMakeLists.txt and dist.sh to the original state. I don't
think we need to upgrade distros used by dist.sh as long as the binaries
produced using these images work.
Now our tar version is recent enough to use the `--sort` option.
We still need to run gzip explicitly in the pipeline because we want
to pass -n (--no-name) to not save the current time as the file
timestamp in a gzip header.
* update x86-64 to Debian 10. This has glibc 2.28 which matches the oldest actually used distro these days (rhel 8) and gcc-8.5 is new enough to bootstrap clang and libc++, which saves the whole gcc build step
* set the riscv64 target to use debian Trixie, which was the first stable release for that arch
* bullseye (arm,ppac,s390), trixie (riscv), and the loongarch sid snapshot all have clang-19 available
* make sure all images install the libc++-19-dev package to get libc++ static version
* MOLD_USE_LIBCXX now configures the build to look for and use llvm's libc++. This is already the default on macOS and BSD
* Added MOLD_STATIC_* flags for several libs to give more config beyond the overall MOLD_MOSTLY_STATIC
When mold relaxes an instruction, the relocation attached to it no
longer matches the rewritten code. With --emit-relocs that mismatch is
written to the output and confuses consumers such as BOLT. This change
addresses it by rewriting each emitted relocation to the type that
matches the relaxed instruction.
RISC-V and LoongArch already did this in a separate rewrite_reloc_types()
pass that could drift from apply_reloc_alloc; fold it into the latter and
delete the pass. ARM64, x86-64, i386 and s390x did not rewrite types
and now do, e.g. x86-64 GOTPCRELX => PC32, i386 GOT32X => GOTOFF, s390x
GOTENT => PC32DBL, ARM64 GOT loads => ADR_PREL_PG_HI21 and TLSDESC =>
IE/LE. Markers that GNU ld also keeps, such as x86-64 TLS and sparc64
GOTDATA_OP, are left unchanged.
Fixes https://github.com/rui314/mold/issues/1610
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A symbol exported with an empty default version (foo@@) was treated as a
plain unversioned symbol, so a version script's "local: *" wildcard
localized it and dropped it from the dynamic symbol table; GNU ld keeps
it exported. Route foo@@ through parse_symbol_version() and assign it
VER_NDX_GLOBAL so it overrides the wildcard.
Signed-off-by: Sun Yuechi <[email protected]>
SFrame is a compact alternative to .eh_frame used for asynchronous stack
unwinding by profilers and debuggers. As of binutils 2.46, gas emits
.sframe (Version 3) by default with -Wa,--gsframe, so object files now
routinely carry it. Like .eh_frame, .sframe can't be treated as opaque
bytes: an output section is a single header followed by a PC-sorted index
of Function Descriptor Entries (FDEs) and a blob of Frame Row Entries
(FREs), so the linker has to parse the inputs and rebuild it.
This adds SFrame Version 3 support. We parse each input .sframe into its
FDEs, drop the entries for garbage-collected or ICF-folded functions,
concatenate the survivors, sort the index by function address and rewrite
the header. The FRE data carries no relocations and is position
independent, so it is copied verbatim; only each FDE's func_start is
relocated, which we resolve to a PC-relative offset (the output always
sets SFRAME_F_FDE_FUNC_START_PCREL). A PT_GNU_SFRAME segment is emitted so
the runtime can find the section.
For a relocatable output we can't resolve addresses yet, so the merged
index is left unsorted and each func_start is emitted as a relocation in
.rela.sframe for the final link to resolve, matching GNU ld -r.
We support only Version 3. A .sframe with the wrong magic is reported as
an error, whereas a section written in an older version is ignored.
Fixes https://github.com/rui314/mold/issues/1511
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Large C++ debug libraries on ARM64 (e.g. deal.II's libdeal_II.g.so) can
have a single batch reference enough distinct PLT-routed DSO imports that
the first-pass thunk group exceeds the old fixed 1 MiB max_thunk_size,
tripping the assertion in create_range_extension_thunks().
Replace the branch_distance/5 formula with explicit per-target batch
sizes (32/8/16 MiB for ARM64/ARM32/PPC) and derive max_thunk_size from
the batch as batch_size/2 (16/4/8 MiB). The reservation in d_thunk_end()
already accounts for max_thunk_size, so a larger cap is safe by
construction, and batch_size/2 is about the largest reservation ARM32's
16 MiB branch range can accommodate without throttling the batch.
Add a regression test that calls 50,000 imported functions from a single
function, which on ARM64 produces a first-pass thunk group well over the
old 1 MiB limit.
Fixes https://github.com/rui314/mold/issues/1538
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
actions/upload-artifact rejects names with a forward slash or colon, so
the test-results upload failed for the opensuse/tumbleweed matrix entry
(and would have failed for gentoo/stage3, debian:11, and the ubuntu:*
entries on a test failure). Sanitize matrix.distro by mapping / and :
to - before using it as the artifact name.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
After relaxation, the relocations attached to the relaxed instructions no
longer match the new, shorter instructions. With --emit-relocs we copied the
input relocations verbatim, emitting stale types at stale offsets.
Rewrite each relocation's offset and type in RelocSection::copy_buf to match
the relaxed code, recomputing the outcome the same way apply_reloc_alloc
recomputes its instruction rewrites. -r output is not relaxed, so its
relocations are passed through unchanged.
Fixes https://github.com/rui314/mold/issues/1584
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The Android packed relocation section (SHT_ANDROID_RELA/REL) holds a
variable-length SLEB128-encoded delta stream, not a table of fixed-size
entries, so sh_entsize should be 0 and the byte stream needs no word
alignment. We were inheriting sizeof(ElfRel) and sizeof(Word) from the
plain .rela.dyn path. These fields aren't consulted at runtime (bionic
locates the blob via DT_ANDROID_RELA/RELASZ), so this only corrects the
metadata and lets the section pack without spurious padding.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
When folding identical sections, ICF was discarding the alignment of the
folded sections and only keeping the alignment of the chosen leader. If
a section with stricter alignment (e.g. 4-byte) was folded into a
section with looser alignment (e.g. 1-byte), the folded function could
be placed at an odd address.
On x86/x86_64, member function pointers use the lowest bit of the
address for virtual/non-virtual distinction. If a non-virtual member
function is folded and placed at an odd address, it is incorrectly
interpreted as virtual, leading to crashes. This fix ensures that we
update the leader's alignment to be the maximum alignment of all folded
sections. We do this in a global pass over all sections (both leaf and
non-leaf merged sections) before the sweep phase.
Fixes#1596
If a relocation points to a preemptible (imported) symbol, it should not
be considered identical to a relocation pointing to a local symbol, even
if their targets currently have identical bodies. Preemptible symbols
can be overridden at runtime, so we must preserve their unique identity.
Fixes#1598
With a PT_LOAD segment, we need to make sure p_offset % p_align ==
p_vaddr % p_align. In set_file_offsets, we try to assign contiguous file
offsets when sections are contiguous in memory. But when the alignment
of section changes and it's bigger than page size, we cannot do that.
The offset might need to be adjusted to follow the alignment rule.
A test binary with
__attribute__((section(".sum"))) __attribute__((aligned(8192))) void sum() {}
Section Headers:
[16] .plt.got PROGBITS 00000000000015c0 0005c0 000008 00 AX 0 0 16
[17] .sum PROGBITS 0000000000002000 001000 00000b 00 AX 0 0 8192
.sum section is assigned contiguous offset to its previous section. As a
result, in create_phdr, .sum is mapped to the same PT_LOAD segment as
previous sections.
Program Headers:
LOAD 0x000588 0x0000000000001588 0x0000000000001588 0x000b80 0x000b80 R E 0x2000
p_offset = 0x588
p_vaddr = 0x1588
p_align = 0x2000
So p_offset % p_align != p_vaddr % p_align, it violates the alignment
rule. As a result, if this scenario happens to a shared library, it
cannot be dlopen() on a system with glibc < 2.35 which is before this
commit:
https://github.com/bminor/glibc/commit/163f625cf9becbb82dfec63a29e566324129c0cd.
Fix it by not assigning contiguous offset when section's alignment is
different and it's bigger than page size.
Several parse-time sites converted `symbol_strtab.data() + st_name`
implicitly to string_view, running strlen on each call. Precompute
a string_view vector parallel to elf_syms and read from it.
InputSection::name() implicitly converts a `const char *` to
string_view, which calls strlen on the shstrtab entry at every call.
Precompute the name as a string_view once in the constructor.
Linking clang Debug is ~1.3% faster; mlir-opt Debug is ~2.6% faster.
ConcurrentMap::insert reads ent.key before writing, so on an empty
slot the read first installs a PTE pointing at the system zero page,
then the CAS triggers wp_page_copy and a cross-CPU TLB shootdown.
Under heavy parallel insertion this shows up as ~3% of link time in
smp_call_function_many_cond plus a much larger inclusive cost in
asm_exc_page_fault.
Pass MAP_POPULATE to pre-fault the whole map in the mmap() thread.
Reads no longer install zero-page PTEs and writes no longer trigger
CoW. Linking clang-23 Debug is ~5% faster, llvm-tblgen ~7%,
clang-tblgen ~9%; max RSS rises by 0.1-0.6%. Falls back cleanly on
platforms that don't define MAP_POPULATE.
Several parse-time sites converted `symbol_strtab.data() + st_name`
implicitly to string_view, running strlen on each call. Precompute
a string_view vector parallel to elf_syms and read from it.
InputSection::name() implicitly converts a `const char *` to
string_view, which calls strlen on the shstrtab entry at every call.
Precompute the name as a string_view once in the constructor.
Linking clang Debug is ~1.3% faster; mlir-opt Debug is ~2.6% faster.
Commit 6d6a128b moved the -C chdir(2) into parse_nonpositional_args(),
which is invoked by both the speculative X86_64 mold_main() and the
re-run via redo_main(). When -C is in effect, the second invocation
re-entered expand_response_files() from the post-chdir directory, so
@rsp with a relative path could no longer be opened.
Save the cwd before parsing and restore it before redo_main() so the
re-run resolves relative paths from the original directory. Also
switch the chdir call itself to std::filesystem::current_path().
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
src/mold.h (HANDLE in MappedFile) and lib/perf.cc (FILETIME /
GetProcessTimes / GetCurrentProcess) both use Windows APIs inside
_WIN32 blocks but neither included <windows.h>. The MSYS2 UCRT64 build
worked because <tbb/enumerable_thread_specific.h> (pulled in via
lib/lib.h) used to #include <windows.h> for its TLS code.
oneTBB 2023.0.0 replaced that bulk include with forward declarations
of just the TLS functions ("declare prototypes of just the needed TLS
functions, instead of including whole windows.h"). When MSYS2 picked
up 2023.0.0, build-msys started failing with "'HANDLE' does not name
a type" at src/mold.h:224 (and the same shape of error in lib/perf.cc
after the mold.h fix landed).
Include <windows.h> explicitly in both translation units so the build
doesn't rely on transitive paths through third-party headers.
Last passing run: https://github.com/rui314/mold/actions/runs/25585325507/job/75112599227
First failing run: https://github.com/rui314/mold/actions/runs/25767054228/job/75681837767
A compile unit's DW_AT_ranges attribute in DW_FORM_rnglistx form holds an
index into the offset table at DW_AT_rnglists_base; the corresponding
entry locates the range list for that DIE. mold was ignoring the index
and iterating every offset, picking up range lists belonging to child
DIEs (subprograms, lexical_blocks, inlined_subroutines) that use
DW_FORM_rnglistx as well.
The resulting .gdb_index address table contained child-DIE ranges nested
inside the CU's range. GDB 17 detects the overlap, prints "warning:
.gdb_index address table has a range (...) that overlaps with an earlier
range, ignoring .gdb_index", and discards the index. Older GDBs accept
the bloated table without complaint -- the duplicate entries all point
to the same CU, so lookups still resolve correctly -- which is why this
went undetected for years.
The bug only triggers when a CU contains a non-contiguous lexical_block,
which clang emits for scopes with EH landing pads from throwing RAII
destructors. Replace the loop with offsets[ranges.value], matching the
DW_FORM_sec_offset branch and GDB's own read_rnglist_index.
The new test invokes clang++ directly since DW_FORM_rnglistx is a
clang-specific behavior; gcc through 13 uses DW_FORM_sec_offset, which
mold has always handled correctly. The test skips cleanly if clang++
isn't installed or doesn't emit rnglistx for the source.
Reported by @MikeWang000000.
Fixes https://github.com/rui314/mold/issues/1578
When --dynamic-linker is passed alongside -shared, mold currently emits
a .interp section and a PT_INTERP program header. Both lld and BFD ld
silently drop --dynamic-linker for -shared outputs because shared
libraries are loaded by the dynamic linker rather than bootstrapped by
the kernel, so PT_INTERP serves no purpose there.
Gate .interp creation on (!shared || pie) so we keep it for regular
executables, -pie executables, and -shared -pie runnable shared
libraries (the libc.so.6-style case), and drop it only for the pure
shared-library case. The PT_INTERP program header at
output-chunks.cc:219 is conditional on ctx.interp being non-null, so
the segment vanishes automatically with no second edit.
Fixes https://github.com/rui314/mold/issues/1572
The Warn() machinery has consulted ctx.arg.suppress_warnings since
6d2dbd42 ("Preparation to support Apple ld64's -w option", March
2023), but no cmdline flag ever set it. Wire -w (the GNU/BFD
spelling) and --no-warnings (the lld long form) to set the field.
Because Warn::Warn early-returns when the flag is true, before the
ctx.arg.fatal_warnings branch sets has_error, -w also cancels
--fatal-warnings — matching lld's documented behavior.
The "warnings only" semantics match GCC, clang, and lld: real errors
still print and still fail the link. BFD ld's -w is broader and
suppresses error messages too, which would hide the cause of link
failures.
Fixes https://github.com/rui314/mold/issues/1493
Linker scripts produced for lld (e.g., the version script Chrome's
build emits) use the POSIX/shell convention `[!...]` to negate a
bracket character class. mold previously accepted only `[^...]`, so
those scripts failed to match. Accept both, treating `!` and `^`
identically at the start of a bracket expression.
Fixes https://github.com/rui314/mold/issues/1575
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Commit 6b8293bf changed the early `ver` computation in
SharedFile::parse so that undef symbols use the raw versym instead
of being forced to VER_NDX_GLOBAL. Some BFD ld versions (notably
binutils 2.35 on Debian 11) emit VER_NDX_LOCAL (0) as the versym
for unversioned undefined dynamic symbols, so the subsequent
`if (ver == VER_NDX_LOCAL) continue;` check silently dropped those
undef references. With the DSO's reference invisible to mold, the
matching object-file definition was never marked as exported, and
at runtime the dynamic loader failed with "undefined symbol".
Restore the old behavior: defined symbols with VER_NDX_LOCAL are
still skipped (they aren't really exposed for dynamic linking),
but undef references are always processed.
Also fix the verdef/verneed merge in the same path. vd_ndx and
vna_other share a single versym index space (a producing linker
assigns them from one counter), so a single vector covers both.
read_verdef and read_verneed are folded into one read_version_strings
that populates that shared-index vector directly, replacing the
buggy `append` concatenation.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
For an undefined symbol in a shared library, mold treated the versym
as VER_NDX_GLOBAL and registered the symbol under its bare name. That
meant an undef foo@VER_1.0 in libbar.so was looked up as just foo, so
a non-default foo@VER_1.0 in another DSO did not satisfy it and
--no-allow-shlib-undefined wrongly reported "undefined symbol: foo".
Parse .gnu.version_r and, when an undef has a versym indexing into it,
register the symbol as name@VERSION so the existing versioned-symbol
resolution path picks up matching definitions.
Fixes https://github.com/rui314/mold/issues/1571
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
`readelf | grep -q` plus `set -o pipefail` is fragile: when grep -q
matches it stops reading immediately, readelf gets SIGPIPE on its next
write, and pipefail propagates the resulting 141. Alpine (musl) hits
this where glibc happens to escape it. Drop -q so grep drains stdin.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The new StartStopMap parameter contains std::string_view, so the
unqualified call to visit() inside mark() and the recursive call inside
visit() itself were resolved to std::visit by ADL on gcc 10 (Debian 11),
breaking the build.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A section whose name is a valid C identifier was treated as a GC root,
keeping it (and everything reachable from it) alive even when nothing
referenced its synthesized __start_/__stop_ marker symbols. That could
pull in unrelated code with undefined references and cause spurious
"undefined symbol" errors that lld and ld.bfd don't produce.
Build a multimap of c-identifier-named sections up front and, during
marking, when a relocation targets __start_<name> or __stop_<name>,
mark every section named <name> alive.
Fixes https://github.com/rui314/mold/issues/1571
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Pre-Android-12 (API 30 and earlier) bionic loaders predate the official
ELF gABI assignment of DT_RELR/DT_RELRSZ/DT_RELRENT and use legacy
OS-specific tag values DT_ANDROID_RELR (0x6fffe000), DT_ANDROID_RELRSZ
(0x6fffe001), DT_ANDROID_RELRENT (0x6fffe003), and section type
SHT_ANDROID_RELR (0x6fffff00). With --use-android-relr-tags, mold emits
those instead of the standard ones so binaries linked with
--pack-dyn-relocs=relr can be loaded on those older devices. The flag
is undocumented (not listed in --help) since it's only relevant to
shipping to legacy Android targets.
https://github.com/rui314/mold/issues/1544
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Implements the Android Packed Relocations Format (APS2) used by Android
binaries to compress dynamic relocations. Two new modes are accepted:
`--pack-dyn-relocs=android` encodes all dynamic relocations as APS2 in
.rela.dyn (SHT_ANDROID_RELA) and emits DT_ANDROID_RELA(SZ) instead of
DT_RELA(SZ); `--pack-dyn-relocs=android+relr` is the same but with
R_RELATIVE relocations diverted to .relr.dyn via the existing RELR path.
The encoder groups consecutive relocations sharing offset_delta and
r_info to amortize SLEB128 overhead. The encoded bytes are decoded by
Android's bionic loader and by llvm-readelf, which the new test uses to
verify correctness.
main.cc gains a final set_osec_offsets() call after fix_synthetic_symbols.
Symbol values assigned there change addends for relocations that
reference synthetic symbols, which can shift the SLEB128-encoded section
size; re-running the layout convergence keeps sh_size consistent with
the final encoding.
Fixes https://github.com/rui314/mold/issues/1544
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Previously each chunk wrote its dynamic relocations into a magic offset
in .rel.dyn (tracked via per-chunk reldyn_offset), with a separate
construct_relr pass building per-chunk relr vectors. Now each chunk
returns its dynrels via a virtual collect_dynrels(), and RelDynSection
aggregates them into a single relocs vector that it owns and writes.
RELR-eligibility is determined by checking r_offset alignment of the
collected ElfRel entries (replacing the ABS_REL_RELR kind).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Generalize the zstd:N syntax added in the previous commit so that the
zlib compression level can also be specified. zlib levels 0 through 9
are accepted, where 0 means no compression. Plain `zlib` remains
equivalent to `zlib:1` (the previous hardcoded default).
Also drop the objcopy/zstdcat round-trip from the zstd-level test in
favor of a simpler readelf check, matching the new zlib-level test, and
add error-message coverage for out-of-range levels.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Change the compression level syntax from zstd-<level> to zstd:<level>
as requested by @rui314 for consistency with lld.
Signed-off-by: Pierluigi Lenoci <[email protected]>
Allow users to specify a zstd compression level (1-22) for debug section
compression via --compress-debug-sections=zstd-N, where N is the desired
level. Higher levels achieve better compression ratios at the cost of
compression speed. The plain 'zstd' option remains equivalent to 'zstd-3'
(the previous hardcoded default).
Closes: https://github.com/rui314/mold/issues/1290
Signed-off-by: Pierluigi Lenoci <[email protected]>
pcaddi computes PC + (imm << 2), so the relaxation from a GOT load
(pcalau12i + ld.d) to pcaddi is valid only when the symbol's address
is 4-byte aligned. Without this check, a reference to a misaligned
symbol was relaxed to a pcaddi that resolves to the address with its
low two bits cleared, producing a wrong pointer.
Fixes https://github.com/rui314/mold/issues/1567
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The parallel writer that fills (type, cu_idx) tuples for each name
appended entries in thread-scheduling order, so names referenced from
multiple compilation units produced different .gdb_index bytes across
runs. Sort each name's tuple array after the parallel append so the
output is reproducible while preserving parallelism.
Fixes https://github.com/rui314/mold/issues/1566
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Previously, --omagic only flipped segment permissions to RWX but
still page-aligned PT_LOAD segments and emitted PT_GNU_RELRO, so
text and data ended up in separate page-padded segments. This
mismatched the documented behavior ("makes the first data segment
not aligned to a page boundary") and GNU ld's --omagic, which also
disables shared linking.
Make --omagic imply --Bstatic, disable RELRO, and stop forcing
PT_LOAD's p_align to the page size. Also gate chunk->is_relro on
z_relro so a disabled RELRO no longer leaks into address layout.
Fixes https://github.com/rui314/mold/issues/1562
When linking with LTO, archive members extracted pre-LTO to satisfy
references from IR objects could cause false duplicate symbol errors.
This happened in two ways:
1. The pre-LTO check_duplicate_symbols flagged conflicts between LTO IR
objects and regular objects. These conflicts are handled by the LTO
backend and should not be reported.
2. After LTO compilation, archive members that were extracted pre-LTO
retained stale is_reachable flags even when the LTO output superseded
all their symbols. The post-LTO duplicate check then saw both the
archive member and the LTO output defining the same symbol.
Fix both issues:
- In check_duplicate_symbols, skip errors where one side is an LTO IR
object and the other is not. Only IR-vs-IR duplicates need catching.
- In do_lto, reset reachability for archive members before the post-LTO
resolve_symbols so that only genuinely needed members are re-extracted.
Fixes https://github.com/rui314/mold/issues/1421
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
This relocation is used by clang's -fexperimental-relative-c++-abi-vtables
to emit 32-bit GOT-relative PC-relative offsets in vtable entries (e.g. for
RTTI pointers). It resolves to G + GOT + A - P as a 32-bit value.
Fixes https://github.com/rui314/mold/issues/1561
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
This patch fixes two distinct blind spots in how `mold` handles
--no-allow-shlib-undefined checks during the final link phase:
1. The Hidden Static Provider Bug (False Negative)
Previously, if a shared library requested a symbol that was provided by a
static object file with STV_HIDDEN, `mold` would silently overwrite the
global symbol visibility to STV_DEFAULT and rubber-stamp the binary,
resulting in a guaranteed `symbol lookup error` at runtime.
Fix: We now interrogate the raw `elf_syms` array of the static object
provider. If the original visibility is hidden/internal, we flag it as an
invalid provider and throw the error, matching GNU ld/lld behavior.
2. The LTO Shadowing Bug (False Positive)
When LTO is heavily utilized, GCC's plugin may optimize away symbols that
are theoretically provided by static objects if they aren't used internally.
If a shared library still needs that symbol, `mold`'s static state
registered the provider as NULL and threw an error, ignoring the fact
that a loaded DSO (like libc.so) offered a valid public fallback.
Fix: Introduced `has_public_dso_fallback()`. Before throwing an undefined
symbol error, `mold` performs a rapid scan of `ctx.dsos`. If a public
fallback exists dynamically, the error is safely suppressed.
Originally discussed in #1535, the debug data generated by
--separate-debug-file is being corrupted. In the best case scenario,
there's many blank/nameless undefined symbols in the symtab, but in
most production scenario I've encountered, the file is corrupt / gdb
refuses to load it:
$ gdb main
Reading symbols from $PWD/main.dbg...
BFD: /$PWD/main.dbg: invalid string offset 4130582 >= 1345 for section `.strtab'
BFD: /$PWD/main.dbg: invalid string offset 16777228 >= 1345 for section `.strtab'
$ readelf -Ws main.dbg
Symbol table '.symtab' contains 139 entries:
[...]
38: 0000000000000000 0 SECTION LOCAL DEFAULT 38 .strtab
39: 0000000000000000 0 SECTION LOCAL DEFAULT 39 .symtab
40: 0018a80200010000 0xb10000003f071500 NOTYPE LOCAL DEFAULT bad section index[3232] <corrupt>
41: 0000003f07140000 0xfd02000100000cc2 NOTYPE LOCAL HIDDEN bad section index[11500] <corrupt>
42: 02000100000cd300 0x3f0712000008a3 <unknown>: 7 LOCAL PROTECTED [<other>: 3c] UND <corrupt>
43: 3f07110000056202 0x100000cf5000000 NOTYPE LOCAL DEFAULT 1 <corrupt>
This appears to be happening because the strtab size is set to be larger
than its true value, causing garbage data to be placed in it, which
breaks the symtab.
This regression occurred in 5b71029, which fixed a different form of
corruption (the PHDR symbol escaping into the section headers, shifting
the section offsets), but created this strtab corruption. The root cause
is that `compute_section_headers` was moved prior to the wiping of
irrelevant chunks. `compute_section_headers` computes the strtab size by
adding up individual chunk strtab contributions. It needs to occur _after_
the wiping so that the strtab doesn't account for the data before the
wipe.
The core of this patch is really just a partial revert of 5b71029--the
compute_section_headers call appears to have been in the correct place
prior to this change; I've added a test that catches the issue by
intentionally bloating the size of the `.plt` section with many external
symbols. This section is wiped in the dbg data, so by bloating it, the
test can inspect for corruption.
Fixes#1535
Signed-off-by: Alex Karle <[email protected]>
When mold auto-detects an unsupported target from ELF headers,
redo_main() previously called abort() with no context. This replaces
it with a descriptive error message matching the style used by the
-m flag path in cmdline.cc.
Fixes#1553
Interestingly, calling fallocate on tmpfs is very slow. It took
almost a second for a fresh ~5 GiB file, even though intuititvely
it should finish very quickly.
Make Script<E>::get_script_output_type support AS_NEEDED inside an INPUT
or GROUP group, so that the libatomic_asneeded.so linker script used by
GCC 16 works.
Fixes#1545
Signed-off-by: Jonathan Wakely <[email protected]>
This change eliminates the cost of zero-initialization on vector
resizing by using malloc/realloc instead. We also parallelize the
loops to write adddress areas and types to .gdb_index.
We read version script files in cmdline.cc, so we need to handle -C
and --chroot early. Otherwise, a version script passed by command line
can cause a "file not found" error.
By popular request, this commit adds a new cmake variable that allows
building mold with a limited set of supported targets, which can help
reduce build time.
The use of this flag is discouraged unless you build mold frequently
for personal use. Otherwise, build mold with the default settings for
simplicity and maximum compatibility. In particular, distributions
should not build mold with a limited set of supported targets.
Fixes https://github.com/rui314/mold/issues/1364
Read-only data sections containing uninitialized variables are essentially
useless because reading from them always returns 0 at any time, so if they
exist, they are there for a reason. We do not want to automatically
convert it to a BSS.
https://github.com/rui314/mold/issues/1514
Currently, when using --static inside --push-state and --pop-state, we
incorrectly mark the global state as static. This commit changes to the
correct behavior where global state is marked as static only when
--static is used at the outer most (outside of any --push-state,
--pop-state).
Quote from SH-4 CPU Core Architecture manual:
The opcode 0xFFFD is guaranteed to be defined in any SH-4
architecture revision. Other unused opcodes may be treated as
reserved in any particular SH-4 implementation.
The only undefined opcode which the architecture guarantees to cause
a General Illegal Instruction Exception is 0xFFFD.
On 64-bit ARM, mimalloc is built with the compiler flag
`-march=armv8.1-a` by default. This flag advises GCC to make use of
ARMv8.1-A-specific instructions such as `ldaddal`. Machines based on the
older ARMv8-A instruction set - for example the Raspberry Pi 4 -
experience 'Illegal instruction (core dumped)' crashes as a result.
To ensure compatibility with machines that do not support ARMv8.1-A,
disable architecture-specific optimisations by setting the CMake option
`MI_NO_OPT_ARCH` to `ON`.
As of mimalloc 2.2.3, 64-bit ARM is the only architecture affected by
the `MI_NO_OPT_ARCH` option.
It turns out that at least clang-18 fails to compile our code to load
a 32-bit native-endian value from memory using shift + bitwise OR into
a single instruction when compiled with `-march=native`. It instead
compiles the code to complicated vector instructions.
It seems that the good old bswap and memcpy are the best option to
prevent the compiler from overthinking.
I asked ChatGPT how to match multiple glob patterns simultaneously with a
given input string, while avoiding the textbook-style NFA-to-DFA conversion.
Then it suggested that I implement a bitvector-based NFA simulation
algorithm that I wasn't aware. I don't think I could have come up with it
myself easily. This is impressive. ChatGPT is so good at programming and
sometimes much better than me! It may not be oto long before AI writes all
the code for me.
Our glob pattern matcher was an exponential-time implementation.
This can easily be improved by avoiding recursion as explained at
https://research.swtch.com/glob.
I learned that tools supporting compressed sections generally support
concatenated zlib streams within a single compressed debug section.
Examples include GNU/LLVM binutils and libbacktrace. So let's emit
compressed data that way, because it's much easier to do.
It is actually desirable to emit multiple small compressed blocks for
extremely large debug sections, because zlib doesn't support compressed
streams larger than 4 GiB on machines where the long type is 32 bits.
Context: https://sourceware.org/pipermail/binutils/2025-June/141514.html
Chunk<E>::uncompressed_data can be very large as it contains an entire
output section contents. std::vector<u8>::resize() zero-initializes the
contents. We want to avoid that cost.
On my AMD Threadripper 7980X machine, zero-initializing 1 GiB of
memory takes ~200 ms. I can confirm that this change shaves off ~1.2s
for an executable with 5 GiB of debug info sections in uncompressed
form.
Unlike .symtab, .dynsym cannot have an extended symbol table, so a
section index that is larger than 65279 is not representable in an
output.
Fixes https://github.com/rui314/mold/issues/1468
Recent versions of BLAKE3 library can be built with OneTBB for parallelization.
If that is enabled, we don't want to build and link our bundled OneTBB to avoid
conflict.
Fixes https://github.com/rui314/mold/issues/1466
I believe it is unlikely that sections such as .debug_line grow beyond
4 GiB to cause relocation overflows, but doing some extra work is
trivial, so just do it.
Previously, if the combined size of DWARF32 and DWARF64 debug
information exceeded 2^32 bytes, a relocation overflow could occur.
Since not all static libraries are built with -gdwarf64, this overflow
issue prevented some very large programs from using static libraries
built without -gdwarf64. Recompiling everything with -gdwarf64 is the
"right" way to fix it, but that's not always feasible for various
reasons.
This commit mitigates the issue.
By sorting the input .debug_info sections so that DWARF32 debug info
comes before DWARF64, a relocation overflow can occur only if the size
of DWARF32 debug info alone exceeds 2^32 bytes.
A call to weak undefined symbol could be relaxed as long as the symbol
could be promoted to a dynamic symbol, where the relaxation is done
against the known PLT address and the address of the symbol itself
doesn't really matter.
Signed-off-by: Yao Zi <[email protected]>
Previously, symbols specified with the `global:` label were unversioned
even if --default-symver was given. Now, such symbols are versioned with
the output file's soname.
Fixes https://github.com/rui314/mold/issues/1448
It looks like `podman build` command does not push the newly created
image with the `latest` tag to ghcr.io, so the following `podman run`
pulled an older image. That caused an issue that the scirpt created
a different tarball than the one on my local machine.
I don't know what is the "proper" fix, but it looks like adding
`--pull=never` to `podman run` forbids it from pulling an older image
from ghcr.io, solving the issue.
If a versioned symbol is a default one, it can be referred to with two
names, e.g. `foo` and `foo@VERSION`. Previously, we forward
`foo@VERSION` to `foo` if it is undefined, but it didn't always work
correctly. Now, we resolve both `foo` and `foo@VERSION` as usual and
then forward `foo@VERSION` to `foo` afterwards.
Fixes https://github.com/rui314/mold/issues/1169
mimalloc 2.2.2 started using __DATE__ and __TIME__ macros which make
the output non-reproducible. We can set the SOURCE_DATE_EPOCH environment
variable to fix the "current" timestamp.
Before 9306f69a61, we completely ignored
duplciate symbol errors in IR object files. After that commit, we've
been accidentally reporting duplicate comdat symbols as errors.
Fixes https://github.com/rui314/mold/issues/1434
It looks like Linux is going to support CET only on x86-64 (i.e. endbr64),
and it is very unlikely that x86-32 will get CET support. So I'll remove
the instruction from the code we generate for x86-32.
Removes the temporary file created by the compiler
from the dependency list created with --dependency-file.
Without this build systems will never report "Nothing to do",
but always do a re-link of the final object.
R_X86_64_CODE_5_{GOTPCRELX,GOTTPOFF,GOTPC32_TLSDESC} and
R_X86_64_CODE_6_{GOTPCRELX,GOTPC32_TLSDESC} are defined just for
completeness (*), so we've implemented all APX relocations with
this change.
* https://gitlab.com/x86-psABIs/x86-64-ABI/-/merge_requests/53
These relocations are for Intel APX (Advanced Performance Extension)
that extends the number of general-purpose registers from 16 to 32.
GNU assembler that comes with GCC 14 or newer can compile the test code.
I believe this is just due to the lower number of users on ARM64
runners compared to x86. However, based on my measurements, ARM64
builds all binaries in 1h30m, whereas x86 takes 2h30m.
We need to stop using `grep -q` because grep terminates immediately after
finding the first match with `-q`, causing SIGPIPE to the process on the
other side of the pipe.
Previously, these symbols were defined relative to a section. That is
logically correct, because the runtiem is supposed to apply IREL
relocs between `__rela_iplt_start` and `__rela_iplt_end`, so their
actual locations shouldn't matter if there's no IREL relocs (i.e. they
refer to the same location.)
However, under some circumstances, glibc reads `__rela_iplt_start`
before applying base relocations and `__rela_iplt_end` after applying
them due to compiler instruction reordering.
Therefore, we need to make them absolute symbols that don't require
base relocations.
Fixes https://github.com/rui314/mold/issues/1407
If both --separate-debug-info and --compress-debug-info are given, we want to
compress debug sections in background so that the delay caused by compression
is not visible to users.
I didn't know if Azure ARM64 servers support ARM32 instructions, but
apparently they do because running ARM32 code is much faster on ARM64
GitHub Actions runners.
Previously, 4 bytes were allocated for each relocation in a text
section to store the number of bytes removed before each relocation.
Now, only the locations where bytes are removed are stored.
This commit improves performance by ~4% when linking clang-19 for RISC-V.
Previously, we allocate 4 byte struct for each relocation to store a
reference to a thunk in case the relocation needs to branch to a thunk.
Most references are null because most relocations don't need thunks.
This commit changes how we manage references to range extension thunks.
Now, thunk addresses are stored to symbols rather than to relocations.
Linking clang-19 for ARM64 is now ~7% faster than before.
ELF Application Binary Interface s390x Supplement Version 1.6.1, p.58 says
that "The doublewords at _GLOBAL_OFFSET_TABLE_[1...2] are reserved for
system use."
We had previously reserved GOT[0] and GOT[1], but we did not reserve GOT[2].
It's most likely that GOT[2] is not actually in use, as it appears that
mold-linked programs work without it. But we need to adhere to the psABI.
On RHEL 10, GCC is configured with the `--with-arch_64=x86-64-v3`
option, which makes it implicitly use the `-march=x86-64-v3`
command-line flag by default.
This breaks the `arch-x86_64-ifunc-alias` test: since the `x86-64-v3`
microarchitecture level includes SSSE3 and AVX2, GCC appears to
interpret the `default` and `ssse3,avx2` targets as the same thing, and
omits generating a resolver function.
Fix the test by forcing GCC's target architecture to the `x86-64`
baseline.
Switch from Docker to Podman for container builds and remove unused
buildx steps. Also this commit updates GitHub Actions workflows to use
redhat-actions/podman-login and docker/setup-qemu-action@v3, removing
references to Docker Buildx.
Fixes: ec1d7120aa ("Use docker only on GitHub Actions")
Signed-off-by: Ruowen Qin <[email protected]>
Debian 11 ships GCC 10 and CMake 3.18, which happen to be precisely the
minimum versions required for building mold from source.
Add a CI pipeline to prevent future commits from inadvertently breaking
compatibility with older toolchains.
Linux upstream has reverted commit 2a010c412853 ("fs: don't block
i_writecount during exec") and the change is merged in the 6.13 cycle.
This means the executable overwriting behavior can be restored on Linux
kernel version 6.13 or later.
Note that there is probably not much we can do for 6.11 / 6.12 kernels.
Although the reversion is backported, many distros do not use upstream
sub-version numbers which means we cannot determine whether the kernel
has this reversion based on its version number.
Signed-off-by: Jinghao Jia <[email protected]>
Even though 16 KiB is the common page size, the Linux kernel for
LoongArch supports 64 KiB pages. Currently, mold-generated binaries
run only on systems with <=16 KiB page.
Note that unlike GNU ld, increasing the maximum page size doesn't
increase the on-disk binary size of generated binaries.
This is an alternative to https://github.com/rui314/mold/pull/1383
The upcoming version of GNU binutils's ld now treats arguments for
`--package-metadata` as percent-encoded strings and decodes them before
writing to .note.package.
Additionally, GNU ld will support `%[<charname>]`, where `<charname>` can be
`comma`, `lbrace`, `rbrace`, `quot`, or `space`. This feature could improve
readability when manually writing command-line arguments.
That said, I don't believe we need to support these metacharacters, for the
following reasons:
1. This option is specifically designed for distribution package maintainers
to embed metadata into binaries, and is not intended for general use.
2. Command-line arguments for this option are always generated by build
scripts, not written manually.
If we need these character names, we can add that later.
Furthermore, GNU ld does not appear to report errors for orphan '%'
characters that are not followed by valid hex values; instead, it simply
passes through such characters. I find this behavior fragile. We will report
an error for invalid '%' characters.
With these considerations in mind, I'll just rename the existing
`--encoded-package-metadata` option to `--package-metadata`.
In case ctest fails in the CI, no error logs are printed. This makes
debugging harder.
So run ctest with `--output-on-failure` to ease debugging in case of
failures.
Combining mimalloc and AddressSanitizer triggers a segmentation fault in
many tests.
Combining mimalloc and ThreadSanitizer leads to duplicate symbols for
the `new` and `delete`
operators, causing the build to fail.
Example message from a failed GitHub Actions build:
```
mold: error: duplicate symbol:
third-party/mimalloc/libmimalloc-debug.a(alloc.c.o):
/usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.tsan_cxx-x86_64.a(tsan_new_delete.cpp.o):
operator delete(void*, std::align_val_t)
```
This allows the compiler to make sure that the implementation of
`set_mimalloc_options()` matches its declaration.
More importantly, it indirectly pulls in `config.h`, where the macros
`MOLD_USE_SYSTEM_MIMALLOC` and `MOLD_USE_MIMALLOC` are conditionally
defined. Without these, the build configuration is ignored.
We don't need to create a copy of license files for third-party software
as we keep an original license file for each subdirectory. It is probably
a bit risky to create copies of license files because it is easy to become
out-of-sync. So I'll just remove this file.
If running in big-endian mode, SH4 processors expects both data and
instructions to be encoded in big-endian. Therefore, nop need to be
reversed for sh4be.
This commit adds experimental big-endian SH4 support. Since we already
support little-endian SH4, it wasn't too difficult to support its big-
endian variant.
Some tests related to C++ exceptions fail due to mysterious errors.
I don't have time to debug it, so I'll disable them for now.
Closes https://github.com/rui314/mold/issues/1273
Including mimalloc-new-delete.h overrides new and delete operators
with mimalloc. If a user choose not to use mimalloc, we shouldn't
include that file.
https://github.com/rui314/mold/issues/1355
RHEL 8.7's version of dwarfdump (2018-01-29) is too old to recognize the
SHF_COMPRESSED flag in debug info, so the compress-debug-sections test
fails despite the debug sections being compressed as intended.
To fix this, we will use readelf to dump information about the
executable. The COMPRESSED flag can be verified using this.
It looks like only the presence of the entry is important, but by
convention, GNU ld and lld set 0 to a DT_AARCH64_VARIANT_PCS dynamic
entry, so we should follow suit.
Keeping DEC Alpha support is difficult because there are no Alpha
machines available for toolchain developers, and even if there were,
no modern Linux distros still support Alpha. For example, Debian
dropped Alpha support in Debian 6.0. As a result, I cannot test my
linker with real-world programs.
In fact, I believe mold has never been able to link real-world large
programs. Object files for Alpha are compiled with the small code
model by default, so it is not easy to support it in the linker.
There's no means to verify that my implementation is correct for Alpha
because Alpha's psABI has never been published by DEC. The most
"reliable" source of correctness is GNU ld's source code and comments.
Last but not least, I believe there are literally zero mold/Alpha
users. Unlike some other retro computers like the m68k, Alpha doesn't
seem to get much love from the community, perhaps because the
availability of Alpha machines was limited in the first place, even in
the 90s.
Therefore, I'll remove Alpha support now. If there's someone who wants
to keep it, we can resurrect and re-test it.
This change should make the mold binary size a little smaller.
This commit reverts 0612ea41f1 and
instead solves the original issue with a different approach.
Now, if a hidden symbol was resolved to a DSO symbol, we'll redo
symbol resolution so that the symbol won't be resolved to the same one
next time.
This should fix our build-all buildbot.
Assume both `foo.a` and `bar.so` define the same symbol `baz`.
If `baz`'s symbol visibility is hidden, it needs to be resolved within
the output file, i.e., from `foo.a`. However, previously, such symbol
was resolved to the one in `bar.so`.
To fix the problem, we'll lower the symbol priority for DSOs.
Fixes https://github.com/rui314/mold/issues/1342
Word-size absolute relocations, such as R_X86_64_64, are special from
the linker's point of view because only such symbols can be promoted
to dynamic symbols. This patch handles them separately.
* Fix infinite loop in `split_by_comma_or_colon()`.
The mere presence of a `,` or a `:` in the input string would make the
function enter an infinite loop, effectively freezing the program until
all available memory was exhausted. The string position in the loop was
not incremented past the `,` or the `;` character that had just been
read.
The function is currently used only in the `--exclude-libs` option. The
freeze is reproducible with:
```bash
$ mold --exclude-libs foo,bar
```
The longer the string after the first `,` or `:` is, the faster memory
exhaustion is reached.
* Skip empty parts in `split_by_comma_or_colon()`.
The LoongArch extreme code model uses the following instructions to
materialize a 64-bit value in a regiter.
pcalau12i $t1, %pc_hi20(g2)
addi.d $t0, $zero, %pc_lo12(g2)
lu32i.d $t0, %pc64_lo20(g2)
lu52i.d $t0, $t0, %pc64_hi12(g2)
The previous relocation formula expected that `page(pc)` would be
consistent for all the instructions, but this assumption is incorrect
if the instruction sequence crosses a 4 KiB boundary.
Now, the LoongArch psABI requires that the machine instructions must be
consecutive, and the relocations for lu32.d and lu52i.d uses
`page(pc - 8)` and `page(pc - 16)` instead of `page(pc)`.
This psABI change gave me an impression that the LoongArch's extreme code
model was poorly designed and inadequately tested. If these instructions
must be consecutive, only a single relocation referring to the beginning
of the instructions would suffice, which relocates all the following four
instructions at once.
Maybe we do not need to support the relocations for the extreme code model
because the code model was buggy at the spec level, which suggests that no
one is using them. But I'm not sure if it is safe to remove them, so let's
just follow the psABI change.
Old zlib does not support crc32_z. The difference between the two is
that crc32() takes an input size as an unsigned int while crc32_z()
takes as size_t. For our use case, size is guranteed to be short, so
crc32() is safe to use.
It looks like emitting PCADDU18I + JIRL instead of just BL for function
calls is a way to go, so we don't want to bother to maintain range
extension thunks for the psABI.
Since no other linkers support range extension thunks for LoongArch other
than us, removing the support should be OK.
Previously, functions that are referred to only by section symbols were
not considered address-taken, and we always rewrote endbr64s with NOPs for
such functions. That caused a runtime fault when such function was called
indirectly.
Now, endbr64s for such symbols are retained.
I tried to build clang-19 as a release build with and without `-z
rewrite-endbr` and counted the number of endbr64 instructions in each
binary. Here is the result:
Before: 110,615
After: 91,799
So the feature reduces the number of gadgets by 17%. Both binaries worked
fine with Intel SDE CPU emulator with `-cet 1 -cet_raise 1`, so I think
it's finally working as expected.
I also tried to build mold itself with and without the feature. Here is
the result:
Before: 27,430
After: 17,725
This is a 35% reduction. I confirmed that mold built with `-z rewrite-endbr`
can self-host under Intel SDE.
The rewrite_endbr pass is extremely fast. It took only 7 milliseconds
for a ~210 MiB clang-19 binary on my Threadripper 7990X machine. We
may want to consider enabling it by default at some point.
glibc's `__register_frame_info_bases` expect that it can read a terminator
if a .eh_frame is empty. Removing the section completely caused a crash bug
on Alpine Linux with musl as libc.
Specifically, mold-generated binaries crash if all the following
condition were met:
1. musl is used as a libc
2. .eh_frame is empty
3. libgcc_s.so is linked
Prior to 06b592683c, condition (3) was not
usually satisfied, so we didn't notice this issue. Now, the shared library
is linked against almost all output files because the file is referenced by
`__deregister_frame_info`.
https://github.com/bminor/glibc/blob/910aae6e5a2196938fc30fa54dd1e96f16774ce7/sysdeps/generic/unwind-dw2-fde.c#L105-L107
Fixes https://github.com/rui314/mold/issues/1309
Specifying the compiler flag `-Wl,--package-metadata=<JSON>` might not
work, because the shells might eat the quotation marks and the compiler
splits the JSON at the commas.
Ubuntu tried to using a specs file to set `--package-metadata` but that
turned out to be too fragile. autopkgtests might use the compiler flags
but the needed environment variables are not set in the test
environment. Debugging a crash of an application build with the -spec
parameter lacks the environment variables. People like to iteratively
continue building the software in the build directory while hacking on
the package and then have no environment variable set.
So introduce a `--encoded-package-metadata` linker flag that takes a
percent-encoded JSON. Percent-encoding is used because it is a
standard and simple to implement.
Bug-Ubutru: https://bugs.launchpad.net/bugs/2071468
Signed-off-by: Benjamin Drung <[email protected]>
Previously, we find a string in a mergeable string section by offset
after adding a relocation addend to a symbol value. Now, we find a
string just by a symbol value and then add a relocation addend later.
Assume a mergeable section containing two strings as an example.
foo\0bar\0
^ ^
.L1 .L2
Previously, a relocation referring to .L2 - 1 referred to the end of
.L1. Now, it refers to one byte before .L2.
Section symbol is an exception to this rule, because it seems it's
what the compiler is assuming. The rules seem arbitrary, but it looks
like we need them for the compatibility with existing object files.
Fixes https://github.com/rui314/mold/issues/1300
This option is to separate debug info to a different file. The debug
info file's filename is stored to the main output file's .gnu_debuglink
section. gdb can read the section contents and followg the link to
find debug info in another file.
Fixes https://github.com/rui314/mold/issues/1294
It is not clearly defined when undefined weak symbols are resolved.
It looks like there are two possible approaches:
1. Promote all weak undefined symbols to dynamic ones so that they'll
have another chance to be resolved at load-time, or
2. Promote weak undefined symbols to dynamic ones only when there are
definitions in other DSOs at link-time.
(1) provides the maximum flexibility. For example, consider a main program
that has a weak undefined symbol `foo` and there's no DSO that defines it
at link-time. In (1), `foo` gets promoted to a dynamic symbol, so that one
of its depending DSO is upgraded to define `foo`, the main executable's
`foo` is resolved to that symbol at load-time. On the other hand, in (2),
`foo` would have already been converted to an absolute symbol at address
zero at link-time, so you need to rebuild the main executable to use the
new definition of `foo` in the shared library.
However, (1) is not compatible with copy relocations. This is because we
need to know the size of the symbol when creating a copy relocation, but
that information is not available unless we have a definition. It's also
not compatible with canonical PLTs because canonical PLTs have non-zero
addresses and therefore weak undefined symbols would always be resolved to
non-zero addresses.
As a workaround, GNU ld promotes weak undefs to dynamic symbols only when
they don't need copy relocations or canonical PLTs. In other words, weak
undef's behavior is different between -fPIC and -fno-PIC. In the former
case, they become dynamic symbols, and vice versa.
I don't think that workaround is a good one. So, mold took the second
approach.
There is, however, another thing to consider. What if we can find a
defined symbol in a DSO that is specified as `-as-needed`? Previously,
mold did not mark the library as "needed" and converted the weak undef
into an absolute symbol.
However, libstdc++ assumes that if weak undef symbol
`__pthread_key_create` is not resolved, it assumes that multi-threading is
not used in the executable, which resulted in a mis-detection with mold.
Therefore, this patch changes the mold's behavior so that it makes weak
undefs to keep DSOs "needed".
Fixes https://github.com/rui314/mold/issues/1286
Use $CXX instead of $CC to detect whether C++ programs could be built
statically. Add function test_cxxflags for convenience.
Fix test error in an environment where C++ programs cannot be built
statically, with error like:
mold: fatal: library not found: c++
Signed-off-by: Yao Zi <[email protected]>
The interaction between these flags are unnecessarily complicated,
but it looks like `--undefined=ignore-in-object-files` needs to
share the same internal flag as `-z defs` so that they override
each other.
Fixes https://github.com/rui314/mold/issues/1270
Teaching the compiler that `buflen` cannot be negative and also doesn't
take a large value allows it to optimize code a little bit more than
it currently does.
I couldn't find any report that the reduced round version is broken
despite being used at a lot of places (e.g. Rust's hash table uses
SipHash-1-3). So I assume it's secure.
`which` isn't in POSIX and several Linux distributions are trying to
remove it from their base system, see e.g. https://lwn.net/Articles/874049/.
Just use `command -v` which is POSIX.
Signed-off-by: Sam James <[email protected]>
We use Aho-Corasick string search algorithm for version script.
If all symbols in a version script do not start with '*', we can
terminate the algorithm early.
I believe some version of objcopy corrupts an object file when
renaming a section. In this change, I use sed instead of objcopy
as a workaround.
Fixes https://github.com/rui314/mold/issues/1244
It looks like Clang recognize `#pragma GCC unroll`. Instead of adding
more C preprocessor directives here, I'd like to rely on the default
compiler optimizations.
If a synthesized symbol overrides an imported symbol, the symbol's
`is_imported` flag must be reset.
Technically, `compute_import_export` is called too early, and this
function call should be moved after `create_synthetic_sections`.
However, fixing the issue that way was very hard because until we
create output sections, we don't know what __start_/__stop_ symbols
need to be synthesized, but in order to create output sections,
we need to compute import/export bits.
So, in this commit, we simply reset `is_imported` flag for
synthesized symbols.
https://github.com/rui314/mold/pull/1236
Previously, .so and .a were of the same priority and therefore symbols
in those were resolved based on their positions in the command line;
whichever file appears first in the command line took precedence.
Here is the problem we are trying to solve with this change: KiCad
passes unnecessary .a files to the linker along with some .so files.
Some symbols are defined both by .a and by .so.
If a symbol is resolved from .a, the linker pulls out the file from
the archive, but because the .a file does not really provide a
complete set of object files, it ended with "undefined symbol" error.
If a symbol is resolved from .so, everything is fine.
This is arguably a bug in KiCad, or at least depending on the order of
files in the command line is very fragile. But maybe this change could
fix the issue without too much side effects. So let's see how it goes.
Fixes https://github.com/rui314/mold/discussions/1234
RISC-V psABI requires that symbol to be exported from an executable
if there's a GP-relative reference. For simplicity, we always export
it from executable as long as it has a .dynamic section.
https://github.com/rui314/mold/issues/1222
Runtime doesn't care where mergeable sections are, but it's reported
that Valgrind does. So, grouping mergeable sections just doesn't seem
to make sense.
https://github.com/rui314/mold/pull/1223
libtool mistakes mold 2.4.1 as GNU ld 2.4 and wrongly concludes that our
linker does not support anonymous versioning and suppresses some uses of
version scripts. That has been causing subtle compatibility issues with
programs that use libtool to create their .so files.
Here is the code that mistakes our linker as GNU ld:
https://git.savannah.gnu.org/cgit/libtool.git/tree/m4/libtool.m4?h=v2.4.7#n5066
As a workaround, I decided to bump our linker version so that the version
number is sufficiently large. This is admittedly ugly but I think it's the
simplest solution for the problem.
Our ConcurrentMap uses linear probing to find unused hash table entry.
It gives up if 128 consective slots are occupied, and the whole process
dies with the "ConcurrentMap is full" error message. So our hash
function's quality must be high.
For .gdb_index, we used to use gdb_index() to compute keys for the
ConcurrentMap. It turned out that the function's quality is poor,
generating very similar output for short strings.
This commit changes the hash function to xxhash.
Fixes https://issues.chromium.org/issues/40276991#comment5
Previously, mold marked an imported symbol as a strong one if the
symbol came from a DSO and was exported as a strong symbol by the DSO.
This logic resulted in a miscomputation of the weakness bit, causing a
compatibility issue with other linkers.
Now, an imported symbol is marked as strong only when there's at least
one strong reference to it. In other words, if all references to an
imported symbol are weak, the symbol will be imported as a weak one.
Fixes https://github.com/llvm/llvm-project/issues/83080
Prior to this change, links on Windows were failing with:
mold: fatal: <input file>: CreateFileMapping failed: 5
"5" is the Windows error code for ERROR_ACCESS_DENIED.
Use native Windows file API in MappedFile. Create read only mapping with
FILE_MAP_COPY view because the file mapping contents may be modified in
process but should not be written back.
On Windows, it is important that files are opened in "binary" mode to
avoid automatic "text" mode conversions which can result in "corrupted"
input/output.
Previously, mold didn't merge sections with the same name but different
section flags into a single section. This behavior is different from other
linkers and caused a compatibility issue.
Now, mold merges input sections by their names and types.
Fixes https://github.com/rui314/mold/issues/1196
std::string is a non-trivial type, this change ensures its
copy-ctor and dtor don't get called.
Found by Clazy (range-loop-reference)
Signed-off-by: Christoph Grüninger <[email protected]>
illumos is a descendent of Solaris/OpenSolaris, and treats
symbols with shndx=SHN_ABS and st_value=0 specially, such that
they should be marked imported. This change adds a special
case for this behavior, wrapped up in a function that #ifdef's
out on other platforms. With this patch, executables linked
with mold on illumos run; without it, they do not.
The original behavior was spotted by Luqman Aden (@luqmana).
This also patches a small bogon in the third-party `mimalloc`
code. I've sent that same fix upstream, but so far the
maintainer has not resonded, so replicate here.
Fixes#1183
Signed-off-by: Dan Cross <[email protected]>
When removing symbols, the indices in .llvm_addrsig also needs to be
updated, but binutils is not aware of this and leaves the table in a
corrupt state.
Use the same test as what LLD uses to reject potentially corrupted addrsig
sections.
Unfortunately, testing this seems tricky. While producing a file with OOB
addrsig index is not hard, it's hard to consistently trigger a segfault
with that access.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Previously, if two or more VERSION clauses match to the same symbol,
the first one took precedence. This was incompatible with GNU ld, which
gives the last one the highesth priority.
This change inverted the priority so that the last one will take
precedence other the others.
Fixes https://github.com/rui314/mold/issues/1158
Using ftruncate() creates holes inside files, that on first (write) page
fault gets allocated an actual disk block. There are several
disadvantages with this:
- Allocating block one-by-one is slower than doing it all at once.
- As we write to the output in parallel the write order is not exactly
sequential. The allocated blocks can also be out-of-order and make the
on-disk representation fragmented.
Instead, use posix_fallocate() which allocates a contiguous on-disk block
(if possible). It's arguably a better fit for linker output, since holes
are unlikely to be present after writing all the output (any contiguous
zeroes should just go into .bss).
I benchmarked this by linking mold with debug symbols. On an 8-core
desktop, it is 5% (1.0s -> 0.95s) faster, but on a 64-core workstation it
is actually 40% slower (0.37s -> 0.52s). The total amount of System time
is reduced (6.7s -> 5.2s), however. I suspect that the reduced page fault
work ends up causing more lock contention.
Despite potential regression on high-core systems I still think this
change is justified because of the better on-disk layout.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Previously, we always tried to demangle a symbol name as a Rust
symbol first and then attempted to demangle it as a C++ symbol.
This resulted in an incorrect demangled result, as the Rust legacy
mangling scheme is not distinguishable from C++.
This patch fix the issue by adding the "is_rust_obj" flag to the
ObjectFile. We try to demangle a symbol as a Rust one before as a C++
only if the flag is true.
Fixes https://github.com/rui314/mold/issues/1159
A `.eh_frame` section contains data for exception handling. Usually,
an object file contains only one `.eh_frame` section, which explains
how to handle exceptions for all text sections in the same object file.
However, it appears that, in rare cases, we need to handle object
files containing multiple `.eh_frame` sections. An example of this is
the `/usr/lib/clang/17/lib/x86_64-redhat-linux-gnu/clang_rt.crtbegin.o`
file, which is provided by the `compiler-rt` package of Fedora 39.
Specifically, I'm using the `quay.io/fedora/fedora:39` Docker image.
The file contains two `.eh_frame` sections.
One `.eh_frame` in the file is of type `STT_X86_64_UNWIND` and the
other is of `STT_PROGBITS`. It's possible that the file was created
with `ld -r`, and the linker failed to merge the two incoming
`.eh_frame` sections into one output section due to the difference in
section types.
We did not expect such inputs and consequently produced corrupted
output files.
This commit improves our linker so that mold can handle multiple
`.eh_frame` sections in a single object file.
Fixes https://github.com/rui314/mold/issues/1157
Previously, symbol names or wildcard patterns in version script files
were processed strictly from top to bottom. When a single symbol
matches two or more patterns, the first one was given precedence.
However, GNU ld appears to treat exact matches differently, assigning
them higher precedence over wildcard patterns. This discrepancy led to
programs linked with Qt 6.6.1 using mold failing to launch.
Fixes https://github.com/rui314/mold/issues/1158
It had been assumed that the TLSDESC code seqeunce looks like this:
lea 0(%rip), %rax
R_X86_64_GOTPC32_TLSDESC foo
call *(%rax)
R_X86_64_TLSDESC_CALL foo
However, LLVM seems to also emit something like this:
lea 0(%rip), %reg
R_X86_64_GOTPC32_TLSDESC foo
...
mov %reg, %rax
...
call *(%rax)
R_X86_64_TLSDESC_CALL foo
That means when we rewrite the LEA instruction to relax the code
sequence, we need to handle destination registers other than %rax.
The wrong assumption caused a program crash as reported as
https://bugs.gentoo.org/914849.
This is a new experimental flag to make room at the end of PHDR
so that post-processing tools can add more entries as needed.
Fixes https://github.com/rui314/mold/issues/1148
Debian 8 was released in April 2015, which is old enough for
binary distribution. Previously, we used to use Debian 10 (Bullseye),
which was released in July 2019.
--dynamic-list, --export-dynamic-symbol and --export-dynamic-symbol-list
have different semantics for executables and DSOs. If the output is an
executable, they specify a list of symbols that are to be exported.
If the output is a shared object, they specify the list of symbols that
are to be interposable.
mold havne't implemented the latter semantics. This commit fixes that
issue.
Fixes https://github.com/rui314/mold/issues/1071
.init_array/.fini_array haven't completely replace .ctors/.dtors, so
we need to convert .ctors/.dtors to .init_array/.fini_array.
Fixes https://github.com/rui314/mold/issues/1113
Previously, we didn't emit a dynamic relocation for a statically-
initialized function pointer pointing to an ifunc on POWER10, e.g.
int fn() __attribute__((ifunc("resolve_fn")));
void *ptr = fn;
Fixes https://github.com/rui314/mold/issues/1142
For --relocatable output, we need to pass through these attributes
for correctness. With this change, we'll have more .rodata sections
such as .rodata.str1.1, rodata.str1.4, etc. in the final executable,
but that should be fine because the runtime doesn't care about
section tables at all. The table isn't even loaded to memory at runtime.
IFUNC symbols are resolved at process startup by executing the function
that the symbol points to. This is used to select the "best" function at
runtime; for instance, the runtime may choose a faster version of memcpy
that uses SIMD instructions if they are available on the current system.
Thus, an IFUNC symbol has two addresses: the initial address (or the
resolver's address) and the resolved address, which is the return value of
the resolver.
In position-independent executables (PIEs), function pointers are loaded
from the GOT indirectly, and symbols are not directly referenced. In such
executables, the initial value of the GOT slot for an IFUNC symbol
contains the resolver address, and this is overwritten at runtime to the
resolved address upon process startup. When user code takes a pointer to
an IFUNC, it always reads the resolved address from GOT.
In contrast, position-dependent executables (PDEs) may have instructions
that directly refer to an IFUNC symbol, such as movabs on x86-64. The GOT
entry for an IFUNC holds the resolved address, so any direct reference
must also produce the resolved address to maintain pointer equality.
(C/C++ standards require that two pointers must be equal if and only if
they are taken for the same symbol.)
Previously, we emitted text relocations to modify instruction operands.
However, text relocations are undesirable and not always reliable. For
example, on ARM64, multiple instructions are used to materialize a
symbol's address, and it's not feasible to issue a dynamic relocation to
alter those instructions since the dynamic loader generally can only
modify 32-bit or 64-bit words.
In this commit, I have adopted a different strategy. An IFUNC symbol now
occupies two consecutive GOT slots in a PDE. The first slot holds the
symbol's PLT address, and the second slot holds the resolved address. The
PLT address is consistently used as the symbol's address throughout the
process, while the second slot is used only by the PLT entry to jump to
the resolved address.
This method ensures pointer equality without the need to emit text
relocations for IFUNC symbols in PDEs.
It seems R_LARCH_PCALA_LO12 is sometimes used against JIRL even though
such use case is incorrect as per the psABI.
At least, `/usr/lib/loongarch64-linux-gnu/libc_nonshared.a(atexit.oS)`
on `cfarm400.cfarm.net` contains such relocation.
We need this patch to self-host mold on that cfarm machine.
Specifically, .llvm.call-graph-profile is always REL even on x86-64.
We can safely ignore the relocation table because it contains only
R_NONE relocations.
Fixes https://github.com/rui314/mold/issues/653
The test expects the output format given by `llvm-readelf`:
```
5: 0000000000010698 0 FUNC GLOBAL DEFAULT [VARIANT_PCS] 13 foo
```
However, in GNU's `readelf` implementation the symbol name comes first:
```
5: 0000000000010698 0 FUNC GLOBAL DEFAULT 13 foo [VARIANT_PCS]
```
Replace the regular expression with two `grep` calls piped together to
support both output formats.
Signed-off-by: Christoph Erhardt <[email protected]>
`test_cflags` uses $CC to test if a given flag is supported, so
we should use $CC instead of $GCC for the rest of the file.
Fixes https://github.com/rui314/mold/issues/1128
This reverts commit 967006479f because
it made the linker to crash. TLSDESC doesn't seem to work even with
GNU ld on ARM32 musl systems, so I'll disable the test.
Fixes https://github.com/rui314/mold/issues/1116
Despite the extern template items Clang 14 seems to insist on instantiating
elf_main(), with the hilarious consequence of the function being duplicated
quadratically to the number of archs.
Move this to a different CU so that the compiler cannot possibly
instantiate it more than once. This significantly speeds up compilation
related to main.cc.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Despite the extern template items Clang 14 seems to insist on instantiating
elf_main(), with the hilarious consequence of the function being duplicated
quadratically to the number of archs.
Move this to a different CU so that the compiler cannot possibly
instantiate it more than once. This significantly speeds up compilation
related to main.cc.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Compiler warnings have the potential to uncover real bugs before they
make it into a commit. For example, the accidental fallthrough
introduced by commit 713d81947a and fixed
by commit 49f38828d1 could have been
detected by GCC.
Remove `-Wunused-variable` since it is implied by `-Wall`.
Add `-Wno-unused-parameter` to suppress warning spam regarding unused
parameters.
Signed-off-by: Christoph Erhardt <[email protected]>
This fixes the following compiler warning:
> enumerated and non-enumerated type in conditional expression [-Wextra]
Signed-off-by: Christoph Erhardt <[email protected]>
endbr64 is a relatively new x86 instruction to mark a location to which an
indirect jump instruction can transfer control. If the control-flow
integrity is turned on (i.e., endbr64 is effective), an indirect jump can
only target an endbr64 or raise a runtime exception. This mechanism makes
some types of control hijacking attacks, such as ROP or JOP, very hard
because an attacker can no longer jump to an arbitrary location.
If `-fcf-protection` is given, GCC emits an endbr64 at the beginning of
each global function conservatively because the address of that function
might be used by other translation units.
In reality, most function addresses aren't taken, so the conservative
assumption ends up emitting lots of unnecessary endbr64 instructions.
This not only bloats the code but also weakens security because an
attacker would have more locations to attack.
The new linker option, `-z rewrite-endbr`, is designed to mitigate this
situation. The linker is in a position to conduct whole-program analysis
on the input files and determine functions whose addresses have never been
taken. If `-z rewrite-endbr` is given, mold performs that analysis and
rewrites the leading endbr64 with a nop if the function's address wasn't
taken.
For evaluation, I built the mold executable itself both with and without
this option. Before using the `-z rewrite-endbr` option, the mold
executable contained 25,994 endbr64 instructions. With the option, the
number becomes 18,035, a 30% reduction.
The new option is exprerimental; we may remove it in a future versions of
the mold linker.
For example, when the leader's instruction is removed and a follower is right
next to it, the follower's relocation can point to the follower's address
itself. In this case the leader's relocation may lie before or after the
follower, so just search for both.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
I made a fair amount of effort to try to support MIPS, but it turned out
that it is much harder than expected. I punted it instead of making
further efforts.
The problem is the MIPS ABI is hostile to the linker in the modern
environment. MIPS object files are still compiled for the small code
model in which GOT entries and data in the small data area are expected
to be accessible with a single machine instruction with a 16-bit
displacement. In other words, .got/.sdata/.sbss are expected to be
smaller than 64 KiB. This might have been a reasonable assumption in the
90s, but it's not suitable for modern applications that can be 1000x
larger than the binaries in the 90s.
MIPS requires the linker to implement tons of workarounds for its legacy
ABI assumptions. Our incomplete MIPS support can build binaries that
pass all our unit tests. However, it needed more effort to support
real-world programs that are larger than our test cases.
At this point, I don't think it is productive to implement workarounds
for the old ABI that is stuck in the 90s. It is honestly annoying to
think about workarounds for the code that is intentionally compiled to
be hostile to the linker. The situation is unfortunate, but if the
Open Source community is still serious about MIPS, they should improve
the ABI and the compiler instead of asking us to implement the legacy
ABI.
Closes https://github.com/rui314/mold/issues/1040
Quote from Arm Cortex-A75 Software Optimization Guide
(PJDOC-466751330-205) p.59:
Consider aligning subroutine entry points and branch targets to
16-byte boundaries, within the bounds of the code-density requirements
of the program. This ensures that the subsequent fetch can maximize
bandwidth following the taken branch by bringing in all useful
instructions.
`CMakeLists.txt` declares a minimum CMake dependency on 3.14.
```
cmake_minimum_required(VERSION 3.13)
```
But the `--install` switch is only available since CMake 3.15, see
<https://stackoverflow.com/a/56457739>.
This commit updates the installation instructions in `README.md` so that
they work with CMake 3.14.
If an as-needed DSO is referenced only by other DSOs and not by
the output file we are creating, we won't mark it as "needed".
Fixes https://github.com/rui314/mold/issues/1036
Although we do not support TLSDESC on any target on which
ctx.tls_begin != ctx.dtp_addr, my understanding is that the origin
of the TLSDESC base relocation is tls_begin.
Object files have to be compiled with `-fPIE` or `-fPIC` for a
position-independent executable, and such object files shouldn't
contain relocations that directly refers to data objects without
GOT.
ARM64's crt1.o may refer to `main` with the following four instructions
instead of with the usual `R_AARCH64_ABS64` relocation:
18: d2e00000 movz x0, #0x0, lsl #48
18: R_AARCH64_MOVW_UABS_G3 main
1c: f2c00000 movk x0, #0x0, lsl #32
1c: R_AARCH64_MOVW_UABS_G2_NC main
20: f2a00000 movk x0, #0x0, lsl #16
20: R_AARCH64_MOVW_UABS_G1_NC main
24: f2800000 movk x0, #0x0
24: R_AARCH64_MOVW_UABS_G0_NC main
28: d2e00003 movz x3, #0x0, lsl #48
So we need to create a PLT if a symbol is referred to by this way.
Fixes https://github.com/rui314/mold/issues/1074
Although incomplete, this implementation is capable of passing the
following tests:
TRIPLE=mips64el-linux-gnuabi64 bash -x ../test/elf/hello-dynamic.sh
TRIPLE=mips64-linux-gnuabi64 bash -x ../test/elf/hello-dynamic.sh
Undefined versioned symbols are handled specially; they are resolved
at the very end of name resolution stage. When they are resolved, we
need to make sure that they are marked as imported symbols.
Fixes https://github.com/rui314/mold/issues/1057
Convenient if you have configured to use mold for a particular project but want
to test a new local build.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
open() can fail with transient reasons, such as running out of file descriptor
limit. Previously, this was treated the same as file not being found, which
caused confusing error messages [1].
With this change anything but ENOENT will be treated as a fatal error. This will
also include e.g. permission errors, but if there's something within the search
path that is not accessible, it's likely better to tell the user instead of
silently skipping it.
[1]: https://github.com/rui314/mold/issues/1026
Signed-off-by: Tatsuyuki Ishi <[email protected]>
STT_GNU_IFUNC is significant only for exported symbols. Setting
non-exported symbol type to STT_GNU_IFUNC actually causes an issue
if the symbol is used as a canonical PLT because the runtime would
try to run the canonical PLT as an IFUNC.
Fixes https://github.com/rui314/mold/issues/1028
I made this change by asking GPT-4-based ChatGPT to improve this
document with an additional note that this document was written by a
non-native English speaker and may have wrong or unnatural phrases.
Fixes the following error:
/elf/thunks.cc:81:32: warning: enum constant in boolean context [-Wint-in-bool-context]
return ty == R_PPC64_REL24 || R_PPC64_REL24_NOTOC;
Signed-off-by: Martin Liska <[email protected]>
Handle .eh_frame in a similar way as mergeable sections.
This seems to be required for GCC CRT compiled with LTO.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
When LTO is enabled for CRT it seems symbols like __EH_FRAME_BEGIN__.lto_priv.0
can be created. Treat it as the same as the non-LTO variant.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Previously, we clear liveness bits for all input files after compiling
IR object files to real object files so that we can do name resolution
completely from scratch. However, that can result in inconsistent name
resolution results before and after LTO.
Consider a weak defined symbol X in an object file. A weak undef doesn't
pull files out of an archive file. However, if symbol X is transformed
to an undefined symbol as a result of LTO, it could pull out a file from
an archive. That can have cascading effects and result in a link failure.
With this fix, all object files that are alive at the time when LTO kicks
in are guaranteed to be kept after LTO.
Fixes https://github.com/rui314/mold/issues/991
Previously, we failed to report an undefined error if we have two
undefined symbols, a strong and a weak one, for the same symbol.
It is an undefined symbol error if an undefined symbol is resolved
to a weak undefined symbol.
Fixes https://github.com/rui314/mold/issues/967
Ideally we wanted to use __builtin_assume() or [[assume]], but the former
is available only in clang and the latter is in C++23.
Fixes https://github.com/rui314/mold/issues/966
Previously, we created section symbols only for memory-allocated secitons.
That resulted in symbols with invalid section indices for the --relocatable
output.
https://github.com/rui314/mold/issues/957
Since the maximum size of RangeExtensionThunk is 100 KiB, it can't contain
too many entries, so the contents is addressable with a 16-bit value.
We don't create more than 32768 range extension thunks (unless the output
is hundreds gibibytes in size), so we can also use i16 to specify the
index of RangeExtensionThunk.
We used to have `--print-dependencies` and `--print-dependencies=full`
options. I think this distinction is necesssary. Now, `=full` is renamed
`--print-dependencies`.
These aren't bug, but I decided to repeat the same code instead of
adding [[fallthrough]] because the code duplication is very small.
Fixes https://github.com/rui314/mold/issues/952
We used to try to avoid using identifiers such as S, A or P more than
once because they were defined as macros expaneded to function calls.
Now they are just variables, so we can use them as many times as we
want without any performance penalty.
Unknown relocations will be reported by scan_relocations(). We don't
want to execute unreachable() statement in get_addend() because it
triggers undefined behavior.
https://github.com/rui314/mold/issues/947
It looks like R_ARM_PLT32 can be applied to non-BL/non-BLX instructions,
so we can't handle it as an alias for R_ARM_CALL.
https://github.com/rui314/mold/issues/934
This reverts commit cbd88ed3f3.
It was a nice abstraction but wasn't worth its complexity unless we
use it in various places. The original simple code is more in line
with other parts of mold.
Fixes https://github.com/rui314/mold/issues/946
We used to use C macros as a poor-man's way to inline Symbol::get_addr(),
but it looks like inlining that member function isn't an overall win.
This change removes `inline` from that member function and depends on the
compiler's optimizations to optimize function calls.
We were inconsistent as to when to clear existing values when applying
relocations. mold was broken from the point of view that bits must be
cleared before applying relcoations. But at the same time, we had code
to clear bits for some types of relocations.
This patch removes all bit clear code for the sake of simplicity.
According to the ARM psABI document, R_ARM_PLT32 is deprecated.
It looks like old compilers emit it for function calls.
Fixes https://github.com/rui314/mold/issues/934
Right now, mold linker does not work with -fno-use-linker-plugin
and thus use assembly for GCC FAT LTO objects. I noticed that
when running GCC test-suite with mold.
Fixes: 804b843983
Signed-off-by: Martin Liska <[email protected]>
Linking symbols with differing types will almost always lead to a crash; warn to
make it easier for the user to figure out the cause.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
This is a tricky case, but if you initialize a global variable with a
pointer to an ifunc function like this:
int bar() __attribute__((ifunc("resolve_bar")));
void *bar2 = bar;
then we need to emit two dynamic relocations, one for `bar()` and the other
for `bar2`. This results in calling `resolve_bar` twice at runtime, but
there's no other way to initialize the function and the variable correctly.
Fixes https://github.com/rui314/mold/issues/915
In the current version of Docker image, OpenSSL libraries are installed
to `/usr/local/lib64`. It is not present in the default dynamic linker
search path, and thus results in the following error when libcrypto is
not statically-linked:
mold: error while loading shared libraries: libcrypto.so.3: cannot
open shared object file: No such file or directory
Fix that by overriding `libdir`, as suggested in
https://github.com/openssl/openssl/blob/master/INSTALL.md#libdir. Also,
run `ldconfig` to update dynamic linker cache.
Signed-off-by: Ruoyu Zhong <[email protected]>
A symbol with `STT_SECTION` doesn't have name (i.e. its `st_name` is zero.)
If such symbol is used as a comdat group name, we need to use a section name
instead of a symbol name as a group name.
Similarly to x86_64, use int3 (0xcc) as padding at the end of plt
and pltgot entries in order to increase security.
Signed-off-by: Martin Liska <[email protected]>
Recent versions of Xcode started creating `_objc_msgSend$foo` symbols,
and we need to synthesize the symbols and place them into `__objc_stubs`
section.
Add an undefined symbol to keep a reference to the defsym target. This prevents
elimination by e.g. LTO or gc-sections. A test is added for the case of LTO
dead-code elimination.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
In preparation for introducing undefined symbols that references external object
files (and should not get the output section overridden).
Signed-off-by: Tatsuyuki Ishi <[email protected]>
SHF_LINK_ORDER bit doesn't matter in an executable, but it looks like
the `strip` command in binutils 2.35 or earlier on ARM32 complains about
`.ARM.exidx` if we do not preserve that bit.
Fixes https://github.com/rui314/mold/issues/889
We used to create copy relocations if we are creating a non-PIC executable
and there's a direct reference to imported data. That works for most
programs but not for GHC (Glasgow Haskell Compiler). GHC places function
machine code and its supplimental data next to each other in the .text
section. If we create a copy relocation for such function, the resulting
executable won't work because (1) copy-relocated data is not executable
and (2) it may separate the function code from its supplemental data.
In this patch, I made a change so that mold creates dynamic relocations
instead of copy relocations if possible.
TLS chunks alignments are special: in addition to having their virtual
addresses aligned, they also have to be aligned when the region of
tls_begin is copied to a new thread's storage area. In other words, their
offset against tls_begin also has to be aligned.
A good way to achieve this is to take the largest alignment requirement of
all TLS sections and make tls_begin also aligned to that.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
FAT LTO objects are objects that contain both native assembly
and LTO bytecode. Identify such files as GCC_LTO_OBJ.
Signed-off-by: Martin Liska <[email protected]>
Previously, we create output sections when input sections are instantiated.
That is not ideal because input sections may be removed, and if that's the
case, any work that has done for the removed section is wasted.
This is an initial patch to reimplement --relocatable. Unlike the previous
implementation, the new one aims to be on par with GNU ld in terms of the
behavior. Previously, we didn't for example merge input sections at all.
Now we do it in the new implementation.
The previous implementation was slow. We also want to fix it in the new one.
The new --relocation feature is not complete yet; there are many rough edges.
But I think this patch is good enough to be submitted.
This work is funded by Mercury (https://mercury.com).
As documented by CMake [^1], prior to CMake 3.13, entries set by
`option()` can override a non-cached old value. The `CMakeLists.txt` in
the bundled mimalloc specifies a minimum CMake version of 3.0, so that
old behaviour is expected. That means no matter whatever value of
`MI_BUILD_STATIC` we set in `mold_add_mimalloc` does not really take
effect (this can be seen by adding some debug messages). That didn't
trouble us because it's on by default anyway.
While the documented solution is to set `CMAKE_POLICY_DEFAULT_CMP0077`
to `NEW`, we could actually also work around this by making them cache
variables. This is fine and valid because their values are not meant to
be changed anyway. (And this workaround is already being used in
`mold_add_tbb`, so by doing this, we are also eliminating some
inconcistency.)
Fixes#874.
[^1]: See `cmake --help-policy CMP0077` or
https://cmake.org/cmake/help/latest/policy/CMP0077.html
Signed-off-by: Zhong Ruoyu <[email protected]>
The square-bracket operator of `std::span` has undefined behaviour for
out-of-bounds element accesses. Resorting to iterators yields defined
behaviour.
This issue (originally described in #298) had been fixed in the past by
commit 03ba8f9325, but it has since
resurfaced with commit 193c245781.
Signed-off-by: Christoph Erhardt <[email protected]>
This makes .gnu.note.property handling generic instead of hardcoded against the
GNU_PROPERTY_X86_FEATURE_1_AND value.
The behavior is based on specification inside GNU gold header comments;
unfortunately a formal document does not exist yet and the comments seem to be
the best reference that can be found.
The support for the actual properties in wild seems scarce; neither gcc nor
clang emits the correct property even when passed -march=x86-64-v3 and using
__builtin_popcnt, for example. Therefore, this does not include a test yet for
the new properties.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Includes value required to handle the "ABI needed" and "ABI used" tags, but
doesn't include the feature values yet (because we don't have a use for it).
Signed-off-by: Tatsuyuki Ishi <[email protected]>
binutils has learned to inject some "x86 ISA needed/used" properties.
Try to disable their emission with -mx86-used-note=no if supported, since
now that mold is gaining support for these properties, they will cause an
mismatch with the expectations.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
This removes the extra slash in the symlink
installation output. It works because both
NEW_ABS and NEW_DIR are absolute paths, and
therefore always begin with a slash.
Signed-off-by: Zhong Ruoyu <[email protected]>
`git pack-refs` (often called by `git gc`) may
move refs from `.git/refs` to
`.git/packed-refs`. If that is the case, a lookup
in `.git/refs` will fail.
Signed-off-by: Zhong Ruoyu <[email protected]>
Earlier entries in a version scripts always get the precedence. Don't let a
non-C++ symbol rule override a preceding C++ symbol rule.
MultiGlob is modified to return the necessary rule index information to the
caller. Since the rule index might not always be contiguous within a
MultiGlob instance (e.g. interleaving ordinary/C++ rules), the meaning of
MultiGlob's `val` is changed so that the minimum `val` itself is returned,
instead of the `val` with minimum index (the earliest added one).
MultiGlob is also used for the macOS backend, but since it does not care
about order yet the code is left as-is.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
We can create peculiar executables with --section-order, and some ELF
loaders don't like some of them. Since the flag is intended to be used
for embedded/kernel programming, we don't need to verify that the
executables work on regular Linux macines.
Fixes https://github.com/rui314/mold/issues/838
We are using OpenSSL only to compute SHA-256 hashes, so we are unaffeted
by CVE-2022-3786 and CVE-2022-3602. But I think we still want to use the
newer version of OpenSSL to avoid unnecessary confusion.
Currently, `CMAKE_INSTALL_FULL_{LIBEXEC,BIN,MAN}DIR`
are used for symlink creation. The problem with
that is it only works when the prefix is not
overridden by `cmake --install <build-dir> --prefix <prefix>`,
because their values are evaluated and hard-coded
into `cmake_install.cmake` during configuration
time, rather than installation time.
While it would be nice if we can get their
overridden values during installation, only
`CMAKE_INSTALL_PREFIX` reflects the overriden
value. (See "INSTALL A PROJECT", cmake(1)).
Therefore, the logic for symlink creation is
rewritten and extracted into a function named
`mold_install_symlink`. Now only `CMAKE_INSTALL_PREFIX`
is used during installation.
A new problem with this change is, if environment
variable `DESTDIR` is not set and `CMAKE_INSTALL_PREFIX`
is set to a relative path (which is perfectly
valid), a slash `/` would be incorrectly
prepended. To avoid that, and maintain
consistency with CMake's behaviour, the install
prefix is first turned into absolute path, stored
in `PREFIX_ABS`.
By the way, it seems okay to use the
`MAKE_DIRECTORY` and `CREATE_LINK` subcommands
under command `file`. So, it shouldn't be needed
to invoke a subprocess.
Signed-off-by: Zhong Ruoyu <[email protected]>
We can already print them, but a counter is more handy when the
executable is large and dumping everything would be too much.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
This isn't perf intensive at all but sweeping through the sections seems
to be the more obvious way compared to going through the symbol
indirections.
Also, reword the comments and timing name to better reflect what it's
doing.
FWIW, it's slightly (5ms) faster.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
A node converges if its descendants contains no cycles and its hash has reached
all its descendants. Previously we checked for convergence on each iteration by
comparing hashes, but this could be tracked more cheaply through a separate
bitset.
This improves cache efficiency and results in a ~50ms speed up for Chromium out
of 310ms for the ICF pass.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
It looks like the runtime doesn't care the non-loadable segments'
p_addr values, but `strip` does. `strip` corrupts an executable if
no value is set to p_vaddr.
Fixes https://github.com/rui314/mold/issues/839
This is an experimental flag and may be deleted in the future.
If --start-stop is given, mold creates __start_/__stop_ symbols for
sections whose names are not in C identifier style.
With toolchains using a relative sysroot, like crosstool-ng, gcc/clang will call the linker
with something like "--sysroot=/path/bin/../machine/sysroot".
The is_in_sysroot detection wasn't working in this case.
Since to_abs_path() uses lexically_normal() for relative paths, it seems consistent to do
the same for absolute paths instead of modifying is_in_sysroot().
They should inherit the original type instead of using a value from the wrong
enum (which showed up as R_X86_64_GOT32).
Signed-off-by: Tatsuyuki Ishi <[email protected]>
When using ld64.mold to link clang for Mach-O, most of the time was
spent in the first call to find_subsection() from
parse_compact_unwind(). Changing its linear search to a binary search
cut an 18-second link time down to about 3 seconds.
Signed-off-by: James Widman <[email protected]>
We only need the RMW to be atomic and doesn't need the ordering guarantee wrt
other memory locations.
Might make ARM a little faster (no machine code change for x86).
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Atomic RMW operations can take hundreds of cycles and tends to be massively
slower than a fence-free alternative.
Add a fast_mark function that optimistically tests if the node is already
visited first, which allows avoiding the fence penalty if we can early exit.
When testing on Chromium, this roughly improved the associated pass's runtime
by 20--30%.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
We've long had a hack to de-prioritize GNU_UNIQUE symbols, due to an edge case
where these symbols could get higher resolution priority yet eliminated in the
later COMDAT elimination stage. But since GNU_UNIQUE symbols were supposed to
have a resolution as strong as GLOBAL symbols, it created unsoundness in the
resolution and could have cause false duplicate symbol errors in later stages.
The root cause is that COMDAT elimination, unlike ICF and SHF_MERGE, removes
symbols from the candidate pool and can actually affect resolution result. Hence
we ought to do the elimination before the final resolution.
Doing so requires fully wiping the resolution instead of just the ones from
eliminated archive members. This has a performance hit that we need to keep an
eye on.
On another note this wiping of resolution is actually justifiable. The old
algorithm did not handle the case of an archive member overriding a DSO symbol.
So wiping is the correct way even in absence of COMDAT elimination.
One caveat is that currently the symbol resolution stage doesn't actually care
about the is_alive flag which is how COMDAT symbols are killed. There are two
other users of this flag: mergeable sections, and eh_frame. The eh_frame section
is already handled with special logic, and we can treat it as if it had no
symbols to resolve. Mergeable sections needs a bit more careful handling, but a
good way is to just leave them unmerged and update the references in the same
later pass where merge happens. This has been done in a prior commit.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
This pass is rather different in that it runs the resolve pass with lower
priority for archive symbols in order to determine extraction. Split it out to
better convey its difference.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
We can skip the is_alive condition by doing the compaction beforehand.
This also allows splitting this function into two phases clearly which will
happen later.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Make sure clear_symbols() are called on IR object files too, so that we'll never
have references to symbols inside IR object files after LTO. Can't really think
of an example that causes problem, but for sanity.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
When the section-header index has the escape value `SHN_XINDEX`, the
actual index must be looked up in the separate `SHT_SYMTAB_SHNDX` table.
Trying to use `SHN_XINDEX` (= 0xffff) as an index results in an
out-of-bounds read. The error can be observed when running the
`x86_64_many-sections.sh` test on RHEL 8 or 9 (but not on Fedora,
because there the assembler doesn't emit section symbols).
Instead of using `st_shndx` directly, call the pre-existing helper
method `get_shndx()` to get the correct behaviour.
Signed-off-by: Christoph Erhardt <[email protected]>
Even if we always enable assert(), we want code to be able to compile
with assert() being expanded to a null statement. I.e., we should not
write code that has side effect in assert().
In order to add a test case for it, I added a cmake option to force
disable assert().
Our assertions are cheap, and the cost of branch instructions that
are never taken is almost free thanks to hardware branch prediction.
So I don't think it's worth to disable it for release builds.
We set a value to .ARM.exidx's sh_shlink to appease GNU binutil's
strip command (this value is not used by runtime), but even strip
doesn't care what value is set! We set a wrong value by accident.
Fixes https://github.com/rui314/mold/issues/804
We've been treating STB_GNU_UNIQUE symbols as if they were weak.
The rationale of doing is this:
1. GCC sometimes creates GNU-unique symbols instead of weak ones
for comdats, and
2. after COMDAT de-duplication, we used to report a duplicate symbol
error for GNU-unique symbols
However, (2) is no longer the case because we do not report a symbol
duplication error if its section is dead.
I'm not sure if this new logic will work for all programs, but I want
to give it a shot.
https://github.com/rui314/mold/issues/524
GNU binutil's strip command doesn't like an .ARM.exidx section with
sh_link set to 0. arm-linux-gnueabihf-strip actually crashes on my
machine when the command is run on such a file.
https://github.com/rui314/mold/issues/801
536e0e7d ("[elf] Align .symtab's ELF attributes with .dynsym's.") fixed the
issue of .dynsym and .symtab disagreeing with each other, but in the
process introduced change to the symtab index. --emit-relocs was assuming
the old assignment logic, which caused mismatches as soon as any
global-to-local demotion happened.
Fix this to use the demotion-aware assignment logic by directly filling in
the indices as we assign them to symtab, and by making get_output_sym_idx
to test global/local using the new is_local() helper.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
I don't know why GCC sometimes creates a PC32 relocation instead of
PLT32 after a TLS_GD/TLS_LDM. I believe it's strictly speaking a
violation of the psABI. But we need to handle such input.
Fixes https://github.com/rui314/mold/issues/794
This matches what was done in `Makefile` in #561, but the check was extended
in the CMake port.
It builds fine without `-latomic` on Mageia 9 armv7hl nodes, fails with
explicit `-latomic`.
Signed-off-by: Rémi Verschelde <[email protected]>
Previously, an undefined symbol in a DSO can mark other DSO as "needed".
But that lead to mark more DSOs needed as necessary. It is also a bit
counter-intuitive.
The new behavior is still slightly different from GNU ld, but I believe
it's now compatible with LLVM lld.
Fixes https://github.com/rui314/mold/issues/786
As stated in the comment, have at least one permutation that checks for missing
(implicit) header includes.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Since we're having lots of arch-generic permutations, headers seems to be a good
place where potential perf improvement are possible.
The feature is gated against CMake 3.16.0 or later, which is when it was
implemented.
Measured on 3990X, with this change it takes:
ninja 1166.22s user 34.45s system 5378% cpu 22.322 total
While before it took:
ninja 1776.81s user 51.30s system 7172% cpu 25.489 total
So a 10% improvement for workstation users and even more if you don't.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
We have been setting the symbol value and size correctly, but the SHNDX was
erroneously set to UND, potentially causing undefined symbol errors. Fix this by
forwarding the appropriate value from the `leader`.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
It looks like we don't have to conservatively set GOT[0] to .dynamic
on s390x. glibc's -static-pie depends on GOT[0], so we need to set for
it, though.
These relocations are defined as follows
R_SPARC_GOTDATA_OP_HIX22 (G >> 10) ^ (G >> 31)
R_SPARC_GOTDATA_OP_LOX10 (G & 0x3ff) | ((G >> 31) & 0x1c00)
where G is an offset into .got. Since .got is much smaller than 4 GiB,
(G >> 31) is always zero, so we can ignore that part.
https://docs.oracle.com/cd/E19120-01/open.solaris/819-0690/chapter6-24/index.html
Fixes:
macho/input-files.cc:310:31: warning: missing initializer for member ‘mold::macho::Subsection<mold::macho::ARM64>::replacer’ [-Wmissing-field-initializers]
elf/mold-wrapper.c:68:15: warning: passing argument 1 of ‘debug_print’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
Signed-off-by: Martin Liska <[email protected]>
We should only override the default export behavior in executable if the default
version was actually overridden by a linker script (i.e. local: *;). Other
options, such as --dynamic-list and --export-dynamic-symbol[-list], doesn't
actually change this default, and should not be considered for this override.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Since around 2015, IBM seems to have been encouraging their customers to
use PowerPC systems in little-endian mode. They ratified the little-
endian ABI for PPC64. Since it was completely a binary-compatibility-
breaking change anyway, they didn't only change the endianness but made a
few improvements to the ABI they have been wanting to make. So the new ABI
was born.
The new ABI is called the ELFv2 ABI. The original, big-endian-only one
is retrospectively called ELFv1. Technically, the new ABI supports both
little endiannesses. but effectively all systems that employ ELFv2 are
using the processor in the little-endian mode.
There are still many PowerPC systems that use the original ELF ABI in the
little-endian mode for binary compatibility. This commit is for such systems.
We already support ELFv2. Look at arch-ppc64v2.cc.
Currently, "73% tests passed, 63 tests failed out of 234" for the ppc64v1
target according to ctest.
As explained in the "A classic way to use mold" section, the latest GCC releases support `mold` out of the box. Rust can take advantage of this to use `mold` too, without going through a system-wide `clang` linker.
I've just tested this on my Linux workstation that runs Debian unstable.
glibc for ARM64 prior to August 2021 arguably wrongly assumes that
the GOT first entry contains the address of the .dynamic section.
That's been removed in the following commit, but without satisfying
that assumption, mold can't create working static position-independent
execuables for ARM64.
https://sourceware.org/git/?p=glibc.git;h=43d06ed218fc8be58987bdfd00e21e5720f0b862
They are failing only on GitHub Actions. I believe they don't work
with older versions of glibc. Until we fix them, I want to disable
them so that the tests are green.
The implementation is far from complete, but mold/s390 is already able
to dynamically link a "Hello World" program for s390 which can
successfully run on qemu-s390x.
If a symbol has both PLT and GOT entries, we used to create its PLT
entry in .plt.got so that the runtime needs to look up the symbol
exactly once. This optimization however doesn't seem to be useful
because the number of such symbols is usually very small (<100 for
a large program). So I don't think its worth its complexity.
.gnu.warning* section may contain strings referenced by other sections,
so we can't discard them unconditionally. In this commit, we simply
keep all .gnu.warning* sections.
This allows CTest to differentiate between tests that were executed
successfully and tests that were skipped.
Signed-off-by: Christoph Erhardt <[email protected]>
We used to extend just a load segment containing RELRO sections.
strip command shrinks such segment so that it fits its contents.
So we need a dummy section for padding.
https://github.com/rui314/mold/issues/732
`readelf --symbols` print out auxiliary information after a symbol name
for PPC64, so something like `foo$` didn't match.
7: 0000000000000000 80 FUNC GLOBAL DEFAULT 3 main [<localentry>: 8]
8: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND .TOC.
9: 0000000000000000 72 FUNC GLOBAL DEFAULT 11 foo [<localentry>: 8]
10: 0000000000000000 40 FUNC GLOBAL DEFAULT 19 bar
Fixes https://github.com/rui314/mold/issues/739
They are both read-only, so by placing them next to each other, we can
reduce the number of the load segments by one, which translates to one
less mmap(2) at runtime.
This commit reverts the following commits:
- 3d33349691 [ELF] Fix -max-page-size
- 14b2ba3b74 Refactor
- 41c762b6c5 Update comments
- d22e85f345 [ELF] Save one memory page for RELRO
- 2941d75c71 [ELF] Do not page-align loadable segments
- a2552b6fd1 Refactor
- 919f58dc5d Fix CI
- ef6e6ecdf4 Refactor
- 9c7b95f108 [ELF] Do not page-align RELRO and TLS segments
I'll revert these changes because they caused a regression on systems
with large page size. Even though I believe most users are not
affected, it's a critical bug as mold creates completely unusable
output files for certain systems. In order to release 1.5.1, I'll
revert all of them.
This fixes the following error when Clang 15 is installed on the system:
> mold: error: out/test/elf/x86_64/run-clang/a.o:(.text):
> R_X86_64_32 relocation at offset 0x5 against symbol `.rodata' can not
> be used; recompile with -fPIC
The error occurs because Clang 15 assumes PIE to be enabled by default:
https://releases.llvm.org/15.0.0/tools/clang/docs/ReleaseNotes.html#build-system-changes
Attempting to produce a position-independent executable from an object
file that was created by `cc` without `-fPIE` caused the linker to bail
out.
Signed-off-by: Christoph Erhardt <[email protected]>
Looks like Intel C/C++ compiler emits this symbol in .eh_frame.
I don't excatly know how this symbol is being used, but it looks like
this symbol is supposed to refer the beginning of the output .eh_frame.
Fixes https://github.com/rui314/mold/issues/685
Previously, we align each loadable segment and the beginning of
readable-writable non-RELRO segment to page boundaries. That's one
posible layout, but it made too many holes in files that resulted
in output file bloat.
That bloat is most visible on targets supporting large pages.
We conservatively link output files for the largest possible page
size so that the output will run both on large pages and usual small
pages. As an example, we use 64 KiB as a page size for ARM64.
So we wasted up to 64 KiB for each loadable segment. A "hello world"
program for exapmle became almost 200 KiB on ARM64 because of that
reason.
This commit changes the file layout. Loadable segments are no longer
aligned to page boundaries. Loadable segments are still placed into
different memory pages. A page at a boundary of two loadable segments
are doubly mapped to memory.
This change reduces output file size. It is most noticeable when an
output is small -- a "hello world" on ARM64 is now smaller than 10
KiB now.
Since the actual writability is controlled by the W bit of the LOAD
segments, I don't think it actually matters. But in the spirit, TLS
segment contains the initial TLS image and should not be writable.
CMake 3.13 should be reasonably old enough. Ubuntu 20.04 includes
CMake 3.16.3. I wanted to support Ubuntu 18.04's CMake, but that was
3.10.2, and that's too old.
Almost all functions and data types are template in mold, which are
parameterized to target type (e.g. X86_64 or ARM32). I'm generally
happy with that design, but there's one drawback; as we add more
targets, compilation gets slower.
Now we support more than 10 targets, so the compiler has to do 10x
more work than it originall did. As a result, compiling output-chunks.cc
(one of our largest file) took more than 30 seconds to compile on a
Threadripper machine. I hasitated to add a support for a new target
because of this problem. We needed to fix it.
This commit tweaks the build system configs so that the build system
generates bunch of intermediate .cc files. Each generated .cc file
instantiates an original .cc file for one target.
The total amount of required computation doesn't change by this hack,
but now we can parallelize the compilation step. That works well on a
modern multi-core machine.
I generally don't like to tweak build system to workaround a tool's
issue (that's why I'm creating mold in the first place), but it looks
like there's no way to speed up compilation other than this. Or,
maybe we can write a faster C++ compiler someday, but that's another
topic.
The original PPC64 psABI was big-endian only which is retrospectively
called the ELF v1 ABI. Around 2016, IBM decided to push little-endian
use of PowerPC platform and ratified a new psABI which is called the
ELF v2 ABI.
Changing the endianness inevitably breaks ABI compatibility, so IBM
took that chance to clean up the ABI. So the difference between v1 and
v2 are not just endianness.
However, v2 isn't technically little-endian only. You can still create
an ELFv2-based big-endian system. Such systems are not popular though.
In many systems, the original ELFv1 big-endian system is called
"ppc64" and the v2 little-endian system is called "ppc64le". But these
names are not very accurate. I think ppc64v1 and ppc64v2 (or ppc64v2le
and ppc64v2be) are more accurate.
Bitfields are placed in an address-increasing order on both little
and big-endian hosts. Therefore, the following struct has the same
layout on both endiannesses.
struct {
uint32_t x : 8;
uint32_t y : 16;
}
The address of `x` is smaller than `y`, though you can't take an
address of a bitfield using the `&` operator in C.
However, if two bitfields are placed the same byte, they are placed in
a reverse order. In LE, bitfields are placed from LSB to MSB, while in
BE, they are placed from MSB to LSB. Therefore, members something like
this
struct {
uint32_t x : 4;
uint32_t y : 4;
}
needs to be swapped.
__tls_get_addr is not available in statically-linked executables,
so we have been relaxing all TLSGD and TLSLD relocations for static
executables. However, I noticed that it's much easier to provide a
replacement function for __tls_get_addr instead of implementing all
TLS relaxations.
With this change, all mold's unit tests pass on qemu-sparc64.
Previously, a relocation is associated either to a symbol or a section
fragment. Now, we synthesize a symbol for each unnamed piece of section
fragment and associate it to a relocation.
A symbol can belong to an input section, an output section or a section
fragment. They are mutually exclusive, and their data structures are
aligned to at least 4 bytes boundaries. So we can use a tagged pointer
to store all of them in a single pointer-size object.
GCC prints out bunch of the following warnings when compiling mold.
```
macho/input-files.cc:717:11: warning: 'void* memset(void*, int, size_t)'
clearing an object of non-trivial type 'struct mold::macho::MachSection';
use assignment or value-initialization instead [-Wclass-memaccess]
717 | memset(hdr, 0, sizeof(*hdr));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
```
This commit suppresses it by making our classes trivial.
https://github.com/rui314/mold/issues/708
Previously, TLSLD relocation on i386/x86-64 produced different values
than it normally does if it's relaxed. If relaxed, it's evaluated to
TP, which is the end address of the TLS block. If not relaxed, it's
evaluated to the beginning of the TLS block.
As a result, we needed to adjust subsequent DTPOFF relocations so that
it is based on the end of the TLS block only when the corresponding
symbol was relaxed. This is very tricky.
I believe the psABI TLS spec specifies this way because they couldn't
find a replacement instruction sequence that is equal to or shorter
then the replacing instructions. `mov %fs:0, %rax` is 9 bytes long,
so it barely fits the location where a TLSLD refers to.
I found that `xor %eax, %eax; mov %fs(%rax), %rax` is shorter than
`mov %fs:0, %rax` while it yields the same value. The two instructions
are 5 bytes in total. By saving 4 bytes, we can now add another `add`
instruction to adjust the resulting value, so that the result is always
relative to the beginning of the TLS block.
On ARM, calling an remaining weak undefined symbol is a NOP. But
if such symbol was promoted to a dynamic symbol, we need to execute
that call instruction instead of converting to a NOP because it may
be resolved to a defined symbol at runtime.
Fixes https://github.com/rui314/mold/issues/660
glibc started printing out the following warning message on process
startup, starting from glibc 2.36:
warning: direct reference to protected function `baz' in
`out/test/elf/x86_64/protected/b.so' may break pointer equality
It does so if it finds there's a canonical PLT against a protected
symbol. The warning message itself makes sense -- there's no way to
create a canonical PLT against a protected symbol while keeping
pointer equality, and you should recompile object files with `-fPIC`
to suppress this warning.
However, in this test, we intentionally create a canonical PLT against
a protected symbol, so we just want to silence it.
Fixes https://github.com/rui314/mold/issues/689
We saw a significant performance improvement with this commit. Here
is before and after numbers when cross-building mold for RV32 on a
simulated 16-core machine:
$ time ./mold.old @rsp
real 0m45.216s
user 0m0.016s
sys 0m0.000s
$ time ./mold.new @rsp
real 0m0.240s
user 0m0.014s
sys 0m0.003s
$ ls -lh mold
-rwxrwxr-x 1 ruiu ruiu 152M Sep 4 17:16 mold
For RISC-V, DTPOFF is computed 0x800 off from the start of each TLS block
to maximize the accessible range for memory access instructions with
signed 12-bit immediate. DTPMOD is usually computed by the dynamic loader,
but when we do for static binaries, we need to subtract 0x800 to offset
the offset.
https://github.com/rui314/mold/issues/461
This reverts commit e65cc7feea because
I don't think this is a correct fix.
We can't emit a R_RISCV_32 dynamic relocation for R_RISCV_HI20
relocation. If we do, the dynamic loader will override an entire 32
bit word with an address of a specified symbol. HI20 is expected to
set a high 20 bits as an instruction operand, so they are not
compatible.
https://github.com/rui314/mold/pull/668
I was not correct as the 'sorry, unimplemented: Thumb-1 'hard-float' VFP ABI
really comes late in target register allocator. Thus one needs to
assembly a function.
Fixes: #655
Signed-off-by: Martin Liska <[email protected]>
I don't think PIE uses R_AARCH64_ADR_PREL_PG_HI21 relocation because
in PIE, we need to use R_AARCH64_CALL26 or R_AARCH64_JUMP26 for branches
and use GOT to take an address of a global object.
- PC64 relocations are not representable using DYNREL because
DYNREL is for an absolute address
- PC-relative relocations should be able to refer PLT entries
in the same ELF module
Fixes#550.
In the Arm32 [spec][spec], R_ARM_GOT_PREL is defined as follows:
R_ARM_GOT_PREL: GOT(S) + A – P
Where:
> GOT(S) is the address of the GOT entry for the symbol S.
If we look at the code a bit, notice that G is actually the _offset_
of S into the GOT, not the symbol's address in GOT. This patch makes
sure that the relocation is computed using the GOT address of the
symbol, thereby fixing the mentioned issue.
[spec]: https://github.com/ARM-software/abi-aa/blob/60a8eb8c55e999d74dac5e368fc9d7e36e38dda4/aaelf32/aaelf32.rst#5612relocation-types
Signed-off-by: Robert Bartlensky <[email protected]>
If a R_ARM_CALL refers an ARM instruction that jumps to Thumb, we
need to rewrite an instruction to BLX. We previously didn't do that.
Fixes https://github.com/rui314/mold/issues/442
The following warnings are seen:
Testing pltgot ... /usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: warning: /tmp/ccpkwmmy.o: missing .note.GNU-stack section implies executable stack
/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
Testing reloc ... /usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: warning: out/test/elf/x86_64/reloc/a.o: missing .note.GNU-stack section implies executable stack
/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
Signed-off-by: Martin Liska <[email protected]>
The precondition check was supposed to find out whether the compiler
supports `-fcfprotection`, but it always failed complaining about an
undefined reference to `main()`. As a consequence, the test case was
never executed.
Add `-c` to omit the link step in the compiler call.
Signed-off-by: Christoph Erhardt <[email protected]>
Right now, the test prints to the output:
Testing defsym2 ... w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
U __cxa_finalize
w __gmon_start__
00000000000010f9 T bar
00000000000010f9 T foo
OK
which is changed to:
Testing defsym2 ... OK
Signed-off-by: Martin Liska <[email protected]>
I was debugging #550, and I noticed a symbol was pointing to the
start of a section, but its shndx was set to SHN_UNDEF. Looking over
the code, I realized that for each section you create a corresponding
output section, but if the input section is mergeable, you also create
a fragment for it whose parent is MergeableSection, which in the end
will be copied into the output binary instead of the output section.
Since the symbol was pointing to the start of a mergeable section, I
realized that it is probably wrong to use the shndx of
`isec->output_section`, because that particular Chunk never actually
makes it to the list of final Chunks for mergeable sections.
Before:
```
$ eu-readelf -s ./target/debug/mold_test | grep _rustc
Num: Value Size Type Bind Vis Ndx Name
77: 000d563c 34 OBJECT LOCAL DEFAULT UNDEF __rustc_debug_gdb_scripts_section__
$ eu-readelf -S ./target/debug/mold_test
[Nr] Name Type Addr Off Size ES Flags Lk Inf Al
...
[22] .debug_gdb_scripts PROGBITS 000d563c 0d563c 000022 0 A 0 0 1
...
```
After:
```
$ eu-readelf -s ./target/debug/mold_test | grep _rustc
Num: Value Size Type Bind Vis Ndx Name
77: 000d563c 34 OBJECT LOCAL DEFAULT 22 __rustc_debug_gdb_scripts_section__
$ eu-readelf -S ./target/debug/mold_test
[Nr] Name Type Addr Off Size ES Flags Lk Inf Al
...
[22] .debug_gdb_scripts PROGBITS 000d563c 0d563c 000022 0 A 0 0 1
...
```
Signed-off-by: Robert Bartlensky <[email protected]>
It seems buggy and causes subtle linker errors. Disable it for now so that
we can focus on getting the essential parts of the linker right.
Fixes https://github.com/rui314/mold/issues/619
These tests were added to the repo, but not the list of tests in the
corresponding CMakeLists. As a result, they are not run in CI when using
CTest.
Signed-off-by: Andrew Kaster <[email protected]>
MSVC has a hard limit of 128 block nesting levels. So you can't write
cascaded "else if"s more than 128. That's a very silly restriction,
but we need to work it around.
Both rust-demangle and xxhash provide usage instructions with include
paths that do not have an extra directory component. This patch fixes
uses of these libraries in the code and avoids including the third-party
directory wholesale.
Signed-off-by: friendlyanon <[email protected]>
Relocation may refer past-the-end location of a section. Therefore,
if section X is [0, 0x1000) and section Y is [0x1000, 0x2000), we
can't determine whether a relocation refers address 0x1000 refers
X or Y. We need to pass a section index.
The `extern "C"` directive in a version script should match a non-C++
mangled name for the sake of compatibility with other linkers. It
caused a regression for libprotobuf.
GCC 11's crtbeginS.o contains `__dso_handle` as a word-size data
which cnotains its own address as a value. Overriding that symbol
programs that depend on the value to crash.
This fix is needed for 7zip.
If a section name is valid as a C identifier (i.e. it doesn't start
with '.' and doesn't contain any punctuators), the linker automatically
creates new symbols by prepending `__start_` and `__stop_` to the
section name.
Previously, we conservatively keep such C identifier sections during
garbage collection. However, since `__start_` and `__stop_` symbols
are hidden symbols, if we do not have `__start_foo` or `__stop_foo`,
we can safely discard section `foo`.
write_symtab feels like we're writing to the input file, which isn't the
case. export_to_symtab conveys the meaning clearer.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
Notably, we previously wrote symbols that have been demoted into STB_LOCAL
as STB_GLOBAL. Fix this by using a logic shared with dynsym to determine
the output attributes.
Also reorder the symbols since the LOCAL ones need to come first.
Found in Mesa test suite.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
parse_dynamic_glob requires the passed string_view to be alive across the
entire linker invocation, so we need to avoid cloning for that.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
According to lld, --dynamic-list "implies -Bsymbolic but does not set
DF_SYMBOLIC" when creating shared objects. Follow suit in this patch.
Note that we don't set DF_SYMBOLIC currently, and arg.Bsymbolic has no
effect on non-shared objects.
Signed-off-by: Tatsuyuki Ishi <[email protected]>
This reverts commit 990b0ac4cd.
This is what the RISC-V psABI says
> The R_RISCV_HI20 refers to an LUI instruction containing the high
> 20-bits to be relocated to an absolute symbol address. The LUI
> instruction is followed by an I-Type instruction (add immediate or
> load) with an R_RISCV_LO12_I relocation or an S-Type instruction
> (store) and an R_RISCV_LO12_S relocation. The addresses for pair of
> relocations are calculated like this:
I read LO12 is guaranteed to always appear after HI20.
`cxa_finalize` takes `__dso_handle` to uniquify identify an ELF module
in memory. Its actual location doesn't matter but needs to be different
for each ELF module.
Fixes https://github.com/rui314/mold/issues/507
This change removes the compact PLT and non-IBT PLT and replaces
them with IBT PLT. The compact PLT wasn't compatible with `-z ibtplt`.
Fixes https://github.com/rui314/mold/issues/581
The command "mold -run foo.ld" can fail when foo.ld is an empty file.
The musl execvp() call fails with "Exec format error" when the script
to be executed is empty. The fix is to create foo.ld with a valid
Unix shebang.
Signed-off-by: Peter Wang <[email protected]>
This fixes a warning produced by GCC 12 related to the initialisation
order of the class members.
For details, see the discussion at
https://github.com/oneapi-src/oneTBB/issues/823. The concern about
potential ABI incompatibility is not relevant for a private bundled
library.
Signed-off-by: Christoph Erhardt <[email protected]>
This notation is supported by both GCC and Clang, whereas
`#pragma unroll` is only supported by Clang.
Signed-off-by: Christoph Erhardt <[email protected]>
Clang warns that forward-declaring a `struct` as `class`
'is valid, but may result in linker errors under the Microsoft C++ ABI'.
Signed-off-by: Christoph Erhardt <[email protected]>
This also fixes the `note-property` test which was hanging on my
machine. Turns out `$CC` was stuck waiting to read from `stdin`,
and as such I decided to write something into it.
Signed-off-by: Robert Bartlensky <[email protected]>
We used to compute a file UUID using only the hashes in the code
signature section. So we computes the same UUID even if two files
are differnet only in some field of the code signature section.
This reverts commit f3e1f728d3.
I'm not 100% sure why, but now the code signature verification works
way faster than before. Maybe it has improved in macOS 12.4 which was
released last month?
This reverts commit f6b661a9051f5657ac701ea9c94dd54a85499365.
We want to make it possible to run two or more mold's `main` function
simultaneously in a single process, so we don't want to make `opt_demangle`
a global variable. That's why it was a thread-local variable.
The output now roughly matches the lld report format, with
each symbol followed by a limited list of places that reference it.
Lld can additionally read debuginfo to report the exact location,
but that would require writing an interpreter for .debug_line.
Signed-off-by: Luboš Luňák <[email protected]>
Setting it on demand is racy, and also not necessary, it's enough
to set it together with ctx.arg.demangle .
Signed-off-by: Luboš Luňák <[email protected]>
`bool foo : 1` is better `u8 foo : 1` because
struct {
bool foo : 1 = 0;
} bar;
bar.foo = 0xf000;
sets `bar.foo` to 1 while
struct {
u8 foo : 1 = 0;
} bar;
bar.foo = 0xf000;
does _not_ set `bar.foo` to 1 because it just truncates a given
integer to one bit and assign it to `bar.foo`.
It looks like SHA1 is slightly faster than SHA256 on a 2021 ARM Mac mini.
SHA1 is not recommended anymore, but for ad-hoc code signing, it should
be enough because anyone can just recompute an ad-hoc code signature for
any arbitrary binary.
We need to compute SHA256 hashes for a __code_signature section,
so recomputing a SHA256 hash of an entire file for an LC_UUID is
a waste of time. We can instead compute it as a tree hash.
We eliminate unneeded symbols at the end of the algorithm, but that
didn't save too much space. In fact, I didn't observe any difference
by eliminating that last step from the algorithm.
This is an unusual scenario, but it can happen on Arch Linux, where
apparently even crt1.o has no .debug_info . So if one for whatever
reason omits -g for all compilations and then uses --gdb-index,
mold aborts.
Signed-off-by: Luboš Luňák <[email protected]>
It was accidentally mapped to --color-diagnostics=always, but it
was a bug. For the sake of compatibility with other tools such as
clang or lld, it should be defined as an alias for `auto`.
Previously, we detected if a given linker plugin supports get_symbols_v3
or not and restart the process if only get_symbols_v2 is supported.
Now, mold restart itself wehn get_symbols_v2 is called for the first time,
eliminating the need for the feature detection.
https://github.com/rui314/mold/issues/454
Previously, if a relocation in an non-alloc section refers a section
fragment, it gets a tombstone value instead of a real value.
Fixes https://github.com/rui314/mold/issues/490
- Rename data types so that we have LittleEndian and BigEndian types.
- Do not assume that "base" integral types are aligned to 2 bytes
boundaries, because that doesn't make much difference in terms of
performance.
- Remove `pu8` type
Core part of this change is cleaning up Mold from UBSan alignment errors.
Following approach inspired by LLVM's ELFTypes is done here:
1. Introduce generic Packed class acting as proxy for unaligned aggregated value.
Every access to such value through proxy is secured by memcpy call.
2. Replace plain struct members with Packed members wherever it matters.
3. Use explicit castings through Packed type for misaligned loads.
For more background please check: https://github.com/rui314/mold/discussions/477
Signed-off-by: Dawid Jurczak <[email protected]>
`off_t` is either an alias for `off32_t` or for `off64_t` on a 32-bit
machine. `off32_t` is the default. But it looks like LLVM gold plugin
is always compiled with `off64_t`.
Because of this difference, the offset of the `handle` member in
`PluginInputFile` differed between mold and LLVMgold.so. mold thought
that the member was at offset 16 of the struct, while LLVMgold.so
thought that it's at offset 24.
That caused a mysterious crash when LLVMgold tries to access the
`handle` member.
Fixes https://github.com/rui314/mold/issues/483
If a given linker LTO plugin does not support the v3 API, we resolve
all symbol names to identify the set of object files that are actually
needed for linking and then restart the linker to do the same thing
again from scratch, excluding the unneeded files from the beginning.
We do this because the v2 API does not provide a way to "unload" an
object file once it is read.
To identify an unneeded object file on the second run, we serialize
its name in the form of `<archive-name>:<offset-in-archive>` or just
a filename. If an object file is in a thin archive, we simply used
its filename.
But if the same file is directly given to a linker as well as as a
thin archive member, the file would accidentally be excluded on the
second run.
This change fixes that issue. With this change, it looks like we can
build GCC 12 with LTO using mold.
Fixes https://github.com/rui314/mold/issues/454
Delete this message and describe the bug here. Note that we only fix bugs in the most recent release of mold, so please make sure that you are using the most recent version.
This page explains the following warning message and how to fix it.
mold emits this message when it sees an object file that may not be
compatible with mold.
```
mold: warning: foo.o: this file may cause a segmentation fault because it requires an executable stack. See https://github.com/rui314/mold/tree/main/docs/execstack.md for more info.
```
# Background
On modern computers, the stack area (to which local variables are
stored) cannot contain executable code. If the control reaches the
stack area, the CPU refuses to execute any code there and the program
is usually terminated due to segmentation fault.
This is a security measure. The stack area used to be executable (old
CPUs generally execute any code as long as it is in a readable memory
region), but that provided an easy attack vector to a malicious user.
They wrote executable code to the stack area using some buffer
overflow bug and jumped there to run arbitrary code in a remote server
process.
To prevent this type of attack, the stack area is no longer executable
since the early 2000s. On Linux, the stack's executable-ness is
controlled by a bit in an executable, and the loader respect that
bit. The bit is set by the linker.
GCC had (and still have) a feature that depends on the executable
stack, so they invented a way to tell the linker to mark the stack
executable. Specifically, if an object file contains a
`.note.GNU-stack` section with the `SHF_EXECSTACK` bit, GNU linker
silently makes the stack of an output file executable.
But the GNU linker's behavior is dangerous. If you accidentally link
an object file that has that marker section, the entire stack area
silently becomes executable, disabling the security mechanism.
Therefore, mold simply ignores that marker section. If you are using
mold, you need to explicitly pass `-z execstack` to the linker to make
Loaded 100 of 2452 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Rui Ueyama
