Magic-Mirror/llvm-capstone
1
0
Fork 0
mirror of https://github.com/capstone-engine/llvm-capstone.git synced 2025-02-17 08:21:13 +00:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

[BOLT] Move from RuntimeDyld to JITLink

Browse Source 
RuntimeDyld has been deprecated in favor of JITLink. [1] This patch
replaces all uses of RuntimeDyld in BOLT with JITLink.

Care has been taken to minimize the impact on the code structure in
order to ease the inspection of this (rather large) changeset. Since
BOLT relied on the RuntimeDyld API in multiple places, this wasn't
always possible though and I'll explain the changes in code structure
first.

Design note: BOLT uses a JIT linker to perform what essentially is
static linking. No linked code is ever executed; the result of linking
is simply written back to an executable file. For this reason, I
restricted myself to the use of the core JITLink library and avoided ORC
as much as possible.

RuntimeDyld contains methods for loading objects (loadObject) and symbol
lookup (getSymbol). Since JITLink doesn't provide a class with a similar
interface, the BOLTLinker abstract class was added to implement it. It
was added to Core since both the Rewrite and RuntimeLibs libraries make
use of it. Wherever a RuntimeDyld object was used before, it was
replaced with a BOLTLinker object.

There is one major difference between the RuntimeDyld and BOLTLinker
interfaces: in JITLink, section allocation and the application of fixups
(relocation) happens in a single call (jitlink::link). That is, there is
no separate method like finalizeWithMemoryManagerLocking in RuntimeDyld.
BOLT used to remap sections between allocating (loadObject) and linking
them (finalizeWithMemoryManagerLocking). This doesn't work anymore with
JITLink. Instead, BOLTLinker::loadObject accepts a callback that is
called before fixups are applied which is used to remap sections.

The actual implementation of the BOLTLinker interface lives in the
JITLinkLinker class in the Rewrite library. It's the only part of the
BOLT code that should directly interact with the JITLink API.

For loading object, JITLinkLinker first creates a LinkGraph
(jitlink::createLinkGraphFromObject) and then links it (jitlink::link).
For the latter, it uses a custom JITLinkContext with the following
properties:
- Use BOLT's ExecutableFileMemoryManager. This one was updated to
  implement the JITLinkMemoryManager interface. Since BOLT never
  executes code, its finalization step is a no-op.
- Pass config: don't use the default target passes since they modify
  DWARF sections in a way that seems incompatible with BOLT. Also run a
  custom pre-prune pass that makes sure sections without symbols are not
  pruned by JITLink.
- Implement symbol lookup. This used to be implemented by
  BOLTSymbolResolver.
- Call the section mapper callback before the final linking step.
- Copy symbol values when the LinkGraph is resolved. Symbols are stored
  inside JITLinkLinker to ensure that later objects (i.e.,
  instrumentation libraries) can find them. This functionality used to
  be provided by RuntimeDyld but I did not find a way to use JITLink
  directly for this.

Some more minor points of interest:
- BinarySection::SectionID: JITLink doesn't have something equivalent to
  RuntimeDyld's Section IDs. Instead, sections can only be referred to
  by name. Hence, SectionID was updated to a string.
- There seem to be no tests for Mach-O. I've tested a small hello-world
  style binary but not more than that.
- On Mach-O, JITLink "normalizes" section names to include the segment
  name. I had to parse the section name back from this manually which
  feels slightly hacky.

[1] https://reviews.llvm.org/D145686#4222642

Reviewed By: rafauler

Differential Revision: https://reviews.llvm.org/D147544
This commit is contained in:
Job Noorman 2023-06-15 10:52:11 +02:00
parent 4d339ec91e
commit 05634f7346
22 changed files with 569 additions and 331 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
bolt/include/bolt/Core/BinaryContext.h
View File

@ -59,7 +59,6 @@ using namespace object;
namespace bolt {
class BinaryFunction;
class ExecutableFileMemoryManager;
/// Information on loadable part of the file.
struct SegmentInfo {
@ -313,10 +312,6 @@ public:
FilterIterator<binary_data_const_iterator>;
using FilteredBinaryDataIterator = FilterIterator<binary_data_iterator>;
/// Memory manager for sections and segments. Used to communicate with ORC
/// among other things.
std::shared_ptr<ExecutableFileMemoryManager> EFMM;
StringRef getFilename() const { return Filename; }
void setFilename(StringRef Name) { Filename = std::string(Name); }

8
bolt/include/bolt/Core/BinarySection.h
View File

@ -90,7 +90,7 @@ class BinarySection {
uint64_t OutputFileOffset{0}; // File offset in the rewritten binary file.
StringRef OutputContents; // Rewritten section contents.
const uint64_t SectionNumber; // Order in which the section was created.
unsigned SectionID{-1u}; // Unique ID used for address mapping.
std::string SectionID; // Unique ID used for address mapping.
// Set by ExecutableFileMemoryManager.
uint32_t Index{0}; // Section index in the output file.
mutable bool IsReordered{false}; // Have the contents been reordered?
@ -430,18 +430,18 @@ public:
}
uint64_t getOutputAddress() const { return OutputAddress; }
uint64_t getOutputFileOffset() const { return OutputFileOffset; }
unsigned getSectionID() const {
StringRef getSectionID() const {
assert(hasValidSectionID() && "trying to use uninitialized section id");
return SectionID;
}
bool hasValidSectionID() const { return SectionID != -1u; }
bool hasValidSectionID() const { return !SectionID.empty(); }
bool hasValidIndex() { return Index != 0; }
uint32_t getIndex() const { return Index; }
// mutation
void setOutputAddress(uint64_t Address) { OutputAddress = Address; }
void setOutputFileOffset(uint64_t Offset) { OutputFileOffset = Offset; }
void setSectionID(unsigned ID) {
void setSectionID(StringRef ID) {
assert(!hasValidSectionID() && "trying to set section id twice");
SectionID = ID;
}

48
bolt/include/bolt/Core/Linker.h Normal file
View File

@ -0,0 +1,48 @@
//===- bolt/Core/Linker.h - BOLTLinker interface ----------------*- C++ -*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file contains the interface BOLT uses for linking.
//
//===----------------------------------------------------------------------===//
#ifndef BOLT_CORE_LINKER_H
#define BOLT_CORE_LINKER_H
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/MemoryBufferRef.h"
#include <cstdint>
#include <functional>
#include <optional>
namespace llvm {
namespace bolt {
class BinarySection;
class BOLTLinker {
public:
using SectionMapper =
std::function<void(const BinarySection &Section, uint64_t Address)>;
using SectionsMapper = std::function<void(SectionMapper)>;
virtual ~BOLTLinker() = default;
/// Load and link \p Obj. \p MapSections will be called before the object is
/// linked to allow section addresses to be remapped. When called, the address
/// of a section can be changed by calling the passed SectionMapper.
virtual void loadObject(MemoryBufferRef Obj, SectionsMapper MapSections) = 0;
/// Return the address of a symbol or std::nullopt if it cannot be found.
virtual std::optional<uint64_t> lookupSymbol(StringRef Name) const = 0;
};
} // namespace bolt
} // namespace llvm
#endif // BOLT_CORE_LINKER_H

64
bolt/include/bolt/Rewrite/ExecutableFileMemoryManager.h
View File

@ -10,7 +10,7 @@
#define BOLT_REWRITE_EXECUTABLE_FILE_MEMORY_MANAGER_H
#include "llvm/ADT/StringRef.h"
#include "llvm/ExecutionEngine/RuntimeDyld.h"
#include "llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h"
#include <cstdint>
#include <string>
@ -20,20 +20,12 @@ namespace bolt {
class BinaryContext;
/// Class responsible for allocating and managing code and data sections.
class ExecutableFileMemoryManager : public RuntimeDyld::MemoryManager {
class ExecutableFileMemoryManager : public jitlink::JITLinkMemoryManager {
private:
uint8_t *allocateSection(uintptr_t Size, unsigned Alignment,
unsigned SectionID, StringRef SectionName,
bool IsCode, bool IsReadOnly);
BinaryContext &BC;
bool AllowStubs;
void updateSection(const jitlink::Section &Section, uint8_t *Contents,
size_t Size, size_t Alignment);
struct AllocInfo {
uint8_t *Address;
size_t Size;
size_t Alignment;
};
SmallVector<AllocInfo, 8> AllocatedSections;
BinaryContext &BC;
// All new sections will be identified by the following prefix.
std::string NewSecPrefix;
@ -50,48 +42,14 @@ public:
// user-supplied objects into the main input executable.
uint32_t ObjectsLoaded{0};
ExecutableFileMemoryManager(BinaryContext &BC, bool AllowStubs)
: BC(BC), AllowStubs(AllowStubs) {}
ExecutableFileMemoryManager(BinaryContext &BC) : BC(BC) {}
~ExecutableFileMemoryManager();
void allocate(const jitlink::JITLinkDylib *JD, jitlink::LinkGraph &G,
OnAllocatedFunction OnAllocated) override;
uint8_t *allocateCodeSection(uintptr_t Size, unsigned Alignment,
unsigned SectionID,
StringRef SectionName) override {
return allocateSection(Size, Alignment, SectionID, SectionName,
/*IsCode=*/true, true);
}
uint8_t *allocateDataSection(uintptr_t Size, unsigned Alignment,
unsigned SectionID, StringRef SectionName,
bool IsReadOnly) override {
return allocateSection(Size, Alignment, SectionID, SectionName,
/*IsCode=*/false, IsReadOnly);
}
// Ignore TLS sections by treating them as a regular data section
TLSSection allocateTLSSection(uintptr_t Size, unsigned Alignment,
unsigned SectionID,
StringRef SectionName) override {
TLSSection Res;
Res.Offset = 0;
Res.InitializationImage = allocateDataSection(
Size, Alignment, SectionID, SectionName, /*IsReadOnly=*/false);
return Res;
}
bool allowStubAllocation() const override { return AllowStubs; }
/// Count processed objects and skip memory finalization.
bool finalizeMemory(std::string *ErrMsg) override {
++ObjectsLoaded;
return false;
}
/// Ignore EH frames.
void registerEHFrames(uint8_t *Addr, uint64_t LoadAddr,
size_t Size) override {}
void deregisterEHFrames() override {}
void deallocate(std::vector<FinalizedAlloc> Allocs,
OnDeallocatedFunction OnDeallocated) override;
using JITLinkMemoryManager::deallocate;
/// Section name management.
void setNewSecPrefix(StringRef Prefix) { NewSecPrefix = Prefix; }

56
bolt/include/bolt/Rewrite/JITLinkLinker.h Normal file
View File

@ -0,0 +1,56 @@
//===- bolt/Rewrite/JITLinkLinker.h - Linker using JITLink ------*- C++ -*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// BOLTLinker using JITLink.
//
//===----------------------------------------------------------------------===//
#ifndef BOLT_REWRITE_JITLINK_LINKER_H
#define BOLT_REWRITE_JITLINK_LINKER_H
#include "bolt/Core/Linker.h"
#include "bolt/Rewrite/ExecutableFileMemoryManager.h"
#include "llvm/ExecutionEngine/JITLink/JITLinkDylib.h"
#include <map>
#include <memory>
#include <vector>
namespace llvm {
namespace bolt {
class BinaryContext;
class JITLinkLinker : public BOLTLinker {
private:
struct Context;
friend struct Context;
BinaryContext &BC;
std::unique_ptr<ExecutableFileMemoryManager> MM;
jitlink::JITLinkDylib Dylib{"main"};
std::vector<ExecutableFileMemoryManager::FinalizedAlloc> Allocs;
std::map<std::string, uint64_t> Symtab;
public:
JITLinkLinker(BinaryContext &BC,
std::unique_ptr<ExecutableFileMemoryManager> MM);
~JITLinkLinker();
void loadObject(MemoryBufferRef Obj, SectionsMapper MapSections) override;
std::optional<uint64_t> lookupSymbol(StringRef Name) const override;
static SmallVector<jitlink::Block *, 2>
orderedBlocks(const jitlink::Section &Section);
static size_t sectionSize(const jitlink::Section &Section);
};
} // namespace bolt
} // namespace llvm
#endif // BOLT_REWRITE_JITLINK_LINKER_H

9
bolt/include/bolt/Rewrite/MachORewriteInstance.h
View File

@ -13,13 +13,13 @@
#ifndef BOLT_REWRITE_MACHO_REWRITE_INSTANCE_H
#define BOLT_REWRITE_MACHO_REWRITE_INSTANCE_H
#include "bolt/Core/Linker.h"
#include "bolt/Utils/NameResolver.h"
#include "llvm/Support/Error.h"
#include <memory>
namespace llvm {
class ToolOutputFile;
class RuntimeDyld;
class raw_pwrite_stream;
namespace object {
class MachOObjectFile;
@ -37,7 +37,7 @@ class MachORewriteInstance {
NameResolver NR;
std::unique_ptr<RuntimeDyld> RTDyld;
std::unique_ptr<BOLTLinker> Linker;
std::unique_ptr<ToolOutputFile> Out;
@ -49,8 +49,9 @@ class MachORewriteInstance {
static StringRef getNewSecPrefix() { return ".bolt.new"; }
static StringRef getOrgSecPrefix() { return ".bolt.org"; }
void mapInstrumentationSection(StringRef SectionName);
void mapCodeSections();
void mapInstrumentationSection(StringRef SectionName,
BOLTLinker::SectionMapper MapSection);
void mapCodeSections(BOLTLinker::SectionMapper MapSection);
void adjustCommandLineOptions();
void readSpecialSections();

9
bolt/include/bolt/Rewrite/RewriteInstance.h
View File

@ -14,6 +14,7 @@
#define BOLT_REWRITE_REWRITE_INSTANCE_H
#include "bolt/Core/BinaryContext.h"
#include "bolt/Core/Linker.h"
#include "bolt/Utils/NameResolver.h"
#include "llvm/ADT/ArrayRef.h"
#include "llvm/MC/StringTableBuilder.h"
@ -206,13 +207,13 @@ private:
std::vector<BinarySection *> getCodeSections();
/// Map all sections to their final addresses.
void mapFileSections(RuntimeDyld &RTDyld);
void mapFileSections(BOLTLinker::SectionMapper MapSection);
/// Map code sections generated by BOLT.
void mapCodeSections(RuntimeDyld &RTDyld);
void mapCodeSections(BOLTLinker::SectionMapper MapSection);
/// Map the rest of allocatable sections.
void mapAllocatableSections(RuntimeDyld &RTDyld);
void mapAllocatableSections(BOLTLinker::SectionMapper MapSection);
/// Update output object's values based on the final \p Layout.
void updateOutputValues(const MCAsmLayout &Layout);
@ -459,7 +460,7 @@ private:
std::unique_ptr<CFIReaderWriter> CFIRdWrt;
// Run ExecutionEngine linker with custom memory manager and symbol resolver.
std::unique_ptr<RuntimeDyld> RTDyld;
std::unique_ptr<BOLTLinker> Linker;
/// Output file where we mix original code from the input binary and
/// optimized code for selected functions.

4
bolt/include/bolt/RuntimeLibs/HugifyRuntimeLibrary.h
View File

@ -28,8 +28,8 @@ public:
void emitBinary(BinaryContext &BC, MCStreamer &Streamer) final {}
void link(BinaryContext &BC, StringRef ToolPath, RuntimeDyld &RTDyld,
std::function<void(RuntimeDyld &)> OnLoad) final;
void link(BinaryContext &BC, StringRef ToolPath, BOLTLinker &Linker,
BOLTLinker::SectionsMapper MapSections) override;
};
} // namespace bolt

4
bolt/include/bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h
View File

@ -35,8 +35,8 @@ public:
void emitBinary(BinaryContext &BC, MCStreamer &Streamer) final;
void link(BinaryContext &BC, StringRef ToolPath, RuntimeDyld &RTDyld,
std::function<void(RuntimeDyld &)> OnLoad) final;
void link(BinaryContext &BC, StringRef ToolPath, BOLTLinker &Linker,
BOLTLinker::SectionsMapper MapSections) override;
private:
std::string buildTables(BinaryContext &BC);

9
bolt/include/bolt/RuntimeLibs/RuntimeLibrary.h
View File

@ -15,6 +15,7 @@
#ifndef BOLT_RUNTIMELIBS_RUNTIME_LIBRARY_H
#define BOLT_RUNTIMELIBS_RUNTIME_LIBRARY_H
#include "bolt/Core/Linker.h"
#include "llvm/ADT/StringRef.h"
#include <functional>
#include <vector>
@ -22,7 +23,6 @@
namespace llvm {
class MCStreamer;
class RuntimeDyld;
namespace bolt {
@ -51,8 +51,8 @@ public:
virtual void emitBinary(BinaryContext &BC, MCStreamer &Streamer) = 0;
/// Link with the library code.
virtual void link(BinaryContext &BC, StringRef ToolPath, RuntimeDyld &RTDyld,
std::function<void(RuntimeDyld &)> OnLoad) = 0;
virtual void link(BinaryContext &BC, StringRef ToolPath, BOLTLinker &Linker,
BOLTLinker::SectionsMapper MapSections) = 0;
protected:
/// The fini and init address set by the runtime library.
@ -63,7 +63,8 @@ protected:
static std::string getLibPath(StringRef ToolPath, StringRef LibFileName);
/// Load a static runtime library specified by \p LibPath.
static void loadLibrary(StringRef LibPath, RuntimeDyld &RTDyld);
static void loadLibrary(StringRef LibPath, BOLTLinker &Linker,
BOLTLinker::SectionsMapper MapSections);
};
} // namespace bolt

4
bolt/lib/Core/BinaryContext.cpp
View File

@ -1975,7 +1975,9 @@ void BinaryContext::deregisterUnusedSections() {
ErrorOr<BinarySection &> AbsSection = getUniqueSectionByName("<absolute>");
for (auto SI = Sections.begin(); SI != Sections.end();) {
BinarySection *Section = *SI;
if (Section->hasSectionRef() || Section->getOutputSize() ||
// We check getOutputData() instead of getOutputSize() because sometimes
// zero-sized .text.cold sections are allocated.
if (Section->hasSectionRef() || Section->getOutputData() ||
(AbsSection && Section == &AbsSection.get())) {
++SI;
continue;

5
bolt/lib/Core/BinaryEmitter.cpp
View File

@ -189,13 +189,16 @@ void BinaryEmitter::emitAll(StringRef OrgSecPrefix) {
if (opts::UpdateDebugSections && BC.isELF()) {
// Force the emission of debug line info into allocatable section to ensure
// RuntimeDyld will process it without ProcessAllSections flag.
// JITLink will process it.
//
// NB: on MachO all sections are required for execution, hence no need
// to change flags/attributes.
MCSectionELF *ELFDwarfLineSection =
static_cast<MCSectionELF *>(BC.MOFI->getDwarfLineSection());
ELFDwarfLineSection->setFlags(ELF::SHF_ALLOC);
MCSectionELF *ELFDwarfLineStrSection =
static_cast<MCSectionELF *>(BC.MOFI->getDwarfLineStrSection());
ELFDwarfLineStrSection->setFlags(ELF::SHF_ALLOC);
}
if (RuntimeLibrary *RtLibrary = BC.getRuntimeLibrary())

15
bolt/lib/Core/DebugData.cpp
View File

@ -1635,15 +1635,6 @@ void DwarfLineTable::emitCU(MCStreamer *MCOS, MCDwarfLineTableParams Params,
"cannot combine raw data with new line entries");
MCOS->emitLabel(getLabel());
MCOS->emitBytes(RawData);
// Emit fake relocation for RuntimeDyld to always allocate the section.
//
// FIXME: remove this once RuntimeDyld stops skipping allocatable sections
// without relocations.
MCOS->emitRelocDirective(
*MCConstantExpr::create(0, *BC.Ctx), "BFD_RELOC_NONE",
MCSymbolRefExpr::create(getLabel(), *BC.Ctx), SMLoc(), *BC.STI);
return;
}
@ -1725,12 +1716,8 @@ void DwarfLineTable::emit(BinaryContext &BC, MCStreamer &Streamer) {
// Still need to write the section out for the ExecutionEngine, and temp in
// memory object we are constructing.
if (LineStr) {
if (LineStr)
LineStr->emitSection(&Streamer);
SmallString<0> Data = LineStr->getFinalizedData();
BC.registerOrUpdateNoteSection(".debug_line_str", copyByteArray(Data.str()),
Data.size());
}
}
} // namespace bolt

3
bolt/lib/Rewrite/CMakeLists.txt
View File

@ -1,7 +1,7 @@
set(LLVM_LINK_COMPONENTS
DebugInfoDWARF
DWP
ExecutionEngine
JITLink
MC
Object
Support
@ -13,6 +13,7 @@ add_llvm_library(LLVMBOLTRewrite
BoltDiff.cpp
DWARFRewriter.cpp
ExecutableFileMemoryManager.cpp
JITLinkLinker.cpp
MachORewriteInstance.cpp
RewriteInstance.cpp

153
bolt/lib/Rewrite/ExecutableFileMemoryManager.cpp
View File

@ -7,7 +7,9 @@
//===----------------------------------------------------------------------===//
#include "bolt/Rewrite/ExecutableFileMemoryManager.h"
#include "bolt/Rewrite/JITLinkLinker.h"
#include "bolt/Rewrite/RewriteInstance.h"
#include "llvm/ExecutionEngine/JITLink/JITLink.h"
#include "llvm/Support/MemAlloc.h"
#undef DEBUG_TYPE
@ -21,30 +23,105 @@ namespace llvm {
namespace bolt {
uint8_t *ExecutableFileMemoryManager::allocateSection(
uintptr_t Size, unsigned Alignment, unsigned SectionID,
StringRef SectionName, bool IsCode, bool IsReadOnly) {
uint8_t *Ret = static_cast<uint8_t *>(llvm::allocate_buffer(Size, Alignment));
AllocatedSections.push_back(AllocInfo{Ret, Size, Alignment});
namespace {
// A Size of 1 might mean an empty section for which RuntimeDyld decided to
// allocate 1 byte. In this case, the allocation will never be initialized
// causing non-deterministic output section contents.
if (Size == 1)
*Ret = 0;
SmallVector<jitlink::Section *> orderedSections(jitlink::LinkGraph &G) {
SmallVector<jitlink::Section *> Sections(
llvm::map_range(G.sections(), [](auto &S) { return &S; }));
llvm::sort(Sections, [](const auto *LHS, const auto *RHS) {
return LHS->getOrdinal() < RHS->getOrdinal();
});
return Sections;
}
size_t sectionAlignment(const jitlink::Section &Section) {
assert(!Section.empty() && "Cannot get alignment for empty section");
return JITLinkLinker::orderedBlocks(Section).front()->getAlignment();
}
StringRef sectionName(const jitlink::Section &Section,
const BinaryContext &BC) {
auto Name = Section.getName();
if (BC.isMachO()) {
// JITLink "normalizes" section names as "SegmentName,SectionName" on
// Mach-O. BOLT internally refers to sections just by the section name so
// strip-off the segment name.
auto SegmentEnd = Name.find(',');
assert(SegmentEnd != StringRef::npos && "Mach-O segment not found");
Name = Name.substr(SegmentEnd + 1);
}
return Name;
}
struct SectionAllocInfo {
void *Address;
size_t Size;
size_t Alignment;
};
struct AllocInfo {
SmallVector<SectionAllocInfo, 8> AllocatedSections;
~AllocInfo() {
for (auto &Section : AllocatedSections)
deallocate_buffer(Section.Address, Section.Size, Section.Alignment);
}
SectionAllocInfo allocateSection(const jitlink::Section &Section) {
auto Size = JITLinkLinker::sectionSize(Section);
auto Alignment = sectionAlignment(Section);
auto *Buf = allocate_buffer(Size, Alignment);
SectionAllocInfo Alloc{Buf, Size, Alignment};
AllocatedSections.push_back(Alloc);
return Alloc;
}
};
struct BOLTInFlightAlloc : ExecutableFileMemoryManager::InFlightAlloc {
// Even though this is passed using a raw pointer in FinalizedAlloc, we keep
// it in a unique_ptr as long as possible to enjoy automatic cleanup when
// something goes wrong.
std::unique_ptr<AllocInfo> Alloc;
public:
BOLTInFlightAlloc(std::unique_ptr<AllocInfo> Alloc)
: Alloc(std::move(Alloc)) {}
virtual void abandon(OnAbandonedFunction OnAbandoned) override {
llvm_unreachable("unexpected abandoned allocation");
}
virtual void finalize(OnFinalizedFunction OnFinalized) override {
OnFinalized(ExecutableFileMemoryManager::FinalizedAlloc(
orc::ExecutorAddr::fromPtr(Alloc.release())));
}
};
} // anonymous namespace
void ExecutableFileMemoryManager::updateSection(
const jitlink::Section &JLSection, uint8_t *Contents, size_t Size,
size_t Alignment) {
auto SectionID = JLSection.getName();
auto SectionName = sectionName(JLSection, BC);
auto Prot = JLSection.getMemProt();
auto IsCode = (Prot & orc::MemProt::Exec) != orc::MemProt::None;
auto IsReadOnly = (Prot & orc::MemProt::Write) == orc::MemProt::None;
// Register a debug section as a note section.
if (!ObjectsLoaded && RewriteInstance::isDebugSection(SectionName)) {
BinarySection &Section =
BC.registerOrUpdateNoteSection(SectionName, Ret, Size, Alignment);
BC.registerOrUpdateNoteSection(SectionName, Contents, Size, Alignment);
Section.setSectionID(SectionID);
assert(!Section.isAllocatable() && "note sections cannot be allocatable");
return Ret;
return;
}
if (!IsCode && (SectionName == ".strtab" || SectionName == ".symtab" ||
SectionName == "" || SectionName.startswith(".rela.")))
return Ret;
return;
SmallVector<char, 256> Buf;
if (ObjectsLoaded > 0) {
@ -70,7 +147,7 @@ uint8_t *ExecutableFileMemoryManager::allocateSection(
"Original section must exist and be allocatable.");
Section = &OrgSection.get();
Section->updateContents(Ret, Size);
Section->updateContents(Contents, Size);
} else {
// If the input contains a section with the section name, rename it in the
// output file to avoid the section name conflict and emit the new section
@ -87,7 +164,7 @@ uint8_t *ExecutableFileMemoryManager::allocateSection(
// sections in the input file.
BinarySection &NewSection = BC.registerOrUpdateSection(
UsePrefix ? NewSecPrefix + SectionName : SectionName, ELF::SHT_PROGBITS,
BinarySection::getFlags(IsReadOnly, IsCode, true), Ret, Size,
BinarySection::getFlags(IsReadOnly, IsCode, true), Contents, Size,
Alignment);
if (UsePrefix)
NewSection.setOutputName(SectionName);
@ -100,17 +177,51 @@ uint8_t *ExecutableFileMemoryManager::allocateSection(
<< " section : " << Section->getOutputName() << " ("
<< Section->getName() << ")"
<< " with size " << Size << ", alignment " << Alignment << " at "
<< Ret << ", ID = " << SectionID << "\n";
<< Contents << ", ID = " << SectionID << "\n";
});
Section->setSectionID(SectionID);
return Ret;
}
ExecutableFileMemoryManager::~ExecutableFileMemoryManager() {
for (const AllocInfo &AI : AllocatedSections)
llvm::deallocate_buffer(AI.Address, AI.Size, AI.Alignment);
void ExecutableFileMemoryManager::allocate(const jitlink::JITLinkDylib *JD,
jitlink::LinkGraph &G,
OnAllocatedFunction OnAllocated) {
auto Alloc = std::make_unique<AllocInfo>();
for (auto *Section : orderedSections(G)) {
if (Section->empty())
continue;
auto SectionAlloc = Alloc->allocateSection(*Section);
updateSection(*Section, static_cast<uint8_t *>(SectionAlloc.Address),
SectionAlloc.Size, SectionAlloc.Alignment);
size_t CurrentOffset = 0;
auto *Buf = static_cast<char *>(SectionAlloc.Address);
for (auto *Block : JITLinkLinker::orderedBlocks(*Section)) {
CurrentOffset = jitlink::alignToBlock(CurrentOffset, *Block);
auto BlockSize = Block->getSize();
auto *BlockBuf = Buf + CurrentOffset;
if (Block->isZeroFill())
std::memset(BlockBuf, 0, BlockSize);
else
std::memcpy(BlockBuf, Block->getContent().data(), BlockSize);
Block->setMutableContent({BlockBuf, Block->getSize()});
CurrentOffset += BlockSize;
}
}
OnAllocated(std::make_unique<BOLTInFlightAlloc>(std::move(Alloc)));
}
void ExecutableFileMemoryManager::deallocate(
std::vector<FinalizedAlloc> Allocs, OnDeallocatedFunction OnDeallocated) {
for (auto &Alloc : Allocs)
delete Alloc.release().toPtr<AllocInfo *>();
OnDeallocated(Error::success());
}
} // namespace bolt

206
bolt/lib/Rewrite/JITLinkLinker.cpp Normal file
View File

@ -0,0 +1,206 @@
//===- bolt/Rewrite/JITLinkLinker.cpp - BOLTLinker using JITLink ----------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
#include "bolt/Rewrite/JITLinkLinker.h"
#include "bolt/Core/BinaryData.h"
#include "bolt/Rewrite/RewriteInstance.h"
#include "llvm/ExecutionEngine/JITLink/JITLink.h"
#include "llvm/ExecutionEngine/Orc/Shared/ExecutorAddress.h"
#include "llvm/ExecutionEngine/Orc/Shared/ExecutorSymbolDef.h"
#include "llvm/Support/Debug.h"
#define DEBUG_TYPE "bolt"
namespace llvm {
namespace bolt {
namespace {
bool hasSymbols(const jitlink::Block &B) {
return llvm::any_of(B.getSection().symbols(),
[&B](const auto &S) { return &S->getBlock() == &B; });
}
/// Liveness in JITLink is based on symbols so sections that do not contain
/// any symbols will always be pruned. This pass adds anonymous symbols to
/// needed sections to prevent pruning.
Error markSectionsLive(jitlink::LinkGraph &G) {
for (auto &Section : G.sections()) {
// We only need allocatable sections.
if (Section.getMemLifetimePolicy() == orc::MemLifetimePolicy::NoAlloc)
continue;
// Skip empty sections.
if (JITLinkLinker::sectionSize(Section) == 0)
continue;
for (auto *Block : Section.blocks()) {
// No need to add symbols if it already has some.
if (hasSymbols(*Block))
continue;
G.addAnonymousSymbol(*Block, /*Offset=*/0, /*Size=*/0,
/*IsCallable=*/false, /*IsLive=*/true);
}
}
return jitlink::markAllSymbolsLive(G);
}
void reassignSectionAddress(jitlink::LinkGraph &LG,
const BinarySection &BinSection, uint64_t Address) {
auto *JLSection = LG.findSectionByName(BinSection.getSectionID());
assert(JLSection && "cannot find section in LinkGraph");
auto BlockAddress = Address;
for (auto *Block : JITLinkLinker::orderedBlocks(*JLSection)) {
// FIXME it would seem to make sense to align here. However, in
// non-relocation mode, we simply use the original address of functions
// which might not be aligned with the minimum alignment used by
// BinaryFunction (2). Example failing test when aligning:
// bolt/test/X86/addr32.s
Block->setAddress(orc::ExecutorAddr(BlockAddress));
BlockAddress += Block->getSize();
}
}
} // anonymous namespace
struct JITLinkLinker::Context : jitlink::JITLinkContext {
JITLinkLinker &Linker;
JITLinkLinker::SectionsMapper MapSections;
Context(JITLinkLinker &Linker, JITLinkLinker::SectionsMapper MapSections)
: JITLinkContext(&Linker.Dylib), Linker(Linker),
MapSections(MapSections) {}
jitlink::JITLinkMemoryManager &getMemoryManager() override {
return *Linker.MM;
}
bool shouldAddDefaultTargetPasses(const Triple &TT) const override {
// The default passes manipulate DWARF sections in a way incompatible with
// BOLT.
// TODO check if we can actually use these passes to remove some of the
// DWARF manipulation done in BOLT.
return false;
}
Error modifyPassConfig(jitlink::LinkGraph &G,
jitlink::PassConfiguration &Config) override {
Config.PrePrunePasses.push_back(markSectionsLive);
return Error::success();
}
void notifyFailed(Error Err) override {
errs() << "BOLT-ERROR: JITLink failed: " << Err << '\n';
exit(1);
}
void
lookup(const LookupMap &Symbols,
std::unique_ptr<jitlink::JITLinkAsyncLookupContinuation> LC) override {
jitlink::AsyncLookupResult AllResults;
for (const auto &Symbol : Symbols) {
std::string SymName = Symbol.first.str();
LLVM_DEBUG(dbgs() << "BOLT: looking for " << SymName << "\n");
if (auto Address = Linker.lookupSymbol(SymName)) {
LLVM_DEBUG(dbgs() << "Resolved to address 0x"
<< Twine::utohexstr(*Address) << "\n");
AllResults[Symbol.first] = orc::ExecutorSymbolDef(
orc::ExecutorAddr(*Address), JITSymbolFlags());
continue;
}
if (const BinaryData *I = Linker.BC.getBinaryDataByName(SymName)) {
uint64_t Address = I->isMoved() && !I->isJumpTable()
? I->getOutputAddress()
: I->getAddress();
LLVM_DEBUG(dbgs() << "Resolved to address 0x"
<< Twine::utohexstr(Address) << "\n");
AllResults[Symbol.first] = orc::ExecutorSymbolDef(
orc::ExecutorAddr(Address), JITSymbolFlags());
continue;
}
LLVM_DEBUG(dbgs() << "Resolved to address 0x0\n");
AllResults[Symbol.first] =
orc::ExecutorSymbolDef(orc::ExecutorAddr(0), JITSymbolFlags());
}
LC->run(std::move(AllResults));
}
Error notifyResolved(jitlink::LinkGraph &G) override {
MapSections([&G](const BinarySection &Section, uint64_t Address) {
reassignSectionAddress(G, Section, Address);
});
for (auto *Symbol : G.defined_symbols()) {
Linker.Symtab.insert(
{Symbol->getName().str(), Symbol->getAddress().getValue()});
}
return Error::success();
}
void notifyFinalized(
jitlink::JITLinkMemoryManager::FinalizedAlloc Alloc) override {
Linker.Allocs.push_back(std::move(Alloc));
++Linker.MM->ObjectsLoaded;
}
};
JITLinkLinker::JITLinkLinker(BinaryContext &BC,
std::unique_ptr<ExecutableFileMemoryManager> MM)
: BC(BC), MM(std::move(MM)) {}
JITLinkLinker::~JITLinkLinker() { cantFail(MM->deallocate(std::move(Allocs))); }
void JITLinkLinker::loadObject(MemoryBufferRef Obj,
SectionsMapper MapSections) {
auto LG = jitlink::createLinkGraphFromObject(Obj);
if (auto E = LG.takeError()) {
errs() << "BOLT-ERROR: JITLink failed: " << E << '\n';
exit(1);
}
auto Ctx = std::make_unique<Context>(*this, MapSections);
jitlink::link(std::move(*LG), std::move(Ctx));
}
std::optional<uint64_t> JITLinkLinker::lookupSymbol(StringRef Name) const {
auto It = Symtab.find(Name.data());
if (It == Symtab.end())
return std::nullopt;
return It->second;
}
SmallVector<jitlink::Block *, 2>
JITLinkLinker::orderedBlocks(const jitlink::Section &Section) {
SmallVector<jitlink::Block *, 2> Blocks(Section.blocks());
llvm::sort(Blocks, [](const auto *LHS, const auto *RHS) {
return LHS->getAddress() < RHS->getAddress();
});
return Blocks;
}
size_t JITLinkLinker::sectionSize(const jitlink::Section &Section) {
size_t Size = 0;
for (const auto *Block : orderedBlocks(Section)) {
Size = jitlink::alignToBlock(Size, *Block);
Size += Block->getSize();
}
return Size;
}
} // namespace bolt
} // namespace llvm

104
bolt/lib/Rewrite/MachORewriteInstance.cpp
View File

@ -17,6 +17,7 @@
#include "bolt/Profile/DataReader.h"
#include "bolt/Rewrite/BinaryPassManager.h"
#include "bolt/Rewrite/ExecutableFileMemoryManager.h"
#include "bolt/Rewrite/JITLinkLinker.h"
#include "bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h"
#include "bolt/Utils/Utils.h"
#include "llvm/MC/MCAsmLayout.h"
@ -386,7 +387,8 @@ void MachORewriteInstance::runOptimizationPasses() {
Manager.runPasses();
}
void MachORewriteInstance::mapInstrumentationSection(StringRef SectionName) {
void MachORewriteInstance::mapInstrumentationSection(
StringRef SectionName, BOLTLinker::SectionMapper MapSection) {
if (!opts::Instrument)
return;
ErrorOr<BinarySection &> Section = BC->getUniqueSectionByName(SectionName);
@ -396,11 +398,11 @@ void MachORewriteInstance::mapInstrumentationSection(StringRef SectionName) {
}
if (!Section->hasValidSectionID())
return;
RTDyld->reassignSectionAddress(Section->getSectionID(),
Section->getAddress());
MapSection(*Section, Section->getAddress());
}
void MachORewriteInstance::mapCodeSections() {
void MachORewriteInstance::mapCodeSections(
BOLTLinker::SectionMapper MapSection) {
for (BinaryFunction *Function : BC->getAllBinaryFunctions()) {
if (!Function->isEmitted())
continue;
@ -417,8 +419,7 @@ void MachORewriteInstance::mapCodeSections() {
LLVM_DEBUG(dbgs() << "BOLT: mapping 0x"
<< Twine::utohexstr(FuncSection->getAllocAddress()) << " to 0x"
<< Twine::utohexstr(Function->getOutputAddress()) << '\n');
RTDyld->reassignSectionAddress(FuncSection->getSectionID(),
Function->getOutputAddress());
MapSection(*FuncSection, Function->getOutputAddress());
Function->setImageAddress(FuncSection->getAllocAddress());
Function->setImageSize(FuncSection->getOutputSize());
}
@ -439,7 +440,7 @@ void MachORewriteInstance::mapCodeSections() {
assert(FuncSection && "cannot find section for function");
Addr = llvm::alignTo(Addr, 4);
FuncSection->setOutputAddress(Addr);
RTDyld->reassignSectionAddress(FuncSection->getSectionID(), Addr);
MapSection(*FuncSection, Addr);
Function->setFileOffset(Addr - BOLT->getAddress() +
BOLT->getInputFileOffset());
Function->setImageAddress(FuncSection->getAllocAddress());
@ -450,34 +451,6 @@ void MachORewriteInstance::mapCodeSections() {
}
}
namespace {
class BOLTSymbolResolver : public LegacyJITSymbolResolver {
BinaryContext &BC;
public:
BOLTSymbolResolver(BinaryContext &BC) : BC(BC) {}
JITSymbol findSymbolInLogicalDylib(const std::string &Name) override {
return JITSymbol(nullptr);
}
JITSymbol findSymbol(const std::string &Name) override {
LLVM_DEBUG(dbgs() << "BOLT: looking for " << Name << "\n");
if (BinaryData *I = BC.getBinaryDataByName(Name)) {
const uint64_t Address = I->isMoved() && !I->isJumpTable()
? I->getOutputAddress()
: I->getAddress();
LLVM_DEBUG(dbgs() << "Resolved to address 0x" << Twine::utohexstr(Address)
<< "\n");
return JITSymbol(Address, JITSymbolFlags());
}
LLVM_DEBUG(dbgs() << "Resolved to address 0x0\n");
return JITSymbol(nullptr);
}
};
} // end anonymous namespace
void MachORewriteInstance::emitAndLink() {
std::error_code EC;
std::unique_ptr<::llvm::ToolOutputFile> TempOut =
@ -503,45 +476,38 @@ void MachORewriteInstance::emitAndLink() {
"error creating in-memory object");
assert(Obj && "createObjectFile cannot return nullptr");
BOLTSymbolResolver Resolver = BOLTSymbolResolver(*BC);
MCAsmLayout FinalLayout(
static_cast<MCObjectStreamer *>(Streamer.get())->getAssembler());
BC->EFMM.reset(new ExecutableFileMemoryManager(*BC, /*AllowStubs*/ false));
BC->EFMM->setOrgSecPrefix(getOrgSecPrefix());
BC->EFMM->setNewSecPrefix(getNewSecPrefix());
auto EFMM = std::make_unique<ExecutableFileMemoryManager>(*BC);
EFMM->setNewSecPrefix(getNewSecPrefix());
EFMM->setOrgSecPrefix(getOrgSecPrefix());
RTDyld.reset(new decltype(RTDyld)::element_type(*BC->EFMM, Resolver));
RTDyld->setProcessAllSections(true);
RTDyld->loadObject(*Obj);
if (RTDyld->hasError()) {
outs() << "BOLT-ERROR: RTDyld failed.\n";
exit(1);
}
Linker = std::make_unique<JITLinkLinker>(*BC, std::move(EFMM));
Linker->loadObject(ObjectMemBuffer->getMemBufferRef(),
[this](auto MapSection) {
// Assign addresses to all sections. If key corresponds
// to the object created by ourselves, call our regular
// mapping function. If we are loading additional objects
// as part of runtime libraries for instrumentation,
// treat them as extra sections.
mapCodeSections(MapSection);
mapInstrumentationSection("__counters", MapSection);
mapInstrumentationSection("__tables", MapSection);
});
// Assign addresses to all sections. If key corresponds to the object
// created by ourselves, call our regular mapping function. If we are
// loading additional objects as part of runtime libraries for
// instrumentation, treat them as extra sections.
mapCodeSections();
mapInstrumentationSection("__counters");
mapInstrumentationSection("__tables");
RTDyld->finalizeWithMemoryManagerLocking();
// TODO: Refactor addRuntimeLibSections to work properly on Mach-O
// and use it here.
//FIXME! Put this in RtLibrary->link
// mapInstrumentationSection("I__setup");
// mapInstrumentationSection("I__fini");
// mapInstrumentationSection("I__data");
// mapInstrumentationSection("I__text");
// mapInstrumentationSection("I__cstring");
// mapInstrumentationSection("I__literal16");
// if (auto *RtLibrary = BC->getRuntimeLibrary()) {
// RtLibrary->link(*BC, ToolPath, *ES, *OLT);
// }
// TODO: Refactor addRuntimeLibSections to work properly on Mach-O
// and use it here.
// if (auto *RtLibrary = BC->getRuntimeLibrary()) {
// RtLibrary->link(*BC, ToolPath, *Linker, [this](auto MapSection) {
// mapInstrumentationSection("I__setup", MapSection);
// mapInstrumentationSection("I__fini", MapSection);
// mapInstrumentationSection("I__data", MapSection);
// mapInstrumentationSection("I__text", MapSection);
// mapInstrumentationSection("I__cstring", MapSection);
// mapInstrumentationSection("I__literal16", MapSection);
// });
// }
}
void MachORewriteInstance::writeInstrumentationSection(StringRef SectionName,

145
bolt/lib/Rewrite/RewriteInstance.cpp
View File

@ -26,6 +26,7 @@
#include "bolt/Rewrite/BinaryPassManager.h"
#include "bolt/Rewrite/DWARFRewriter.h"
#include "bolt/Rewrite/ExecutableFileMemoryManager.h"
#include "bolt/Rewrite/JITLinkLinker.h"
#include "bolt/RuntimeLibs/HugifyRuntimeLibrary.h"
#include "bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h"
#include "bolt/Utils/CommandLineOpts.h"
@ -33,7 +34,6 @@
#include "llvm/ADT/STLExtras.h"
#include "llvm/DebugInfo/DWARF/DWARFContext.h"
#include "llvm/DebugInfo/DWARF/DWARFDebugFrame.h"
#include "llvm/ExecutionEngine/RuntimeDyld.h"
#include "llvm/IR/Function.h"
#include "llvm/MC/MCAsmBackend.h"
#include "llvm/MC/MCAsmInfo.h"
@ -3299,69 +3299,6 @@ void RewriteInstance::runOptimizationPasses() {
BinaryFunctionPassManager::runAllPasses(*BC);
}
namespace {
class BOLTSymbolResolver : public JITSymbolResolver {
BinaryContext &BC;
public:
BOLTSymbolResolver(BinaryContext &BC) : BC(BC) {}
// We are responsible for all symbols
Expected<LookupSet> getResponsibilitySet(const LookupSet &Symbols) override {
return Symbols;
}
// Some of our symbols may resolve to zero and this should not be an error
bool allowsZeroSymbols() override { return true; }
/// Resolves the address of each symbol requested
void lookup(const LookupSet &Symbols,
OnResolvedFunction OnResolved) override {
JITSymbolResolver::LookupResult AllResults;
if (BC.EFMM->ObjectsLoaded) {
for (const StringRef &Symbol : Symbols) {
std::string SymName = Symbol.str();
LLVM_DEBUG(dbgs() << "BOLT: looking for " << SymName << "\n");
// Resolve to a PLT entry if possible
if (const BinaryData *I = BC.getPLTBinaryDataByName(SymName)) {
AllResults[Symbol] =
JITEvaluatedSymbol(I->getAddress(), JITSymbolFlags());
continue;
}
OnResolved(make_error<StringError>(
"Symbol not found required by runtime: " + Symbol,
inconvertibleErrorCode()));
return;
}
OnResolved(std::move(AllResults));
return;
}
for (const StringRef &Symbol : Symbols) {
std::string SymName = Symbol.str();
LLVM_DEBUG(dbgs() << "BOLT: looking for " << SymName << "\n");
if (BinaryData *I = BC.getBinaryDataByName(SymName)) {
uint64_t Address = I->isMoved() && !I->isJumpTable()
? I->getOutputAddress()
: I->getAddress();
LLVM_DEBUG(dbgs() << "Resolved to address 0x"
<< Twine::utohexstr(Address) << "\n");
AllResults[Symbol] = JITEvaluatedSymbol(Address, JITSymbolFlags());
continue;
}
LLVM_DEBUG(dbgs() << "Resolved to address 0x0\n");
AllResults[Symbol] = JITEvaluatedSymbol(0, JITSymbolFlags());
}
OnResolved(std::move(AllResults));
}
};
} // anonymous namespace
void RewriteInstance::preregisterSections() {
// Preregister sections before emission to set their order in the output.
const unsigned ROFlags = BinarySection::getFlags(/*IsReadOnly*/ true,
@ -3435,39 +3372,18 @@ void RewriteInstance::emitAndLink() {
// Get output object as ObjectFile.
std::unique_ptr<MemoryBuffer> ObjectMemBuffer =
MemoryBuffer::getMemBuffer(BOS->str(), "in-memory object file", false);
std::unique_ptr<object::ObjectFile> Obj = cantFail(
object::ObjectFile::createObjectFile(ObjectMemBuffer->getMemBufferRef()),
"error creating in-memory object");
BOLTSymbolResolver Resolver = BOLTSymbolResolver(*BC);
auto EFMM = std::make_unique<ExecutableFileMemoryManager>(*BC);
EFMM->setNewSecPrefix(getNewSecPrefix());
EFMM->setOrgSecPrefix(getOrgSecPrefix());
Linker = std::make_unique<JITLinkLinker>(*BC, std::move(EFMM));
Linker->loadObject(ObjectMemBuffer->getMemBufferRef(),
[this](auto MapSection) { mapFileSections(MapSection); });
MCAsmLayout FinalLayout(
static_cast<MCObjectStreamer *>(Streamer.get())->getAssembler());
// Disable stubs because RuntimeDyld may try to increase the size of
// sections accounting for stubs. We need those sections to match the
// same size seen in the input binary, in case this section is a copy
// of the original one seen in the binary.
BC->EFMM.reset(new ExecutableFileMemoryManager(*BC, /*AllowStubs=*/false));
BC->EFMM->setNewSecPrefix(getNewSecPrefix());
BC->EFMM->setOrgSecPrefix(getOrgSecPrefix());
RTDyld.reset(new decltype(RTDyld)::element_type(*BC->EFMM, Resolver));
RTDyld->setProcessAllSections(false);
RTDyld->loadObject(*Obj);
// Assign addresses to all sections. If key corresponds to the object
// created by ourselves, call our regular mapping function. If we are
// loading additional objects as part of runtime libraries for
// instrumentation, treat them as extra sections.
mapFileSections(*RTDyld);
RTDyld->finalizeWithMemoryManagerLocking();
if (RTDyld->hasError()) {
errs() << "BOLT-ERROR: RTDyld failed: " << RTDyld->getErrorString() << "\n";
exit(1);
}
// Update output addresses based on the new section map and
// layout. Only do this for the object created by ourselves.
updateOutputValues(FinalLayout);
@ -3476,9 +3392,9 @@ void RewriteInstance::emitAndLink() {
DebugInfoRewriter->updateLineTableOffsets(FinalLayout);
if (RuntimeLibrary *RtLibrary = BC->getRuntimeLibrary())
RtLibrary->link(*BC, ToolPath, *RTDyld, [this](RuntimeDyld &R) {
RtLibrary->link(*BC, ToolPath, *Linker, [this](auto MapSection) {
// Map newly registered sections.
this->mapAllocatableSections(*RTDyld);
this->mapAllocatableSections(MapSection);
});
// Once the code is emitted, we can rename function sections to actual
@ -3874,7 +3790,7 @@ void RewriteInstance::updateLKMarkers() {
<< '\n';
}
void RewriteInstance::mapFileSections(RuntimeDyld &RTDyld) {
void RewriteInstance::mapFileSections(BOLTLinker::SectionMapper MapSection) {
BC->deregisterUnusedSections();
// If no new .eh_frame was written, remove relocated original .eh_frame.
@ -3884,19 +3800,18 @@ void RewriteInstance::mapFileSections(RuntimeDyld &RTDyld) {
BinarySection *NewEHFrameSection =
getSection(getNewSecPrefix() + getEHFrameSectionName());
if (!NewEHFrameSection || !NewEHFrameSection->isFinalized()) {
// RTDyld will still have to process relocations for the section, hence
// JITLink will still have to process relocations for the section, hence
// we need to assign it the address that wouldn't result in relocation
// processing failure.
RTDyld.reassignSectionAddress(RelocatedEHFrameSection->getSectionID(),
NextAvailableAddress);
MapSection(*RelocatedEHFrameSection, NextAvailableAddress);
BC->deregisterSection(*RelocatedEHFrameSection);
}
}
mapCodeSections(RTDyld);
mapCodeSections(MapSection);
// Map the rest of the sections.
mapAllocatableSections(RTDyld);
mapAllocatableSections(MapSection);
}
std::vector<BinarySection *> RewriteInstance::getCodeSections() {
@ -3925,7 +3840,7 @@ std::vector<BinarySection *> RewriteInstance::getCodeSections() {
return CodeSections;
}
void RewriteInstance::mapCodeSections(RuntimeDyld &RTDyld) {
void RewriteInstance::mapCodeSections(BOLTLinker::SectionMapper MapSection) {
if (BC->HasRelocations) {
// Map sections for functions with pre-assigned addresses.
for (BinaryFunction *InjectedFunction : BC->getInjectedBinaryFunctions()) {
@ -3937,8 +3852,7 @@ void RewriteInstance::mapCodeSections(RuntimeDyld &RTDyld) {
InjectedFunction->getCodeSection();
assert(FunctionSection && "function should have section");
FunctionSection->setOutputAddress(OutputAddress);
RTDyld.reassignSectionAddress(FunctionSection->getSectionID(),
OutputAddress);
MapSection(*FunctionSection, OutputAddress);
InjectedFunction->setImageAddress(FunctionSection->getAllocAddress());
InjectedFunction->setImageSize(FunctionSection->getOutputSize());
}
@ -4014,8 +3928,7 @@ void RewriteInstance::mapCodeSections(RuntimeDyld &RTDyld) {
dbgs() << "BOLT: mapping " << Section->getName() << " at 0x"
<< Twine::utohexstr(Section->getAllocAddress()) << " to 0x"
<< Twine::utohexstr(Section->getOutputAddress()) << '\n');
RTDyld.reassignSectionAddress(Section->getSectionID(),
Section->getOutputAddress());
MapSection(*Section, Section->getOutputAddress());
Section->setOutputFileOffset(
getFileOffsetForAddress(Section->getOutputAddress()));
}
@ -4045,8 +3958,7 @@ void RewriteInstance::mapCodeSections(RuntimeDyld &RTDyld) {
<< Twine::utohexstr(FuncSection->getAllocAddress())
<< " to 0x" << Twine::utohexstr(Function.getAddress())
<< '\n');
RTDyld.reassignSectionAddress(FuncSection->getSectionID(),
Function.getAddress());
MapSection(*FuncSection, Function.getAddress());
Function.setImageAddress(FuncSection->getAllocAddress());
Function.setImageSize(FuncSection->getOutputSize());
if (Function.getImageSize() > Function.getMaxSize()) {
@ -4064,7 +3976,7 @@ void RewriteInstance::mapCodeSections(RuntimeDyld &RTDyld) {
LLVM_DEBUG(dbgs() << "BOLT-DEBUG: mapping JT " << Section.getName()
<< " to 0x" << Twine::utohexstr(JT->getAddress())
<< '\n');
RTDyld.reassignSectionAddress(Section.getSectionID(), JT->getAddress());
MapSection(Section, JT->getAddress());
}
}
@ -4100,7 +4012,7 @@ void RewriteInstance::mapCodeSections(RuntimeDyld &RTDyld) {
dbgs() << formatv(
"BOLT: mapping cold fragment {0:x+} to {1:x+} with size {2:x+}\n",
FF.getImageAddress(), FF.getAddress(), FF.getImageSize()));
RTDyld.reassignSectionAddress(ColdSection->getSectionID(), FF.getAddress());
MapSection(*ColdSection, FF.getAddress());
if (TooLarge)
BC->deregisterSection(*ColdSection);
@ -4130,7 +4042,8 @@ void RewriteInstance::mapCodeSections(RuntimeDyld &RTDyld) {
}
}
void RewriteInstance::mapAllocatableSections(RuntimeDyld &RTDyld) {
void RewriteInstance::mapAllocatableSections(
BOLTLinker::SectionMapper MapSection) {
// Allocate read-only sections first, then writable sections.
enum : uint8_t { ST_READONLY, ST_READWRITE };
for (uint8_t SType = ST_READONLY; SType <= ST_READWRITE; ++SType) {
@ -4164,8 +4077,7 @@ void RewriteInstance::mapAllocatableSections(RuntimeDyld &RTDyld) {
});
Section.setOutputAddress(Section.getAddress());
Section.setOutputFileOffset(Section.getInputFileOffset());
RTDyld.reassignSectionAddress(Section.getSectionID(),
Section.getAddress());
MapSection(Section, Section.getAddress());
} else {
NextAvailableAddress =
alignTo(NextAvailableAddress, Section.getAlignment());
@ -4178,8 +4090,7 @@ void RewriteInstance::mapAllocatableSections(RuntimeDyld &RTDyld) {
<< '\n';
});
RTDyld.reassignSectionAddress(Section.getSectionID(),
NextAvailableAddress);
MapSection(Section, NextAvailableAddress);
Section.setOutputAddress(NextAvailableAddress);
Section.setOutputFileOffset(
getFileOffsetForAddress(NextAvailableAddress));
@ -5992,9 +5903,9 @@ void RewriteInstance::writeEHFrameHeader() {
}
uint64_t RewriteInstance::getNewValueForSymbol(const StringRef Name) {
uint64_t Value = RTDyld->getSymbol(Name).getAddress();
if (Value != 0)
return Value;
auto Value = Linker->lookupSymbol(Name);
if (Value)
return *Value;
// Return the original value if we haven't emitted the symbol.
BinaryData *BD = BC->getBinaryDataByName(Name);

2
bolt/lib/RuntimeLibs/CMakeLists.txt
View File

@ -1,8 +1,8 @@
set(LLVM_LINK_COMPONENTS
BinaryFormat
JITLink
MC
Object
RuntimeDyld
Support
)

17
bolt/lib/RuntimeLibs/HugifyRuntimeLibrary.cpp
View File

@ -12,7 +12,7 @@
#include "bolt/RuntimeLibs/HugifyRuntimeLibrary.h"
#include "bolt/Core/BinaryFunction.h"
#include "llvm/ExecutionEngine/RuntimeDyld.h"
#include "bolt/Core/Linker.h"
#include "llvm/MC/MCStreamer.h"
#include "llvm/Support/Alignment.h"
#include "llvm/Support/CommandLine.h"
@ -61,20 +61,15 @@ void HugifyRuntimeLibrary::adjustCommandLineOptions(
}
void HugifyRuntimeLibrary::link(BinaryContext &BC, StringRef ToolPath,
RuntimeDyld &RTDyld,
std::function<void(RuntimeDyld &)> OnLoad) {
BOLTLinker &Linker,
BOLTLinker::SectionsMapper MapSections) {
std::string LibPath = getLibPath(ToolPath, opts::RuntimeHugifyLib);
loadLibrary(LibPath, RTDyld);
OnLoad(RTDyld);
RTDyld.finalizeWithMemoryManagerLocking();
if (RTDyld.hasError()) {
outs() << "BOLT-ERROR: RTDyld failed: " << RTDyld.getErrorString() << "\n";
exit(1);
}
loadLibrary(LibPath, Linker, MapSections);
assert(!RuntimeStartAddress &&
"We don't currently support linking multiple runtime libraries");
RuntimeStartAddress = RTDyld.getSymbol("__bolt_hugify_self").getAddress();
RuntimeStartAddress = Linker.lookupSymbol("__bolt_hugify_self").value_or(0);
if (!RuntimeStartAddress) {
errs() << "BOLT-ERROR: instrumentation library does not define "
"__bolt_hugify_self: "

20
bolt/lib/RuntimeLibs/InstrumentationRuntimeLibrary.cpp
View File

@ -13,8 +13,8 @@
#include "bolt/RuntimeLibs/InstrumentationRuntimeLibrary.h"
#include "bolt/Core/BinaryFunction.h"
#include "bolt/Core/JumpTable.h"
#include "bolt/Core/Linker.h"
#include "bolt/Utils/CommandLineOpts.h"
#include "llvm/ExecutionEngine/RuntimeDyld.h"
#include "llvm/MC/MCStreamer.h"
#include "llvm/Support/Alignment.h"
#include "llvm/Support/CommandLine.h"
@ -191,28 +191,22 @@ void InstrumentationRuntimeLibrary::emitBinary(BinaryContext &BC,
}
void InstrumentationRuntimeLibrary::link(
BinaryContext &BC, StringRef ToolPath, RuntimeDyld &RTDyld,
std::function<void(RuntimeDyld &)> OnLoad) {
BinaryContext &BC, StringRef ToolPath, BOLTLinker &Linker,
BOLTLinker::SectionsMapper MapSections) {
std::string LibPath = getLibPath(ToolPath, opts::RuntimeInstrumentationLib);
loadLibrary(LibPath, RTDyld);
OnLoad(RTDyld);
RTDyld.finalizeWithMemoryManagerLocking();
if (RTDyld.hasError()) {
outs() << "BOLT-ERROR: RTDyld failed: " << RTDyld.getErrorString() << "\n";
exit(1);
}
loadLibrary(LibPath, Linker, MapSections);
if (BC.isMachO())
return;
RuntimeFiniAddress = RTDyld.getSymbol("__bolt_instr_fini").getAddress();
RuntimeFiniAddress = Linker.lookupSymbol("__bolt_instr_fini").value_or(0);
if (!RuntimeFiniAddress) {
errs() << "BOLT-ERROR: instrumentation library does not define "
"__bolt_instr_fini: "
<< LibPath << "\n";
exit(1);
}
RuntimeStartAddress = RTDyld.getSymbol("__bolt_instr_start").getAddress();
RuntimeStartAddress = Linker.lookupSymbol("__bolt_instr_start").value_or(0);
if (!RuntimeStartAddress) {
errs() << "BOLT-ERROR: instrumentation library does not define "
"__bolt_instr_start: "
@ -224,7 +218,7 @@ void InstrumentationRuntimeLibrary::link(
<< Twine::utohexstr(RuntimeFiniAddress) << "\n";
outs() << "BOLT-INFO: clear procedure is 0x"
<< Twine::utohexstr(
RTDyld.getSymbol("__bolt_instr_clear_counters").getAddress())
Linker.lookupSymbol("__bolt_instr_clear_counters").value_or(0))
<< "\n";
emitTablesAsELFNote(BC);

10
bolt/lib/RuntimeLibs/RuntimeLibrary.cpp
View File

@ -11,11 +11,12 @@
//===----------------------------------------------------------------------===//
#include "bolt/RuntimeLibs/RuntimeLibrary.h"
#include "bolt/Core/Linker.h"
#include "bolt/RuntimeLibs/RuntimeLibraryVariables.inc"
#include "bolt/Utils/Utils.h"
#include "llvm/BinaryFormat/Magic.h"
#include "llvm/ExecutionEngine/RuntimeDyld.h"
#include "llvm/Object/Archive.h"
#include "llvm/Object/ObjectFile.h"
#include "llvm/Support/Path.h"
#define DEBUG_TYPE "bolt-rtlib"
@ -44,7 +45,8 @@ std::string RuntimeLibrary::getLibPath(StringRef ToolPath,
return std::string(LibPath.str());
}
void RuntimeLibrary::loadLibrary(StringRef LibPath, RuntimeDyld &RTDyld) {
void RuntimeLibrary::loadLibrary(StringRef LibPath, BOLTLinker &Linker,
BOLTLinker::SectionsMapper MapSections) {
ErrorOr<std::unique_ptr<MemoryBuffer>> MaybeBuf =
MemoryBuffer::getFile(LibPath, false, false);
check_error(MaybeBuf.getError(), LibPath);
@ -57,7 +59,7 @@ void RuntimeLibrary::loadLibrary(StringRef LibPath, RuntimeDyld &RTDyld) {
for (const object::Archive::Child &C : Archive.children(Err)) {
std::unique_ptr<object::Binary> Bin = cantFail(C.getAsBinary());
if (object::ObjectFile *Obj = dyn_cast<object::ObjectFile>(&*Bin))
RTDyld.loadObject(*Obj);
Linker.loadObject(Obj->getMemoryBufferRef(), MapSections);
}
check_error(std::move(Err), B->getBufferIdentifier());
} else if (Magic == file_magic::elf_relocatable ||
@ -65,7 +67,7 @@ void RuntimeLibrary::loadLibrary(StringRef LibPath, RuntimeDyld &RTDyld) {
std::unique_ptr<object::ObjectFile> Obj = cantFail(
object::ObjectFile::createObjectFile(B.get()->getMemBufferRef()),
"error creating in-memory object");
RTDyld.loadObject(*Obj);
Linker.loadObject(Obj->getMemoryBufferRef(), MapSections);
} else {
errs() << "BOLT-ERROR: unrecognized library format: " << LibPath << "\n";
exit(1);