diff --git a/llvm/docs/FatLTO.rst b/llvm/docs/FatLTO.rst new file mode 100644 index 000000000000..5773a30b7ae4 --- /dev/null +++ b/llvm/docs/FatLTO.rst @@ -0,0 +1,77 @@ +=================== +FatLTO +=================== +.. contents:: + :local: + :depth: 2 + +.. toctree:: + :maxdepth: 1 + +Introduction +============ + +FatLTO objects are a special type of `fat object file +`_ that contain LTO compatible IR in +addition to generated object code, instead of containing object code for +multiple target architectures. This allows users to defer the choice of whether +to use LTO or not to link-time, and has been a feature available in other +compilers, like `GCC +`_, for some time. + +Under FatLTO the compiler can emit standard object files which contain both the +machine code in the ``.text`` section and LLVM bitcode in the ``.llvm.lto`` +section. + +Overview +======== + +Within LLVM, FatLTO is supported by choosing the ``FatLTODefaultPipeline``. +This pipeline will: + +#) Clone the IR module. +#) Run the pre-link (Thin)LTO pipeline using the cloned module. +#) Embed the pre-link bitcode in a special ``.llvm.lto`` section. +#) Optimize the unmodified copy of the module using the normal compilation pipeline. +#) Emit the object file, including the new ``.llvm.lto`` section. + +.. NOTE + + At the time of writing, we conservatively run independent pipelines to + generate the bitcode section and the object code, which happen to be + identical to those used outside of FatLTO. This results in compiled + artifacts that are identical to those produced by the default and (Thin)LTO + pipelines. However, this is not a guarantee, and we reserve the right to + change this at any time. Explicitly, users should not rely on the produced + bitcode or object code to mach their non-LTO counterparts precisely. They + will exhibit similar performance characteristics, but may not be bit-for-bit + the same. + +Internally, the ``.llvm.lto`` section is created by running the +``EmbedBitcodePass`` at the start of the ``PerModuleDefaultPipeline``. This +pass is responsible for cloning and optimizing the module with the appropriate +LTO pipeline and emitting the ``.llvm.lto`` section. Afterwards, the +``PerModuleDefaultPipeline`` runs normally and the compiler can emit the fat +object file. + +Limitations +=========== + +Linkers +------- + +Currently, using LTO with LLVM fat lto objects is supported by LLD and by the +GNU linkers via :doc:`GoldPlugin`. This may change in the future, but +extending support to other linkers isn't planned for now. + +.. NOTE + For standard linking the fat object files should be usable by any + linker capable of using ELF objects, since the ``.llvm.lto`` section is + marked ``SHF_EXLUDE``. + +Supported File Formats +---------------------- + +The current implementation only supports ELF files. At time of writing, it is +unclear if it will be useful to support other object file formats like ``COFF`` +or ``Mach-O``. diff --git a/llvm/docs/ReleaseNotes.rst b/llvm/docs/ReleaseNotes.rst index 6efc991ff509..e086c235c929 100644 --- a/llvm/docs/ReleaseNotes.rst +++ b/llvm/docs/ReleaseNotes.rst @@ -80,6 +80,12 @@ Changes to LLVM infrastructure * InstructionSimplify APIs now require instructions be inserted into a parent function. +* A new FatLTO pipeline was added to support generating object files that have + both machine code and LTO compatible bitcode. See the :doc:`FatLTO` + documentation and the original + `RFC `_ + for more details. + Changes to building LLVM ------------------------ diff --git a/llvm/docs/UserGuides.rst b/llvm/docs/UserGuides.rst index 517a23e95920..d7335911abe1 100644 --- a/llvm/docs/UserGuides.rst +++ b/llvm/docs/UserGuides.rst @@ -32,6 +32,7 @@ intermediate LLVM representation. DebuggingJITedCode DirectXUsage Docker + FatLTO ExtendingLLVM GoldPlugin HowToBuildOnARM diff --git a/llvm/include/llvm/Passes/PassBuilder.h b/llvm/include/llvm/Passes/PassBuilder.h index 585c335cf572..f31c32102e69 100644 --- a/llvm/include/llvm/Passes/PassBuilder.h +++ b/llvm/include/llvm/Passes/PassBuilder.h @@ -234,6 +234,18 @@ public: ModulePassManager buildPerModuleDefaultPipeline(OptimizationLevel Level, bool LTOPreLink = false); + /// Build a fat object default optimization pipeline. + /// + /// This builds a pipeline that runs the LTO/ThinLTO pre-link pipeline, and + /// emits a section containing the pre-link bitcode along side the object code + /// generated by running the PerModuleDefaultPipeline, used when compiling + /// without LTO. It clones the module and runs the LTO/non-LTO pipelines + /// separately to avoid any inconsistencies with an ad-hoc pipeline that tries + /// to approximate the PerModuleDefaultPipeline from the pre-link LTO + /// pipelines. + ModulePassManager buildFatLTODefaultPipeline(OptimizationLevel Level, + bool ThinLTO, bool EmitSummary); + /// Build a pre-link, ThinLTO-targeting default optimization pipeline to /// a pass manager. /// diff --git a/llvm/include/llvm/Transforms/IPO/EmbedBitcodePass.h b/llvm/include/llvm/Transforms/IPO/EmbedBitcodePass.h new file mode 100644 index 000000000000..f323c61483fd --- /dev/null +++ b/llvm/include/llvm/Transforms/IPO/EmbedBitcodePass.h @@ -0,0 +1,58 @@ +//===-- EmbedBitcodePass.h - Embeds bitcode into global ---------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +/// \file +/// +/// This file provides a pass which clones the current module and runs the +/// provided pass pipeline on the clone. The optimized module is stored into a +/// global variable in the `.llvm.lto` section. Primarily, this pass is used +/// to support the FatLTO pipeline, but could be used to generate a bitcode +/// section for any arbitrary pass pipeline without changing the current module. +/// +//===----------------------------------------------------------------------===// +// +#ifndef LLVM_TRANSFORMS_IPO_EMBEDBITCODEPASS_H +#define LLVM_TRANSFORMS_IPO_EMBEDBITCODEPASS_H + +#include "llvm/IR/PassManager.h" + +namespace llvm { +class Module; +class ModulePass; +class Pass; + +struct EmbedBitcodeOptions { + EmbedBitcodeOptions() : EmbedBitcodeOptions(false, false) {} + EmbedBitcodeOptions(bool IsThinLTO, bool EmitLTOSummary) + : IsThinLTO(IsThinLTO), EmitLTOSummary(EmitLTOSummary) {} + bool IsThinLTO; + bool EmitLTOSummary; +}; + +/// Pass embeds a copy of the module optimized with the provided pass pipeline +/// into a global variable. +class EmbedBitcodePass : public PassInfoMixin { + bool IsThinLTO; + bool EmitLTOSummary; + ModulePassManager MPM; + +public: + EmbedBitcodePass(EmbedBitcodeOptions Opts) + : EmbedBitcodePass(Opts.IsThinLTO, Opts.EmitLTOSummary, + ModulePassManager()) {} + EmbedBitcodePass(bool IsThinLTO, bool EmitLTOSummary, ModulePassManager &&MPM) + : IsThinLTO(IsThinLTO), EmitLTOSummary(EmitLTOSummary), + MPM(std::move(MPM)) {} + + PreservedAnalyses run(Module &M, ModuleAnalysisManager &); + + static bool isRequired() { return true; } +}; + +} // end namespace llvm. + +#endif diff --git a/llvm/lib/Object/ObjectFile.cpp b/llvm/lib/Object/ObjectFile.cpp index c953a5692cbb..0820187f32e1 100644 --- a/llvm/lib/Object/ObjectFile.cpp +++ b/llvm/lib/Object/ObjectFile.cpp @@ -79,7 +79,7 @@ uint32_t ObjectFile::getSymbolAlignment(DataRefImpl DRI) const { return 0; } bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const { Expected NameOrErr = getSectionName(Sec); if (NameOrErr) - return *NameOrErr == ".llvmbc"; + return *NameOrErr == ".llvmbc" || *NameOrErr == ".llvm.lto"; consumeError(NameOrErr.takeError()); return false; } diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp index a4972c871348..d7d7db8c828e 100644 --- a/llvm/lib/Passes/PassBuilder.cpp +++ b/llvm/lib/Passes/PassBuilder.cpp @@ -102,6 +102,7 @@ #include "llvm/Transforms/IPO/CrossDSOCFI.h" #include "llvm/Transforms/IPO/DeadArgumentElimination.h" #include "llvm/Transforms/IPO/ElimAvailExtern.h" +#include "llvm/Transforms/IPO/EmbedBitcodePass.h" #include "llvm/Transforms/IPO/ForceFunctionAttrs.h" #include "llvm/Transforms/IPO/FunctionAttrs.h" #include "llvm/Transforms/IPO/FunctionImport.h" @@ -738,6 +739,26 @@ Expected parseHWASanPassOptions(StringRef Params) { return Result; } +Expected parseEmbedBitcodePassOptions(StringRef Params) { + EmbedBitcodeOptions Result; + while (!Params.empty()) { + StringRef ParamName; + std::tie(ParamName, Params) = Params.split(';'); + + if (ParamName == "thinlto") { + Result.IsThinLTO = true; + } else if (ParamName == "emit-summary") { + Result.EmitLTOSummary = true; + } else { + return make_error( + formatv("invalid EmbedBitcode pass parameter '{0}' ", ParamName) + .str(), + inconvertibleErrorCode()); + } + } + return Result; +} + Expected parseMSanPassOptions(StringRef Params) { MemorySanitizerOptions Result; while (!Params.empty()) { diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp b/llvm/lib/Passes/PassBuilderPipelines.cpp index 34d3b9d7467e..515ef1bb3501 100644 --- a/llvm/lib/Passes/PassBuilderPipelines.cpp +++ b/llvm/lib/Passes/PassBuilderPipelines.cpp @@ -46,6 +46,7 @@ #include "llvm/Transforms/IPO/CrossDSOCFI.h" #include "llvm/Transforms/IPO/DeadArgumentElimination.h" #include "llvm/Transforms/IPO/ElimAvailExtern.h" +#include "llvm/Transforms/IPO/EmbedBitcodePass.h" #include "llvm/Transforms/IPO/ForceFunctionAttrs.h" #include "llvm/Transforms/IPO/FunctionAttrs.h" #include "llvm/Transforms/IPO/GlobalDCE.h" @@ -1464,7 +1465,18 @@ PassBuilder::buildPerModuleDefaultPipeline(OptimizationLevel Level, if (LTOPreLink) addRequiredLTOPreLinkPasses(MPM); + return MPM; +} +ModulePassManager +PassBuilder::buildFatLTODefaultPipeline(OptimizationLevel Level, bool ThinLTO, + bool EmitSummary) { + ModulePassManager MPM; + MPM.addPass(EmbedBitcodePass(ThinLTO, EmitSummary, + ThinLTO + ? buildThinLTOPreLinkDefaultPipeline(Level) + : buildLTOPreLinkDefaultPipeline(Level))); + MPM.addPass(buildPerModuleDefaultPipeline(Level)); return MPM; } diff --git a/llvm/lib/Passes/PassRegistry.def b/llvm/lib/Passes/PassRegistry.def index 7a2035cf07b3..1879dea7de16 100644 --- a/llvm/lib/Passes/PassRegistry.def +++ b/llvm/lib/Passes/PassRegistry.def @@ -170,6 +170,13 @@ MODULE_PASS_WITH_PARAMS("ipsccp", }, parseIPSCCPOptions, "no-func-spec;func-spec") +MODULE_PASS_WITH_PARAMS("embed-bitcode", + "EmbedBitcodePass", + [](EmbedBitcodeOptions Opts) { + return EmbedBitcodePass(Opts); + }, + parseEmbedBitcodePassOptions, + "thinlto;emit-summary") #undef MODULE_PASS_WITH_PARAMS #ifndef CGSCC_ANALYSIS diff --git a/llvm/lib/Transforms/IPO/CMakeLists.txt b/llvm/lib/Transforms/IPO/CMakeLists.txt index e03aff0f65d7..034f1587ae8d 100644 --- a/llvm/lib/Transforms/IPO/CMakeLists.txt +++ b/llvm/lib/Transforms/IPO/CMakeLists.txt @@ -11,6 +11,7 @@ add_llvm_component_library(LLVMipo CrossDSOCFI.cpp DeadArgumentElimination.cpp ElimAvailExtern.cpp + EmbedBitcodePass.cpp ExtractGV.cpp ForceFunctionAttrs.cpp FunctionAttrs.cpp diff --git a/llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp b/llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp new file mode 100644 index 000000000000..8b0cefeaf711 --- /dev/null +++ b/llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp @@ -0,0 +1,51 @@ +//===- EmbedBitcodePass.cpp - Pass that embeds the bitcode into a global---===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#include "llvm/Transforms/IPO/EmbedBitcodePass.h" +#include "llvm/Bitcode/BitcodeWriter.h" +#include "llvm/Bitcode/BitcodeWriterPass.h" +#include "llvm/IR/Constants.h" +#include "llvm/IR/PassManager.h" +#include "llvm/InitializePasses.h" +#include "llvm/Pass.h" +#include "llvm/Passes/PassBuilder.h" +#include "llvm/Support/ErrorHandling.h" +#include "llvm/Support/MemoryBuffer.h" +#include "llvm/TargetParser/Triple.h" +#include "llvm/Transforms/IPO/ThinLTOBitcodeWriter.h" +#include "llvm/Transforms/Utils/Cloning.h" +#include "llvm/Transforms/Utils/ModuleUtils.h" + +using namespace llvm; + +PreservedAnalyses EmbedBitcodePass::run(Module &M, ModuleAnalysisManager &AM) { + if (M.getGlobalVariable("llvm.embedded.module", /*AllowInternal=*/true)) + report_fatal_error("Can only embed the module once", + /*gen_crash_diag=*/false); + + Triple T(M.getTargetTriple()); + if (T.getObjectFormat() != Triple::ELF) + report_fatal_error( + "EmbedBitcode pass currently only supports ELF object format", + /*gen_crash_diag=*/false); + + std::unique_ptr NewModule = CloneModule(M); + MPM.run(*NewModule, AM); + + std::string Data; + raw_string_ostream OS(Data); + if (IsThinLTO) + ThinLTOBitcodeWriterPass(OS, /*ThinLinkOS=*/nullptr).run(*NewModule, AM); + else + BitcodeWriterPass(OS, /*ShouldPreserveUseListOrder=*/false, EmitLTOSummary) + .run(*NewModule, AM); + + embedBufferInModule(M, MemoryBufferRef(Data, "ModuleData"), ".llvm.lto"); + + return PreservedAnalyses::all(); +} diff --git a/llvm/test/Transforms/EmbedBitcode/embed-multiple.ll b/llvm/test/Transforms/EmbedBitcode/embed-multiple.ll new file mode 100644 index 000000000000..32a7fbe2fd8c --- /dev/null +++ b/llvm/test/Transforms/EmbedBitcode/embed-multiple.ll @@ -0,0 +1,6 @@ +; RUN: not opt --mtriple x86_64-unknown-linux-gnu < %s -passes=embed-bitcode -S 2>&1 | FileCheck %s + +@a = global i32 1 +@llvm.embedded.module = private constant [4 x i8] c"BC\C0\DE" + +; CHECK: LLVM ERROR: Can only embed the module once diff --git a/llvm/test/Transforms/EmbedBitcode/embed-unsupported-object-format.ll b/llvm/test/Transforms/EmbedBitcode/embed-unsupported-object-format.ll new file mode 100644 index 000000000000..ca04f496c8a4 --- /dev/null +++ b/llvm/test/Transforms/EmbedBitcode/embed-unsupported-object-format.ll @@ -0,0 +1,5 @@ +; RUN: not opt --mtriple powerpc64-unknown-aix < %s -passes=embed-bitcode -S 2>&1 | FileCheck %s + +@a = global i32 1 + +; CHECK: LLVM ERROR: EmbedBitcode pass currently only supports ELF object format diff --git a/llvm/test/Transforms/EmbedBitcode/embed.ll b/llvm/test/Transforms/EmbedBitcode/embed.ll new file mode 100644 index 000000000000..a54a6442ba0f --- /dev/null +++ b/llvm/test/Transforms/EmbedBitcode/embed.ll @@ -0,0 +1,43 @@ +; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode" -S | FileCheck %s +; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode" -S | FileCheck %s +; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode" -S | FileCheck %s +; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode" -S | FileCheck %s + +@a = global i32 1 + +; CHECK: @a = global i32 1 +;; Make sure the module is in the correct section. +; CHECK: @llvm.embedded.object = private constant {{.*}}, section ".llvm.lto", align 1 + +;; Ensure that the metadata is in llvm.compiler.used. +; CHECK: @llvm.compiler.used = appending global [1 x ptr] [ptr @llvm.embedded.object], section "llvm.metadata" + +;; Make sure the metadata correlates to the .llvm.lto section. +; CHECK: !llvm.embedded.objects = !{!1} +; CHECK: !0 = !{} +; CHECK: !{ptr @llvm.embedded.object, !".llvm.lto"} + + +;; Ensure that the .llvm.lto section has SHT_EXCLUDE set. +; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode" -S \ +; RUN: | llc --mtriple x86_64-unknown-linux-gnu -filetype=obj \ +; RUN: | llvm-readobj - --sections --elf-output-style=JSON --pretty-print \ +; RUN: | FileCheck %s --check-prefix=EXCLUDE + +; EXCLUDE: "Name": ".llvm.lto", +; EXCLUDE-NEXT: "Value": 7 +; EXCLUDE-NEXT: }, +; EXCLUDE-NEXT: "Type": { +; EXCLUDE-NEXT: "Name": "SHT_PROGBITS", +; EXCLUDE-NEXT: "Value": 1 +; EXCLUDE-NEXT: }, +; EXCLUDE-NEXT: "Flags": { +; EXCLUDE-NEXT: "Value": 2147483648, +; EXCLUDE-NEXT: "Flags": [ +; EXCLUDE-NEXT: { +; EXCLUDE-NEXT: "Name": "SHF_EXCLUDE", +; EXCLUDE-NEXT: "Value": 2147483648 +; EXCLUDE-NEXT: } +; EXCLUDE-NEXT: ] +; EXCLUDE-NEXT: }, +