Nick Terrell ac02e6927c [LLD][ELF] Support --[no-]mmap-output-file with F_no_mmap ... Summary: Add a flag `F_no_mmap` to `FileOutputBuffer` to support `--[no-]mmap-output-file` in ELF LLD. LLD currently explicitly ignores this flag for compatibility with GNU ld and gold. We need this flag to speed up link time for large binaries in certain scenarios. When we link some of our larger binaries we find that LLD takes 50+ GB of memory, which causes memory pressure. The memory pressure causes the VM to flush dirty pages of the output file to disk. This is normally okay, since we should be flushing cold pages. However, when using BtrFS with compression we need to write 128KB at a time when we flush a page. If any page in that 128KB block is written again, then it must be flushed a second time, and so on. Since LLD doesn't write sequentially this causes write amplification. The same 128KB block will end up being flushed multiple times, causing the linker to many times more IO than necessary. We've observed 3-5x faster builds with -no-mmap-output-file when we hit this scenario. The bad scenario only applies to compressed filesystems, which group together multiple pages into a single compressed block. I've tested BtrFS, but the problem will be present for any compressed filesystem on Linux, since it is caused by the VM. Silently ignoring --no-mmap-output-file caused a silent regression when we switched from gold to lld. We pass --no-mmap-output-file to fix this edge case, but since lld silently ignored the flag we didn't realize it wasn't being respected. Benchmark building a 9 GB binary that exposes this edge case. I linked 3 times with --mmap-output-file and 3 times with --no-mmap-output-file and took the average. The machine has 24 cores @ 2.4 GHz, 112 GB of RAM, BtrFS mounted with -compress-force=zstd, and an 80% full disk. | Mode | Time | |---------|-------| | mmap | 894 s | | no mmap | 126 s | When compression is disabled, BtrFS performs just as well with and without mmap on this benchmark. I was unable to reproduce the regression with any binaries in lld-speed-test. Reviewed By: ruiu, MaskRay Differential Revision: https://reviews.llvm.org/D69294		2019-10-29 15:49:08 -07:00
..
DynamicLibrary	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
AlignmentTest.cpp	Fix https://bugs.llvm.org/show_bug.cgi?id=43791	2019-10-28 10:40:35 +01:00
AlignOfTest.cpp	[NFC] Remove LLVM_ALIGNAS	2019-07-31 03:22:08 +00:00
AllocatorTest.cpp	[Alignment][NFC] Move and type functions from MathExtras to Alignment	2019-10-14 13:14:34 +00:00
AnnotationsTest.cpp	[Support] Try to unbreak windows buildbot	2019-04-25 11:57:40 +00:00
ARMAttributeParser.cpp	[ARM] add target arch definitions for 8.1-M and MVE	2019-05-30 12:57:04 +00:00
ArrayRecyclerTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
BinaryStreamTest.cpp	[llvm] Migrate llvm::make_unique to std::make_unique	2019-08-15 15:54:37 +00:00
BlockFrequencyTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
BranchProbabilityTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
CachePruningTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
Casting.cpp	[llvm] Migrate llvm::make_unique to std::make_unique	2019-08-15 15:54:37 +00:00
CheckedArithmeticTest.cpp	Add checkMulAdd helper function to CheckedArithmetic	2018-06-13 18:32:02 +00:00
Chrono.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
CMakeLists.txt	Revert "Disable exit-on-SIGPIPE in lldb"	2019-10-24 13:19:49 -07:00
CommandLineTest.cpp	Revert Recommit "[CommandLine] Remove OptionCategory and SubCommand caches from the Option class."	2019-07-11 08:54:28 +00:00
CompressionTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ConvertUTFTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
CrashRecoveryTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
CRCTest.cpp	Unify the two CRC implementations	2019-10-09 09:06:30 +00:00
DataExtractorTest.cpp	Add error handling to the DataExtractor class	2019-08-27 11:24:08 +00:00
DebugCounterTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
DebugTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
DJBTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
EndianStreamTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
EndianTest.cpp	[Support/Endian] Add support for endian-specific enums	2019-03-11 09:06:18 +00:00
ErrnoTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ErrorOrTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ErrorTest.cpp	[Error] Make llvm::cantFail include the original error messages	2019-10-17 21:54:15 +00:00
FileCheckTest.cpp	[NFC] [FileCheck] Reapply fix init of objects in unit tests	2019-10-04 15:47:23 +00:00
FileCollectorTest.cpp	[FileCollector] test: disable the directory entry collection checks on windows	2019-07-30 00:27:33 +00:00
FileOutputBufferTest.cpp	[LLD][ELF] Support --[no-]mmap-output-file with F_no_mmap	2019-10-29 15:49:08 -07:00
FileUtilitiesTest.cpp	[Support] Add writeFileAtomically() to FileUtilities	2019-09-05 18:10:29 +00:00
formatted_raw_ostream_test.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
FormatVariadicTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
GlobPatternTest.cpp	Reapply r375051: [support] GlobPattern: add support for \ and [!...], and allow ] in more places	2019-10-17 18:09:05 +00:00
Host.cpp	[llvm] Migrate llvm::make_unique to std::make_unique	2019-08-15 15:54:37 +00:00
ItaniumManglingCanonicalizerTest.cpp	Update more file headers across all of the LLVM projects in the monorepo	2019-01-19 10:56:40 +00:00
JSONTest.cpp	[Support] Add JSON streaming output API, faster where the heavy value types aren't needed.	2019-04-25 12:51:42 +00:00
KnownBitsTest.cpp	[KnownBits] Add computeForAddCarry()	2019-04-12 18:18:08 +00:00
LEB128Test.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
LineIteratorTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
LockFileManagerTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ManagedStatic.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
MatchersTest.cpp	[Support] Add a GTest matcher for Optional<T>	2019-04-25 09:03:32 +00:00
MathExtrasTest.cpp	[Alignment][NFC] Support compile time constants	2019-10-14 09:04:15 +00:00
MD5Test.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
MemoryBufferTest.cpp	[Support] Improve readNativeFile(Slice) interface	2019-08-22 08:13:30 +00:00
MemoryTest.cpp	[Support] Renamed member 'Size' to 'AllocatedSize' in MemoryBlock and OwningMemoryBlock.	2019-05-20 20:53:05 +00:00
NativeFormatTests.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ParallelTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
Path.cpp	[Support] Improve readNativeFile(Slice) interface	2019-08-22 08:13:30 +00:00
ProcessTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ProgramTest.cpp	Revert "raw_ostream: add operator<< overload for std::error_code"	2019-08-14 13:59:04 +00:00
raw_ostream_test.cpp	Rename F_{None,Text,Append} to OF_{None,Text,Append}. NFC	2019-08-05 05:43:48 +00:00
raw_pwrite_stream_test.cpp	Revert "raw_ostream: add operator<< overload for std::error_code"	2019-08-14 13:59:04 +00:00
raw_sha1_ostream_test.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
RegexTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ReplaceFileTest.cpp	Revert "raw_ostream: add operator<< overload for std::error_code"	2019-08-14 13:59:04 +00:00
ReverseIterationTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ScaledNumberTest.cpp	Replace llvm::isPodLike<...> by llvm::is_trivially_copyable<...>	2019-01-20 21:19:56 +00:00
SourceMgrTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
SpecialCaseListTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
StringPool.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
SwapByteOrderTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
SymbolRemappingReaderTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
TargetParserTest.cpp	[AArch64] Add support for Transactional Memory Extension (TME)	2019-07-31 12:52:17 +00:00
TarWriterTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
TaskQueueTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
Threading.cpp	Reland "[Support] Add a way to run a function on a detached thread""	2019-10-23 15:51:44 +02:00
ThreadLocalTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
ThreadPool.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
TimerTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
TrailingObjectsTest.cpp	[Alignment][NFC] Move and type functions from MathExtras to Alignment	2019-10-14 13:14:34 +00:00
TrigramIndexTest.cpp	[llvm] Migrate llvm::make_unique to std::make_unique	2019-08-15 15:54:37 +00:00
TypeNameTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
TypeTraitsTest.cpp	Fix compilation warnings when compiling with GCC 7.3	2019-05-06 13:41:54 +00:00
UnicodeTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
VersionTupleTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
VirtualFileSystemTest.cpp	Fix an unused variable introduced in rL374955 / rG21703543.	2019-10-16 00:52:00 +00:00
xxhashTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00
YAMLIOTest.cpp	[llvm] Migrate llvm::make_unique to std::make_unique	2019-08-15 15:54:37 +00:00
YAMLParserTest.cpp	Update the file headers across all of the LLVM projects in the monorepo	2019-01-19 08:50:56 +00:00