Commit Graph

61 Commits

Author SHA1 Message Date
Mehdi Amini
eb79e6e85d ThinLTO: Make aliases explicit in the summary
To be able to work accurately on the reference graph when taking
decision about internalizing, promoting, renaming, etc. We need
to have the alias information explicit.

Differential Revision: http://reviews.llvm.org/D18836

From: Mehdi Amini <mehdi.amini@apple.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@266517 91177308-0d34-0410-b5e6-96231b3b80d8
2016-04-16 06:56:44 +00:00
Mehdi Amini
387bdb41b0 Revert "Make aliases explicit in the summary"
Inadvertently commited...

This reverts commit e618ec9378.

From: Mehdi Amini <mehdi.amini@apple.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@266215 91177308-0d34-0410-b5e6-96231b3b80d8
2016-04-13 17:20:07 +00:00
Mehdi Amini
e618ec9378 Make aliases explicit in the summary
Summary:
To be able to work accurately on the reference graph when taking decision
about internalizing, promoting, renaming, etc. We need to have the alias
information explicit.

Reviewers: tejohnson

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D18836

From: Mehdi Amini <mehdi.amini@apple.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@266214 91177308-0d34-0410-b5e6-96231b3b80d8
2016-04-13 17:18:42 +00:00
Mehdi Amini
d2f4701e4a Add a module Hash in the bitcode and the combined index, implementing a kind of "build-id"
This is intended to be used for ThinLTO incremental build.

Differential Revision: http://reviews.llvm.org/D18213

This is a recommit of r265095 after fixing the Windows issues.

From: Mehdi Amini <mehdi.amini@apple.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@265111 91177308-0d34-0410-b5e6-96231b3b80d8
2016-04-01 05:33:11 +00:00
Mehdi Amini
1fc660ff51 Revert "Add support for computing SHA1 in LLVM"
This reverts commit r265096, r265095, and r265094.
Windows build is broken, and the validation does not pass.

From: Mehdi Amini <mehdi.amini@apple.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@265102 91177308-0d34-0410-b5e6-96231b3b80d8
2016-04-01 03:03:21 +00:00
Mehdi Amini
433ac20b17 Add missing test for the "Module hash in bitcode" added in r265095
From: Mehdi Amini <mehdi.amini@apple.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@265096 91177308-0d34-0410-b5e6-96231b3b80d8
2016-04-01 01:37:52 +00:00
Duncan P. N. Exon Smith
fbf768a4d3 Support: Implement StreamingMemoryObject::getPointer
The implementation is fairly obvious.  This is preparation for using
some blobs in bitcode.

For clarity (and perhaps future-proofing?), I moved the call to
JumpToBit in BitstreamCursor::readRecord ahead of calling
MemoryObject::getPointer, since JumpToBit can theoretically (a) read
bytes, which (b) invalidates the blob pointer.

This isn't strictly necessary the two memory objects we have:

  - The return of RawMemoryObject::getPointer is valid until the memory
    object is destroyed.

  - StreamingMemoryObject::getPointer is valid until the next chunk is
    read from the stream.  Since the JumpToBit call is only going ahead
    to a word boundary, we'll never load another chunk.

However, reordering makes it clear by inspection that the blob returned
by BitstreamCursor::readRecord will be valid.

I added some tests for StreamingMemoryObject::getPointer and
BitstreamCursor::readRecord.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@264549 91177308-0d34-0410-b5e6-96231b3b80d8
2016-03-27 23:00:59 +00:00
Duncan P. N. Exon Smith
d49b92d802 Revert "Bitcode: Collect all MDString records into a single blob"
This reverts commit r264409 since it failed to bootstrap:
http://lab.llvm.org:8080/green/job/clang-stage2-configure-Rlto_build/8302/

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@264410 91177308-0d34-0410-b5e6-96231b3b80d8
2016-03-25 15:22:27 +00:00
Duncan P. N. Exon Smith
d7084fcebd Bitcode: Collect all MDString records into a single blob
Optimize output of MDStrings in bitcode.  This emits them in big blocks
(currently 1024) in a pair of records:
  - BULK_STRING_SIZES: the sizes of the strings in the block, and
  - BULK_STRING_DATA: a single blob, which is the concatenation of all
    the strings.

Inspired by Mehdi's similar patch, http://reviews.llvm.org/D18342, this
should (a) slightly reduce bitcode size, since there is less record
overhead, and (b) greatly improve reading speed, since blobs are super
cheap to deserialize.

I needed to add support for blobs to streaming input to get the test
suite passing.
  - StreamingMemoryObject::getPointer reads ahead and returns the
    address of the blob.
  - To avoid a possible reallocation of StreamingMemoryObject::Bytes,
    BitstreamCursor::readRecord needs to move the call to JumpToEnd
    forward so that getPointer is the last bitstream operation.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@264409 91177308-0d34-0410-b5e6-96231b3b80d8
2016-03-25 14:40:18 +00:00
Teresa Johnson
dc6615addb [ThinLTO] Support for reference graph in per-module and combined summary.
Summary:
This patch adds support for including a full reference graph including
call graph edges and other GV references in the summary.

The reference graph edges can be used to make importing decisions
without materializing any source modules, can be used in the plugin
to make file staging decisions for distributed build systems, and is
expected to have other uses.

The call graph edges are recorded in each function summary in the
bitcode via a list of <CalleeValueIds, StaticCount> tuples when no PGO
data exists, or <CalleeValueId, StaticCount, ProfileCount> pairs when
there is PGO, where the ValueId can be mapped to the function GUID via
the ValueSymbolTable. In the function index in memory, the call graph
edges reference the target via the CalleeGUID instead of the
CalleeValueId.

The reference graph edges are recorded in each summary record with a
list of referenced value IDs, which can be mapped to value GUID via the
ValueSymbolTable.

Addtionally, a new summary record type is added to record references
from global variable initializers. A number of bitcode records and data
structures have been renamed to reflect the newly expanded scope of the
summary beyond functions. More cleanup will follow.

Reviewers: joker.eph, davidxl

Subscribers: joker.eph, llvm-commits

Differential Revision: http://reviews.llvm.org/D17212

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@263275 91177308-0d34-0410-b5e6-96231b3b80d8
2016-03-11 18:52:24 +00:00
Teresa Johnson
0060160bcf Restore "[ThinLTO] Use MD5 hash in function index." with fix
This restores commit r260408, along with a fix for a bot failure.

The bot failure was caused by dereferencing a unique_ptr in the same
call instruction parameter list where it was passed via std::move.
Apparently due to luck this was not exposed when I built the compiler
with clang, only with gcc.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@260442 91177308-0d34-0410-b5e6-96231b3b80d8
2016-02-10 21:55:02 +00:00
Teresa Johnson
bc947a0d8d Revert "[ThinLTO] Use MD5 hash in function index." due to bot failure
This reverts commit r260408. Bot failure that I need to investigate.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@260412 91177308-0d34-0410-b5e6-96231b3b80d8
2016-02-10 19:11:15 +00:00
Teresa Johnson
f38049c501 [ThinLTO] Use MD5 hash in function index.
Summary:
This patch uses the lower 64-bits of the MD5 hash of a function name as
a GUID in the function index, instead of storing function names. Any
local functions are first given a global name by prepending the original
source file name. This is the same naming scheme and GUID used by PGO in
the indexed profile format.

This change has a couple of benefits. The primary benefit is size
reduction in the combined index file, for example 483.xalancbmk's
combined index file was reduced by around 70%. It should also result in
memory savings for the index file in memory, as the in-memory map is
also indexed by the hash instead of the string.

Second, this enables integration with indirect call promotion, since the
indirect call profile targets are recorded using the same global naming
convention and hash. This will enable the function importer to easily
locate function summaries for indirect call profile targets to enable
their import and subsequent promotion.

The original source file name is recorded in the bitcode in a new
module-level record for use in the ThinLTO backend pipeline.

Reviewers: davidxl, joker.eph

Subscribers: llvm-commits, joker.eph

Differential Revision: http://reviews.llvm.org/D17028

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@260408 91177308-0d34-0410-b5e6-96231b3b80d8
2016-02-10 18:57:54 +00:00
Filipe Cabecinhas
c3c89b471b Error out when faced with value names containing '\0'
Bug found with afl-fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@252048 91177308-0d34-0410-b5e6-96231b3b80d8
2015-11-04 14:53:36 +00:00
Filipe Cabecinhas
32a2349a79 Don't assert if materializing before seeing any function bodies
This assert was reachable from user input. A minimized test case (no
FUNCTION_BLOCK_ID record) is attached.

Bug found with afl-fuzz

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@251910 91177308-0d34-0410-b5e6-96231b3b80d8
2015-11-03 13:48:26 +00:00
Filipe Cabecinhas
5f6abbdbfd Revert "Don't assert if materializing before seeing any function bodies"
This reverts r251667 since it broke the bots.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@251671 91177308-0d34-0410-b5e6-96231b3b80d8
2015-10-30 00:00:58 +00:00
Filipe Cabecinhas
b0ed17be63 Don't assert if materializing before seeing any function bodies
This assert was reachable from user input. A minimized test case (no
FUNCTION_BLOCK_ID record) is attached.

Bug found with afl-fuzz

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@251667 91177308-0d34-0410-b5e6-96231b3b80d8
2015-10-29 23:37:28 +00:00
Filipe Cabecinhas
fdb28cdd8b Make sure the CastInst is valid before trying to create it
Bug found with afl-fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@249396 91177308-0d34-0410-b5e6-96231b3b80d8
2015-10-06 12:37:54 +00:00
Derek Schuff
f96e27f256 Fix testing for end of stream in bitstream reader.
This fixes a bug found while working on the bitcode reader. In
particular, the method BitstreamReader::AtEndOfStream doesn't always
behave correctly when processing a data streamer. The method
fillCurWord doesn't properly set CurWord/BitsInCurWord if the data
streamer was already at eof, but GetBytes had not yet set the
ObjectSize field of the streaming memory object.

This patch fixes this problem, and provides a test to show that
this problem has been fixed.

Patch by Karl Schimpf.

Differential Revision: http://reviews.llvm.org/D11391

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@243890 91177308-0d34-0410-b5e6-96231b3b80d8
2015-08-03 18:01:50 +00:00
Rafael Espindola
9afb3a2589 Fix for PR23310: llvm-dis crashes when trying to upgrade an intrinsic.
When trying to upgrade @llvm.x86.sse2.psrl.dq while parsing a module,
BitcodeReader adds the function to its worklist twice, resulting in a
crash when accessing it the second time.

This patch replaces the worklist vector by a map.

Patch by Philip Pfaffe.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@241281 91177308-0d34-0410-b5e6-96231b3b80d8
2015-07-02 16:22:40 +00:00
Rafael Espindola
0c650627ca Improve handling of end of file in the bitcode reader.
Before this patch the bitcode reader would read a module from a file
that contained in order:

* Any number of non MODULE_BLOCK sub blocks.
* One MODULE_BLOCK
* Any number of non MODULE_BLOCK sub blocks.
* 4 '\n' characters to handle OS X's ranlib.

Since we support lazy reading of modules, any information that is relevant
for the module has to be in the MODULE_BLOCK or before it. We don't gain
anything from checking what is after.

This patch then changes the reader to stop once the MODULE_BLOCK has been
successfully parsed.

This avoids the ugly special case for .bc files in an archive and makes it
easier to embed bitcode files.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@239845 91177308-0d34-0410-b5e6-96231b3b80d8
2015-06-16 20:03:39 +00:00
Rafael Espindola
0021a54ae5 Add a test for padded bitcode files.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@239829 91177308-0d34-0410-b5e6-96231b3b80d8
2015-06-16 16:36:15 +00:00
Filipe Cabecinhas
a3a2cab504 [BitcodeReader] Diagnose type mismatches with aliases
Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238895 91177308-0d34-0410-b5e6-96231b3b80d8
2015-06-03 01:30:13 +00:00
Filipe Cabecinhas
1a2a0ee8a0 [Bitcode] Minimize the test to not conflict with others
Source for the test:
@bloom = global <3 x i32> <i32 0, i32 1, i32 42>

Plus bit twiddling to set the vector numelts to 0 (in the bc file).

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238894 91177308-0d34-0410-b5e6-96231b3b80d8
2015-06-03 01:30:08 +00:00
Filipe Cabecinhas
287f68d654 [BitcodeReader] Check vector size before trying to create a VectorType
Bug found with AFL fuzz

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238891 91177308-0d34-0410-b5e6-96231b3b80d8
2015-06-03 00:05:30 +00:00
Filipe Cabecinhas
3b821159da [BitcodeReader] Change an assert to a call to a call to Error()
It's reachable from user input.

Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238633 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-30 00:17:20 +00:00
Filipe Cabecinhas
1c0b496636 [BitcodeReader] Change assert to report_fatal_error
It can be triggered by user input.

Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238272 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-27 01:05:40 +00:00
Filipe Cabecinhas
316f9e05ce [BitstreamReader] Make sure the Array operand type is an encoding
Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238269 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-27 00:48:43 +00:00
Filipe Cabecinhas
c756772b8d [BitcodeReader] Make sure abbrev records have at least one operand (record code)
Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238265 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-26 23:52:21 +00:00
Filipe Cabecinhas
1c74d4768a [BitcodeReader] Sanity check on Comdat ID
Shouldn't be an assert, since user input can trigger it.

Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@238261 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-26 23:00:56 +00:00
Filipe Cabecinhas
27ca64f3d3 Change a reachable unreachable to a fatal error.
Summary:
Also tagged a FIXME comment, and added information about why it breaks.

Bug found using AFL fuzz.

Reviewers: rafael, craig.topper

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D9729

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@237709 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-19 18:18:10 +00:00
Filipe Cabecinhas
bf27e11e02 [BitcodeReader] Error out if we read an invalid function argument type
Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@237650 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-19 01:21:06 +00:00
Filipe Cabecinhas
c3ccd67d72 [BitcodeReader] It's a malformed block if CodeLenWidth is too big
Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@237646 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-19 00:34:17 +00:00
Filipe Cabecinhas
b2bf278152 [BitcodeReader] Make sure the type of the inserted value matches the type of the aggregate at those indices
Bug found with AFL-fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@237628 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-18 22:27:11 +00:00
Filipe Cabecinhas
70a2c7260a Extract the load/store type verification to a separate function.
Summary:
Added isLoadableOrStorableType to PointerType.

We were doing some checks in some places, occasionally assert()ing instead
of telling the caller. With this patch, I'm putting all type checking in
the same place for load/store type instructions, and verifying the same
thing every time.

I also added a check for load/store of a function type.

Applied extracted check to Load, Store, and Cmpxcg.

I don't have exhaustive tests for all of these, but all Error() calls in
TypeCheckLoadStoreInst are being tested (in invalid.test).

Reviewers: dblaikie, rafael

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D9785

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@237619 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-18 21:48:55 +00:00
Filipe Cabecinhas
ddf553bb8d [BitcodeReader] Don't allow INSERTVAL/EXTRACTVAL with 0 indices
This would trigger an assertion later.

Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@237494 91177308-0d34-0410-b5e6-96231b3b80d8
2015-05-16 00:33:12 +00:00
Filipe Cabecinhas
deedba2a36 Don't overflow GCTable
Summary: Bug found with AFL fuzz.

Reviewers: rafael, dexonsmith

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D9361

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@236200 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-30 04:09:41 +00:00
Filipe Cabecinhas
8b2199e2a2 Make sure Op->getType() is a PointerType before we cast<> it.
Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@236193 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-30 01:13:31 +00:00
Filipe Cabecinhas
a607be94ca Make sure we don't resize(0) when we get a fwdref with Idx == UINT_MAX
Make it an error instead.

Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@236190 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-30 00:52:42 +00:00
Filipe Cabecinhas
99ebc9e004 Check that we have a valid PointerType element type before calling get()
Same as r236073 but for PointerType.

Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@236079 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-29 02:27:28 +00:00
Filipe Cabecinhas
7b30f32d3d Turn an assert into report_fatal_error since it's reachable based on user input
Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@236076 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-29 01:58:31 +00:00
Filipe Cabecinhas
3b4a565b8a Make sure that isValidElementType(Type) before calling {Array,Struct}Type::get(Type)
Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@236073 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-29 01:27:01 +00:00
Filipe Cabecinhas
7acbf56112 Relax an assert when there's a type mismatch in forward references
Summary:
We don't seem to need to assert here, since this function's callers expect
to get a nullptr on error. This way we don't assert on user input.

Bug found with AFL fuzz.

Reviewers: rafael

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D9308

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@236027 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-28 20:18:47 +00:00
David Blaikie
c5b1f0a49d [opaque pointer type] bitcode: add explicit callee type to invoke instructions
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@235735 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-24 18:06:06 +00:00
Filipe Cabecinhas
45a69610a7 [BitcodeReader] Fix asserts when we read a non-vector type for insert/extract/shuffle
Added some additional checking for vector types + tests.

Bug found with AFL fuzz.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@235710 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-24 11:30:15 +00:00
Filipe Cabecinhas
0236022390 Be more strict about the operand for the array type in BitcodeReader
Summary: Bug found with AFL fuzz.

Reviewers: rafael

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D9016

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@235596 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-23 13:38:21 +00:00
Filipe Cabecinhas
81f9bd3e19 Verify sizes when trying to read a BitcodeAbbrevOp
Summary:
Make sure the abbrev operands are valid and that we can read/skip them
afterwards.

Bug found with AFL fuzz.

Reviewers: rafael

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D9030

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@235595 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-23 13:25:35 +00:00
Filipe Cabecinhas
e16cac587a Have more strict type checks when creating BinOp nodes in BitcodeReader
Summary: Bug found with AFL.

Reviewers: rafael, bkramer

Subscribers: llvm-commits

Differential Revision: http://reviews.llvm.org/D9015

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@235489 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-22 09:06:21 +00:00
David Blaikie
5023f151a7 [opaque pointer type] Explicit pointee type for call instruction
Use an extra bit in the CCInfo to flag the newer version of the
instructiont hat includes the type explicitly.

Tested the newer error cases I added, but didn't add tests for the finer
granularity improvements to existing error paths.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@235160 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-17 06:40:14 +00:00
Filipe Cabecinhas
f72ee4c275 Revert "Verify sizes when trying to read a VBR"
This reverts r234984 since it seems to break some bots (most of them
seemed arm*-selfhost).

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@234998 91177308-0d34-0410-b5e6-96231b3b80d8
2015-04-15 11:10:17 +00:00