Summary:
Implement guard widening in LLVM. Description from GuardWidening.cpp:
The semantics of the `@llvm.experimental.guard` intrinsic lets LLVM
transform it so that it fails more often that it did before the
transform. This optimization is called "widening" and can be used hoist
and common runtime checks in situations like these:
```
%cmp0 = 7 u< Length
call @llvm.experimental.guard(i1 %cmp0) [ "deopt"(...) ]
call @unknown_side_effects()
%cmp1 = 9 u< Length
call @llvm.experimental.guard(i1 %cmp1) [ "deopt"(...) ]
...
```
to
```
%cmp0 = 9 u< Length
call @llvm.experimental.guard(i1 %cmp0) [ "deopt"(...) ]
call @unknown_side_effects()
...
```
If `%cmp0` is false, `@llvm.experimental.guard` will "deoptimize" back
to a generic implementation of the same function, which will have the
correct semantics from that point onward. It is always _legal_ to
deoptimize (so replacing `%cmp0` with false is "correct"), though it may
not always be profitable to do so.
NB! This pass is a work in progress. It hasn't been tuned to be
"production ready" yet. It is known to have quadriatic running time and
will not scale to large numbers of guards
Reviewers: reames, atrick, bogner, apilipenko, nlewycky
Subscribers: mcrosier, llvm-commits
Differential Revision: http://reviews.llvm.org/D20143
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269997 91177308-0d34-0410-b5e6-96231b3b80d8
Previously, we didn't add their and their operands cost, which could've
resulted in unrolling loops for no actual benefit.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269985 91177308-0d34-0410-b5e6-96231b3b80d8
In truncateToMinimalBitwidths() we were RAUW'ing an instruction then erasing it. However, that intruction could be cached in the map we're iterating over. The first check is "I->use_empty()" which in most cases would return true, as the (deleted) object was RAUW'd first so would have zero use count. However in some cases the object could have been polluted or written over and this wouldn't be the case. Also it makes valgrind, asan and traditionalists who don't like their compiler to crash sad.
No testcase as there are no externally visible symptoms apart from a crash if the stars align.
Fixes PR26509.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269908 91177308-0d34-0410-b5e6-96231b3b80d8
This bug was introduced in r269728 and is the likely cause of many stage 2 ubsan bot failures.
I'll add a test in a follow-up commit assuming this fixes things properly.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269797 91177308-0d34-0410-b5e6-96231b3b80d8
Guards are expected to basically never fail. Reflect this in the branch
probabilities in their lowered form.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269791 91177308-0d34-0410-b5e6-96231b3b80d8
This is assertion is no longer necessary since we never record
constants in the live set anyway. (They are never recorded in
the initial live set, and constant bases are removed near line 2119)
Differential Revision: http://reviews.llvm.org/D20293
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269764 91177308-0d34-0410-b5e6-96231b3b80d8
Fix a bug introduced with rL269426 :
[InstCombine] canonicalize* LE/GE vector integer comparisons to LT/GT (PR26701, PR26819)
We were assuming that a ConstantDataVector / ConstantVector / ConstantAggregateZero operand of
an ICMP was composed of ConstantInt elements, but it might have ConstantExpr or UndefValue
elements. Handle those appropriately.
Also, refactor this function to join the scalar and vector paths and eliminate the switches.
Differential Revision: http://reviews.llvm.org/D20289
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269728 91177308-0d34-0410-b5e6-96231b3b80d8
Transition InstrProf and Coverage over to the stricter Error/Expected
interface.
Changes since the initial commit:
- Address undefined-var-template warning.
- Fix error message printing in llvm-profdata.
- Check errors in loadTestingFormat() + annotateAllFunctions().
- Defer error handling in InstrProfIterator to InstrProfReader.
Differential Revision: http://reviews.llvm.org/D19901
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269694 91177308-0d34-0410-b5e6-96231b3b80d8
The selection of the vectorization factor currently doesn't consider
interleaved accesses. The vectorization factor is based on the maximum safe
dependence distance computed by LAA. However, for loops with interleaved
groups, we should instead base the vectorization factor on the maximum safe
dependence distance divided by the maximum interleave factor of all the
interleaved groups. Interleaved accesses not in a group will be scalarized.
Differential Revision: http://reviews.llvm.org/D20241
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269659 91177308-0d34-0410-b5e6-96231b3b80d8
TargetLibraryInfoWrapperPass is a dependency of
SCCP but it's not listed as such. Chandler pointed
out this is an easy mistake to make which only
surfaces in weird crashes with some flag combinations.
This code will go away anyway at some point in the
future, but as long as it's (still) exercised, try
to make it correct.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269589 91177308-0d34-0410-b5e6-96231b3b80d8
This reverts commit r269491. It triggers warnings with Clang, breaking
builds for -Werror users including several build bots.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269547 91177308-0d34-0410-b5e6-96231b3b80d8
Transition InstrProf and Coverage over to the stricter Error/Expected
interface.
Changes since the initial commit:
- Fix error message printing in llvm-profdata.
- Check errors in loadTestingFormat() + annotateAllFunctions().
- Defer error handling in InstrProfIterator to InstrProfReader.
Differential Revision: http://reviews.llvm.org/D19901
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269491 91177308-0d34-0410-b5e6-96231b3b80d8
Currently there is no reasonable way to control the warnings in the 'use' phase
of the IRPGO pass. This is problematic because the output can be somewhat
spammy. This patch adds some flags which allow us to optionally disable these
warnings. The current upstream behavior will remain the default.
Patch by Jake VanAdrighem (jvanadrighem@gmail.com)
Differential Revision: http://reviews.llvm.org/D20195
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269437 91177308-0d34-0410-b5e6-96231b3b80d8
Summary: This change fix the bug in isProfitableToUseMemset() where MaxIntSize shoule be in byte, not bit.
Reviewers: arsenm, joker.eph, mcrosier
Subscribers: mcrosier, llvm-commits
Differential Revision: http://reviews.llvm.org/D20176
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269433 91177308-0d34-0410-b5e6-96231b3b80d8
*We don't currently handle the edge case constants (min/max values), so it's not a complete
canonicalization.
To fully solve the motivating bugs, we need to enhance this to recognize a zero vector
too because that's a ConstantAggregateZero which is a ConstantData, not a ConstantVector
or a ConstantDataVector.
Differential Revision: http://reviews.llvm.org/D17859
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269426 91177308-0d34-0410-b5e6-96231b3b80d8
This reverts commit r269388.
It caused some bots to fail, I'm reverting it until I investigate the
issue.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269395 91177308-0d34-0410-b5e6-96231b3b80d8
This should fix some compile-time regressions after r267672. Thanks to
Chris Matthews for bisecting it.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269392 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
...loop after the last iteration.
This is really hard to do correctly. The core problem is that we need to
model liveness through the induction PHIs from iteration to iteration in
order to get the correct results, and we need to correctly de-duplicate
the common subgraphs of instructions feeding some subset of the
induction PHIs. All of this can be driven either from a side effect at
some iteration or from the loop values used after the loop finishes.
This patch implements this by storing the forward-propagating analysis
of each instruction in a cache to recall whether it was free and whether
it has become live and thus counted toward the total unroll cost. Then,
at each sink for a value in the loop, we recursively walk back through
every value that feeds the sink, including looping back through the
iterations as needed, until we have marked the entire input graph as
live. Because we cache this, we never visit instructions more than twice
-- once when we analyze them and put them into the cache, and once when
we count their cost towards the unrolled loop. Also, because the cache
is only two bits and because we are dealing with relatively small
iteration counts, we can store all of this very densely in memory to
avoid this from becoming an excessively slow analysis.
The code here is still pretty gross. I would appreciate suggestions
about better ways to factor or split this up, I've stared too long at
the algorithmic side to really have a good sense of what the design
should probably look at.
Also, it might seem like we should do all of this bottom-up, but I think
that is a red herring. Specifically, the simplification power is *much*
greater working top-down. We can forward propagate very effectively,
even across strange and interesting recurrances around the backedge.
Because we use data to propagate, this doesn't cause a state space
explosion. Doing this level of constant folding, etc, would be very
expensive to do bottom-up because it wouldn't be until the last moment
that you could collapse everything. The current solution is essentially
a top-down simplification with a bottom-up cost accounting which seems
to get the best of both worlds. It makes the simplification incremental
and powerful while leaving everything dead until we *know* it is needed.
Finally, a core property of this approach is its *monotonicity*. At all
times, the current UnrolledCost is a conservatively low estimate. This
ensures that we will never early-exit from the analysis due to exceeding
a threshold when if we had continued, the cost would have gone back
below the threshold. These kinds of bugs can cause incredibly hard to
track down random changes to behavior.
We could use a techinque similar (but much simpler) within the inliner
as well to avoid considering speculated code in the inline cost.
Reviewers: chandlerc
Subscribers: sanjoy, mzolotukhin, llvm-commits
Differential Revision: http://reviews.llvm.org/D11758
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269388 91177308-0d34-0410-b5e6-96231b3b80d8
Ported DA to the new PM by splitting the former DependenceAnalysis Pass
into a DependenceInfo result type and DependenceAnalysisWrapperPass type
and adding a new PM-style DependenceAnalysis analysis pass returning the
DependenceInfo.
Patch by Philip Pfaffe, most of the review by Justin.
Differential Revision: http://reviews.llvm.org/D18834
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269370 91177308-0d34-0410-b5e6-96231b3b80d8
Split FCMP//ICMP/SEL from the basic arithmetic cost functions. They were not sharing any notable code path (just the return) and were repeatedly testing the opcode.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@269348 91177308-0d34-0410-b5e6-96231b3b80d8
