Summary:
TargetBaseAlign is no longer required since LSV checks if target allows misaligned accesses.
A constant defining a base alignment is still needed for stack accesses where alignment can be adjusted.
Reviewers: llvm-commits, jlebar
Subscribers: mzolotukhin, arsenm
Differential Revision: https://reviews.llvm.org/D22936
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@277038 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Given the crash in D22878, this patch converts the load/store vectorizer
to use explicit Instruction*s wherever possible. This is an overall
simplification and should be an improvement in safety, as we have fewer
naked cast<>s, and now where we use Value*, we really mean something
different from Instruction*.
This patch also gets rid of some cast<>s around Value*s returned by
Builder. Given that Builder constant-folds everything, we can't assume
much about what we get out of it.
One downside of this patch is that we have to copy our chain before
calling propagateMetadata. But I don't think this is a big deal, as our
chains are very small (usually 2 or 4 elems).
Reviewers: asbirlea
Subscribers: mzolotukhin, llvm-commits, arsenm
Differential Revision: https://reviews.llvm.org/D22887
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276938 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
When we ask the builder to create a bitcast on a constant, we get back a
constant, not an instruction.
Reviewers: asbirlea
Subscribers: jholewinski, mzolotukhin, llvm-commits, arsenm
Differential Revision: https://reviews.llvm.org/D22878
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276922 91177308-0d34-0410-b5e6-96231b3b80d8
Allowed loop vectorization with secondary FP IVs. Like this:
float *A;
float x = init;
for (int i=0; i < N; ++i) {
A[i] = x;
x -= fp_inc;
}
The auto-vectorization is possible when the induction binary operator is "fast" or the function has "unsafe" attribute.
Differential Revision: https://reviews.llvm.org/D21330
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276554 91177308-0d34-0410-b5e6-96231b3b80d8
When vectorizing a tree rooted at a store bundle, we currently try to sort the
stores before building the tree, so that the stores can be vectorized. For other
trees, the order of the root bundle - which determines the order of all other
bundles - is arbitrary. That is bad, since if a leaf bundle of consecutive loads
happens to appear in the wrong order, we will not vectorize it.
This is partially mitigated when the root is a binary operator, by trying to
build a "reversed" tree when that's considered profitable. This patch extends the
workaround we have for binops to trees rooted in a horizontal reduction.
This fixes PR28474.
Differential Revision: https://reviews.llvm.org/D22554
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276477 91177308-0d34-0410-b5e6-96231b3b80d8
This patch moves the update instruction for vectorized integer induction phi
nodes to the end of the latch block. This ensures consistent placement of all
induction updates across all the kinds of int inductions we create (scalar,
splat vector, or vector phi).
Differential Revision: https://reviews.llvm.org/D22416
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276339 91177308-0d34-0410-b5e6-96231b3b80d8
Test coverage is provided by modifying the function in the FP-math
testcase that we are allowed to vectorize.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276223 91177308-0d34-0410-b5e6-96231b3b80d8
The earlier change added hotness attribute to missed-optimization
remarks. This follows up with the analysis remarks (the ones explaining
the reason for the missed optimization).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276192 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Previously we wouldn't move loads/stores across instructions that had
side-effects, where that was defined as may-write or may-throw. But
this is not sufficiently restrictive: Stores can't safely be moved
across instructions that may load.
This patch also adds a DEBUG check that all instructions in our chain
are either loads or stores.
Reviewers: asbirlea
Subscribers: llvm-commits, jholewinski, arsenm, mzolotukhin
Differential Revision: https://reviews.llvm.org/D22547
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276171 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Previously if we had a chain that contained a side-effecting
instruction, we wouldn't vectorize it at all. Now we'll vectorize
everything that comes before the side-effecting instruction.
Reviewers: asbirlea
Subscribers: arsenm, jholewinski, llvm-commits, mzolotukhin
Differential Revision: https://reviews.llvm.org/D22536
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276170 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
getVectorizablePrefix previously didn't work properly in the face of
aliasing loads/stores. It unwittingly assumed that the loads/stores
appeared in the BB in address order. If they didn't, it would do the
wrong thing.
Reviewers: asbirlea, tstellarAMD
Subscribers: arsenm, llvm-commits, mzolotukhin
Differential Revision: https://reviews.llvm.org/D22535
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276072 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Previously, the insertion point for stores was the last instruction in
Chain *before calling getVectorizablePrefixEndIdx*. Thus if
getVectorizablePrefixEndIdx didn't return Chain.size(), we still would
insert at the last instruction in Chain.
This patch changes our internal API a bit in an attempt to make it less
prone to this sort of error. As a result, we end up recalculating the
Chain's boundary instructions, but I think worrying about the speed hit
of this is a premature optimization right now.
Reviewers: asbirlea, tstellarAMD
Subscribers: mzolotukhin, arsenm, llvm-commits
Differential Revision: https://reviews.llvm.org/D22534
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276056 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
The DEBUG message was hard to read because two Values were being printed
on the same line with only the delimiter "aliases". This change makes
us print each Value on its own line.
Reviewers: asbirlea
Subscribers: llvm-commits, arsenm, mzolotukhin
Differential Revision: https://reviews.llvm.org/D22533
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276055 91177308-0d34-0410-b5e6-96231b3b80d8
For instructions in uniform set, they will not have vector versions so
add them to VecValuesToIgnore.
For induction vars, those only used in uniform instructions or consecutive
ptrs instructions have already been added to VecValuesToIgnore above. For
those induction vars which are only used in uniform instructions or
non-consecutive/non-gather scatter ptr instructions, the related phi and
update will also be added into VecValuesToIgnore set.
The change will make the vector RegUsages estimation less conservative.
Differential Revision: https://reviews.llvm.org/D20474
The recommit fixed the testcase global_alias.ll.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275936 91177308-0d34-0410-b5e6-96231b3b80d8
For instructions in uniform set, they will not have vector versions so
add them to VecValuesToIgnore.
For induction vars, those only used in uniform instructions or consecutive
ptrs instructions have already been added to VecValuesToIgnore above. For
those induction vars which are only used in uniform instructions or
non-consecutive/non-gather scatter ptr instructions, the related phi and
update will also be added into VecValuesToIgnore set.
The change will make the vector RegUsages estimation less conservative.
Differential Revision: https://reviews.llvm.org/D20474
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275912 91177308-0d34-0410-b5e6-96231b3b80d8
This patch swaps A and B in the interleaved access analysis and clarifies
related comments. The algorithm is more intuitive if we let access A precede
access B in program order rather than the reverse. This change was requested in
the review of D19984.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275567 91177308-0d34-0410-b5e6-96231b3b80d8
We now collect all accesses with a constant stride, not just the ones with a
stride greater than one. This change was requested in the review of D19984.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275473 91177308-0d34-0410-b5e6-96231b3b80d8
This patch prevents increases in the number of instructions, pre-instcombine,
due to induction variable scalarization. An increase in instructions can lead
to an increase in the compile-time required to simplify the induction
variables. We now maintain a new map for scalarized induction variables to
prevent us from converting between the scalar and vector forms.
This patch should resolve compile-time regressions seen after r274627.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275419 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
LSV used to abort vectorizing a chain for interleaved load/store accesses that alias.
Allow a valid prefix of the chain to be vectorized, mark just the prefix and retry vectorizing the remaining chain.
Reviewers: llvm-commits, jlebar, arsenm
Subscribers: mzolotukhin
Differential Revision: http://reviews.llvm.org/D22119
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275317 91177308-0d34-0410-b5e6-96231b3b80d8
The LCSSA pass itself will not generate several redundant PHI nodes in a single
exit block. However, such redundant PHI nodes don't violate LCSSA form, and may
be introduced by passes that preserve LCSSA, and/or preserved by the LCSSA pass
itself. So, assuming a single PHI node per exit block is not safe.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275217 91177308-0d34-0410-b5e6-96231b3b80d8
Use range-based for loops instead of doing everything manually.
Use auto when appropriate.
No functional change is intended.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275205 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Aiming to correct the ordering of loads/stores. This patch changes the
insert point for loads to the position of the first load.
It updates the ordering method for loads to insert before, rather than after.
Before this patch the following sequence:
"load a[1], store a[1], store a[0], load a[2]"
Would incorrectly vectorize to "store a[0,1], load a[1,2]".
The correctness check was assuming the insertion point for loads is at
the position of the first load, when in practice it was at the last
load. An alternative fix would have been to invert the correctness check.
The current fix changes insert position but also requires reordering of
instructions before the vectorized load.
Updated testcases to reflect the changes.
Reviewers: tstellarAMD, llvm-commits, jlebar, arsenm
Subscribers: mzolotukhin
Differential Revision: http://reviews.llvm.org/D22071
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@275117 91177308-0d34-0410-b5e6-96231b3b80d8
We currently always vectorize induction variables. However, if an induction
variable is only used for counting loop iterations or computing addresses with
getelementptr instructions, we don't need to do this. Vectorizing these trivial
induction variables can create vector code that is difficult to simplify later
on. This is especially true when the unroll factor is greater than one, and we
create vector arithmetic when computing step vectors. With this patch, we check
if an induction variable is only used for counting iterations or computing
addresses, and if so, scalarize the arithmetic when computing step vectors
instead. This allows for greater simplification.
This patch addresses the suboptimal pointer arithmetic sequence seen in
PR27881.
Reference: https://llvm.org/bugs/show_bug.cgi?id=27881
Differential Revision: http://reviews.llvm.org/D21620
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@274627 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
GetBoundryInstruction returns the last instruction as the instruction which follows or end(). Otherwise the last instruction in the boundry set is not being tested by isVectorizable().
Partially solve reordering of instructions. More extensive solution to follow.
Reviewers: tstellarAMD, llvm-commits, jlebar
Subscribers: escha, arsenm, mzolotukhin
Differential Revision: http://reviews.llvm.org/D21934
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@274389 91177308-0d34-0410-b5e6-96231b3b80d8
If OpB has an ADD NSW/NUW, we can use that to prove that adding 1
to OpA won't wrap if OpA + 1 == OpB.
Patch by Fiona Glaser
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@274324 91177308-0d34-0410-b5e6-96231b3b80d8
If no alignment was set on the load/stores, it would vectorize
to the new type even though this increases the default alignment.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@274323 91177308-0d34-0410-b5e6-96231b3b80d8
This needs to use inttoptr/ptrtoint if combining an int and pointer
load. If a pointer is used always do an integer load.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@274321 91177308-0d34-0410-b5e6-96231b3b80d8
