Summary: The code did not check if operand was undef before casting it to Instruction.
Reviewers: RKSimon, ABataev, dtemirbulatov
Reviewed By: ABataev
Subscribers: uabelho
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61024
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@359136 91177308-0d34-0410-b5e6-96231b3b80d8
As it's causing some bot failures (and per request from kbarton).
This reverts commit r358543/ab70da07286e618016e78247e4a24fcb84077fda.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@358546 91177308-0d34-0410-b5e6-96231b3b80d8
This is a refactoring patch which should have all the functionality of the current code. Its goal is twofold:
i. Cleanup and simplify the reordering code, and
ii. Generalize reordering so that it will work for an arbitrary number of operands, not just 2.
This is the second patch in a series of patches that will enable operand reordering across chains of operations. An example of this was presented in EuroLLVM'18 https://www.youtube.com/watch?v=gIEn34LvyNo .
Committed on behalf of @vporpo (Vasileios Porpodas)
Differential Revision: https://reviews.llvm.org/D59973
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@358519 91177308-0d34-0410-b5e6-96231b3b80d8
In PR41304:
https://bugs.llvm.org/show_bug.cgi?id=41304
...we have a case where we want to fold a binop of select-shuffle (blended) values.
Rather than try to match commuted variants of the pattern, we can canonicalize the
shuffles and check for mask equality with commuted operands.
We don't produce arbitrary shuffle masks in instcombine, but select-shuffles are a
special case that the backend is required to handle because we already canonicalize
vector select to this shuffle form.
So there should be no codegen difference from this change. It's possible that this
improves CSE in IR though.
Differential Revision: https://reviews.llvm.org/D60016
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@357366 91177308-0d34-0410-b5e6-96231b3b80d8
For the cases where the icmp/fcmp predicate is commutative, use reorderInputsAccordingToOpcode to collect and commute the operands.
This requires a helper to recognise commutativity in both general Instruction and CmpInstr types - the CmpInst::isCommutative doesn't overload the Instruction::isCommutative method for reasons I'm not clear on (maybe because its based on predicate not opcode?!?).
Differential Revision: https://reviews.llvm.org/D59992
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@357266 91177308-0d34-0410-b5e6-96231b3b80d8
Some predicates are fully commutative - we should be able to easily commute their operands to improve vectorization
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@357202 91177308-0d34-0410-b5e6-96231b3b80d8
As discussed on D59738, this generalizes reorderInputsAccordingToOpcode to handle multiple + non-commutative instructions so we can get rid of reorderAltShuffleOperands and make use of the extra canonicalizations that reorderInputsAccordingToOpcode brings.
Differential Revision: https://reviews.llvm.org/D59784
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@356939 91177308-0d34-0410-b5e6-96231b3b80d8
Remove attempts to commute non-Instructions to the LHS - the codegen changes appear to rely on chance more than anything else and also have a tendency to fight existing instcombine canonicalization which moves constants to the RHS of commutable binary ops.
This is prep work towards:
(a) reusing reorderInputsAccordingToOpcode for alt-shuffles and removing the similar reorderAltShuffleOperands
(b) improving reordering to optimized cases with commutable and non-commutable instructions to still find splat/consecutive ops.
Differential Revision: https://reviews.llvm.org/D59738
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@356913 91177308-0d34-0410-b5e6-96231b3b80d8
If either the main/alt opcodes isn't commutable we may end up with the splats not correctly commuted to the same side.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@356837 91177308-0d34-0410-b5e6-96231b3b80d8
'tmpXXX' conflicts with the auto-generated script regex names.
That could cause mask a bug or fail if the output changes.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@356790 91177308-0d34-0410-b5e6-96231b3b80d8
x86-64 is an invalid architecture in triples. Changing it to the correct
triple (x86_64) changes some tests, because SLP is not deemed profitable
any more.
Reviewers: ABataev, RKSimon, spatel
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D58931
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@355420 91177308-0d34-0410-b5e6-96231b3b80d8
This requires a couple of tweaks to existing vectorization functions as they were assuming that only the second call argument (ctlz/cttz/powi) could ever be the 'always scalar' argument, but for smul.fix + umul.fix its the third argument.
Differential Revision: https://reviews.llvm.org/D58616
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@354790 91177308-0d34-0410-b5e6-96231b3b80d8
As this has broken the lto bootstrap build for 3 days and is
showing a significant regression on the Dither_benchmark results (from
the LLVM benchmark suite) -- specifically, on the
BENCHMARK_FLOYD_DITHER_128, BENCHMARK_FLOYD_DITHER_256, and
BENCHMARK_FLOYD_DITHER_512; the others are unchanged. These have
regressed by about 28% on Skylake, 34% on Haswell, and over 40% on
Sandybridge.
This reverts commit r353923.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@354434 91177308-0d34-0410-b5e6-96231b3b80d8
Try to use 64-bit SLP vectorization. In addition to horizontal instrs
this change triggers optimizations for partial vector operations (for instance,
using low halfs of 128-bit registers xmm0 and xmm1 to multiply <2 x float> by
<2 x float>).
Fixes llvm.org/PR32433
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@353923 91177308-0d34-0410-b5e6-96231b3b80d8
Add generic costs calculation for SADDSAT/SSUBSAT intrinsics, this uses generic costs for sadd_with_overflow/ssub_with_overflow, an extra sign comparison + a selects based on the sign/overflow.
This completes PR40316
Differential Revision: https://reviews.llvm.org/D57239
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@352315 91177308-0d34-0410-b5e6-96231b3b80d8
For the power9 CPU, vector operations consume a pair of execution units rather
than one execution unit like a scalar operation. Update the target transform
cost functions to reflect the higher cost of vector operations when targeting
Power9.
Patch by RolandF.
Differential revision: https://reviews.llvm.org/D55461
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@352261 91177308-0d34-0410-b5e6-96231b3b80d8
Prior to SSE41 (and sometimes on AVX1), vector select has to be performed as a ((X & C)|(Y & ~C)) bit select.
Exposes a couple of issues with the min/max reduction costs (which only go down to SSE42 for some reason).
The increase pre-SSE41 selection costs also prevent a couple of tests from firing any longer, so I've either tweaked the target or added AVX tests as well to the existing SSE2 tests.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351685 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Sometimes the SLP vectorizer tries to vectorize the horizontal reduction
nodes during regular vectorization. This may happen inside of the loops,
when there are some vectorizable PHIs. Patch fixes this by checking if
the node is the reduction node and thus it must not be vectorized, it must
be gathered.
Reviewers: RKSimon, spatel, hfinkel, fedor.sergeev
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D56783
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351349 91177308-0d34-0410-b5e6-96231b3b80d8
official Git repository.
Remove the directions for using git-svn, and demote the prominence of
the svn instructions.
Also, fix a few other issues while I'm in there:
* Mention LLVM_ENABLE_PROJECTS more.
* Getting started doesn't need to mention test-suite, but should
mention clang and the other projects.
* Remove mentions of "configure", since that's long gone.
I've also adjusted a few other mentions of svn to point to github, but
have not done so comprehensively.
Differential Revision: https://reviews.llvm.org/D56654
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351130 91177308-0d34-0410-b5e6-96231b3b80d8
Summary: The comment says we need 3 extracts and a select at the end. But didn't we just account for the select in the vector cost above. Aren't we just extracting the single element after taking the min/max in the vector register?
Reviewers: RKSimon, spatel, ABataev
Reviewed By: RKSimon
Subscribers: javed.absar, kristof.beyls, llvm-commits
Differential Revision: https://reviews.llvm.org/D55480
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348739 91177308-0d34-0410-b5e6-96231b3b80d8
We were overcounting the number of arithmetic operations needed at each level before we reach a legal type. We were using the full vector type for that level, but we are going to split the input vector at that level in half. So the effective arithmetic operation cost at that level is half the width.
So for example on 8i32 on an sse target. Were were calculating the cost of an 8i32 op which is likely 2 for basic integer. Then after the loop we count 2 more v4i32 ops. For a total arith cost of 4. But if you look at the assembly there would only be 3 arithmetic ops.
There are still more bugs in this code that I'm going to work on next. The non pairwise code shouldn't count extract subvectors in the loop. There are no extracts, the types are split in registers. For pairwise we need to use 2 two src permute shuffles.
Differential Revision: https://reviews.llvm.org/D55397
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348621 91177308-0d34-0410-b5e6-96231b3b80d8
We were adding the entire scalarization extraction cost for reductions, which returns the total cost of extracting every element of a vector type.
For reductions we don't need to do this - we just need to extract the 0'th element after the reduction pattern has completed.
Fixes PR37731
Rebased and reapplied after being reverted in rL347541 due to PR39774 - which was fixed by D54955/rL347759 and D55017/rL347997
Differential Revision: https://reviews.llvm.org/D54585
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348076 91177308-0d34-0410-b5e6-96231b3b80d8
This will make these tests immune if we ever change the default behavior of -march=skylake-avx512 to prefer 256 bit vectors.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348046 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
An additional fix for PR39774. Need to update the references for the
RedcutionRoot instruction when it is replaced during the vectorization
phase to avoid compiler crash on reduction vectorization.
Reviewers: RKSimon, spatel
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D55017
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347997 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
If the original reduction root instruction was vectorized, it might be
removed from the tree. It means that the insertion point may become
invalidated and the whole vectorization of the reduction leads to the
incorrect output result.
The ReductionRoot instruction must be marked as externally used so it
could not be removed. Otherwise it might cause inconsistency with the
cost model and we may end up with too optimistic optimization.
Reviewers: RKSimon, spatel, hfinkel, mkuper
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D54955
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347759 91177308-0d34-0410-b5e6-96231b3b80d8
This reverts commit r346970.
It was causing PR39774, a crash in slp-vectorizer on a rather simple loop
with just a bunch of 'and's in the body.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347541 91177308-0d34-0410-b5e6-96231b3b80d8
We were adding the entire scalarization extraction cost for reductions, which returns the total cost of extracting every element of a vector type.
For reductions we don't need to do this - we just need to extract the 0'th element after the reduction pattern has completed.
Fixes PR37731
Differential Revision: https://reviews.llvm.org/D54585
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@346970 91177308-0d34-0410-b5e6-96231b3b80d8
