Pull out repeated code for extraction index operand and source vector value type.
Use isNullConstant helper to check for zero extraction index.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295670 91177308-0d34-0410-b5e6-96231b3b80d8
Its more profitable to go through memory (1 cycles throughput)
than using VMOVD + VPERMV/PSHUFB sequence ( 2/3 cycles throughput) to implement EXTRACT_VECTOR_ELT with variable index.
IACA tool was used to get performace estimation (https://software.intel.com/en-us/articles/intel-architecture-code-analyzer)
For example for var_shuffle_v16i8_v16i8_xxxxxxxxxxxxxxxx_i8 test from vector-shuffle-variable-128.ll I get 26 cycles vs 79 cycles.
Removing the VINSERT node, we don't need it any more.
Differential Revision: https://reviews.llvm.org/D29690
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295660 91177308-0d34-0410-b5e6-96231b3b80d8
Replaces existing approach that could only search BUILD_VECTOR nodes.
Requires getTargetConstantBitsFromNode to discriminate cases with all/partial UNDEF bits in each element - this should also be useful when we get around to supporting getTargetShuffleMaskIndices with UNDEF elements.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295613 91177308-0d34-0410-b5e6-96231b3b80d8
As discussed on D27692, this permits another domain to be used to combine a shuffle at high depths.
We currently set the required depth at 4 or more combined shuffles, this is probably too high for most targets but is a good starting point and already helps avoid a number of costly variable shuffles.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295608 91177308-0d34-0410-b5e6-96231b3b80d8
Add the infrastructure to flag whether float and/or int domains are permitable.
A future patch will enable domain crossing based off shuffle depth and the value types of the source vectors.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295604 91177308-0d34-0410-b5e6-96231b3b80d8
Minor performance speedup - if any call to getShuffleScalarElt fails to get a result, don't both calling for the remaining elements as EltsFromConsecutiveLoads will fail anyhow.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295235 91177308-0d34-0410-b5e6-96231b3b80d8
Only do this for integer types currently - floats types (in particular insertps) load folding often fails with this.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295208 91177308-0d34-0410-b5e6-96231b3b80d8
Add support for specifying an UNPCK input as ZERO, particularly improves ZEXT cases with non-zero offsets
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295169 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
We don't seem to have great rules on what a valid VBROADCAST node looks like. And as a consequence we end up with a lot of patterns to try to catch everything. We have patterns with scalar inputs, 128-bit vector inputs, 256-bit vector inputs, and 512-bit vector inputs.
As you can see from the things improved here we are currently missing patterns for 128-bit loads being extended to 256-bit before the vbroadcast.
I'd like to propose that VBROADCAST should always take a 128-bit vector type as input. As a first step towards that this patch adds an EXTRACT_SUBVECTOR in front of VBROADCAST when the input is 256 or 512-bits. In the future I would like to add scalar_to_vector around all the scalar operations. And maybe we should consider adding a VBROADCAST+load node to avoid separating loads from the broadcasting operation when the load itself isn't foldable.
This requires an additional change in target shuffle combining to look for the extract subvector and look through it to find the original operand. I'm sure this change isn't perfect but was enough to fix a few test failures that were being caused.
Another interesting thing I noticed is that the changes in masked_gather_scatter.ll show cases were we don't remove a useless insert into element 1 before broadcasting element 0.
Reviewers: delena, RKSimon, zvi
Reviewed By: zvi
Subscribers: igorb, llvm-commits
Differential Revision: https://reviews.llvm.org/D28747
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295155 91177308-0d34-0410-b5e6-96231b3b80d8
Don't bother setting the V1/V2 operands again for unary shuffles.
Don't bother legalizing the value type unless the match succeeds.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@295051 91177308-0d34-0410-b5e6-96231b3b80d8
We now detect that both the extract and insert indices are non-zero and convert to a shuffle. This will be lowered as a blend for 256-bit vectors or as a vshuf operations for 512-bit vectors.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294931 91177308-0d34-0410-b5e6-96231b3b80d8
This results in the simplifications inside of getNode running while we're legalizing nodes popped off the worklist during the final DAG combine. This basically makes a DAG combine like operation occur during this legalize step, but we don't handle something quite the same way. I think we don't recursively added the removed nodes to the DAG combiner worklist.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294929 91177308-0d34-0410-b5e6-96231b3b80d8
The target shuffle match function arguments were using the term 'Ops' but the function names referred to them as 'Inputs' - use 'Inputs' consistently.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294900 91177308-0d34-0410-b5e6-96231b3b80d8
Initial 256-bit vector support - 512-bit support requires extra checks for AVX512BW support (PMOVZXBW) that will be handled in a future patch.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294896 91177308-0d34-0410-b5e6-96231b3b80d8
Generalize VSEXT/VZEXT constant folding to work with any target constant bits source not just BUILD_VECTOR .
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294873 91177308-0d34-0410-b5e6-96231b3b80d8
Removes duplicate constant extraction code in getTargetShuffleMaskIndices.
getTargetConstantBitsFromNode - adds support for VZEXT_MOVL(SCALAR_TO_VECTOR) and fail if the caller doesn't support undef bits.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294856 91177308-0d34-0410-b5e6-96231b3b80d8
Since r274013, we've been looking through bitcasts on broadcast inputs.
In the scalar-folding case (from a load, build_vector, or sc2vec),
the input type didn't matter, as we'd simply bitcast the resulting
scalar back.
However, when broadcasting a 128-bit-lane-aligned element, we create an
EXTRACT_SUBVECTOR. Use proper types, by creating an extract_subvector
of the original input type.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294774 91177308-0d34-0410-b5e6-96231b3b80d8
In some cases we call getTargetConstantBitsFromNode for nodes that haven't been lowered from BUILD_VECTOR yet
Note: We're getting very close to being able to move most of the constant extraction code from getTargetShuffleMaskIndices into getTargetConstantBitsFromNode
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294746 91177308-0d34-0410-b5e6-96231b3b80d8
LowerBuildVectorv16i8/LowerBuildVectorv8i16 insert values into a UNDEF vector if the build vector doesn't contain any zero elements, resulting in register dependencies with a previous use of the register.
This patch attempts to break the register dependency by either always zeroing the vector before hand or (if we're inserting to the 0'th element) by using VZEXT_MOVL(SCALAR_TO_VECTOR(i32 AEXT(Elt))) which lowers to (V)MOVD and performs a similar function. Additionally (V)MOVD is a shorter instruction than PINSRB/PINSRW. We already do something similar for SSE41 PINSRD.
On pre-SSE41 LowerBuildVectorv16i8 we go a little further and use VZEXT_MOVL(SCALAR_TO_VECTOR(i32 ZEXT(Elt))) if the build vector contains zeros to avoid the vector zeroing at the cost of a scalar zero extension, which can probably be brought over to the other cases in a future patch in some cases (load folding etc.)
Differential Revision: https://reviews.llvm.org/D29720
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294581 91177308-0d34-0410-b5e6-96231b3b80d8
This patch does the following.
1. Adds an Intrinsic int_x86_clzero which works with __builtin_ia32_clzero
2. Identifies clzero feature using cpuid info. (Function:8000_0008, Checks if EBX[0]=1)
3. Adds the clzero feature under znver1 architecture.
4. The custom inserter is added in Lowering.
5. A testcase is added to check the intrinsic.
6. The clzero instruction is added to assembler test.
Patch by Ganesh Gopalasubramanian with a couple formatting tweaks, a disassembler test, and using update_llc_test.py from me.
Differential revision: https://reviews.llvm.org/D29385
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294558 91177308-0d34-0410-b5e6-96231b3b80d8
