This adds new function getMemcpyCost to TTI so that the cost of a memcpy can be
modeled and queried. The default implementation returns Expensive, but targets
can override this function to model the cost more accurately.
Differential Revision: https://reviews.llvm.org/D59252
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@356555 91177308-0d34-0410-b5e6-96231b3b80d8
There are a few different issues, mostly stemming from using
generation based checks for anything instead of subtarget
features. Stop adding flat-address-space as a feature for HSA, as it
should only be a device property. This was incorrectly allowing flat
instructions to select for SI.
Increase the default generation for HSA to avoid the encoding error
when emitting objects. This has some other side effects from various
checks which probably should be separate subtarget features (in the
cost model and for dealing with the DS offset folding issue).
Partial fix for bug 41070. It should probably be an error to try using
amdhsa without flat support.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@356347 91177308-0d34-0410-b5e6-96231b3b80d8
AMDGPU would like to have MVTs for v3i32, v3f32, v5i32, v5f32. This
commit does not add them, but makes preparatory changes:
* Fixed assumptions of power-of-2 vector type in kernel arg handling,
and added v5 kernel arg tests and v3/v5 shader arg tests.
* Added v5 tests for cost analysis.
* Added vec3/vec5 arg test cases.
Some of this patch is from Matt Arsenault, also of AMD.
Differential Revision: https://reviews.llvm.org/D58928
Change-Id: I7279d6b4841464d2080eb255ef3c589e268eabcd
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@356342 91177308-0d34-0410-b5e6-96231b3b80d8
Based on an IR equivalent of target lowering's generic expansion - target specific costs will typically be lower (IR doesn't have a good mull/mulh equivalent) but we need a baseline.
Differential Revision: https://reviews.llvm.org/D57925
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@354774 91177308-0d34-0410-b5e6-96231b3b80d8
Add generic costs calculation for SADDSAT/SSUBSAT intrinsics, this uses generic costs for sadd_with_overflow/ssub_with_overflow, an extra sign comparison + a selects based on the sign/overflow.
This completes PR40316
Differential Revision: https://reviews.llvm.org/D57239
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@352315 91177308-0d34-0410-b5e6-96231b3b80d8
For the power9 CPU, vector operations consume a pair of execution units rather
than one execution unit like a scalar operation. Update the target transform
cost functions to reflect the higher cost of vector operations when targeting
Power9.
Patch by RolandF.
Differential revision: https://reviews.llvm.org/D55461
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@352261 91177308-0d34-0410-b5e6-96231b3b80d8
This patch replaces the existing LLVMVectorSameWidth matcher with LLVMScalarOrSameVectorWidth.
The matching args must be either scalars or vectors with the same number of elements, but in either case the scalar/element type can differ, specified by LLVMScalarOrSameVectorWidth.
I've updated the _overflow intrinsics to demonstrate this - allowing it to return a i1 or <N x i1> overflow result, matching the scalar/vectorwidth of the other (add/sub/mul) result type.
The masked load/store/gather/scatter intrinsics have also been updated to use this, although as we specify the reference type to be llvm_anyvector_ty we guarantee the mask will be <N x i1> so no change in behaviour
Differential Revision: https://reviews.llvm.org/D57090
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351957 91177308-0d34-0410-b5e6-96231b3b80d8
First step towards PR40376, this patch adds support for getCmpSelInstrCost to use the (optional) Instruction CmpInst predicate to indicate the type of integer comparison we're performing and alter the costs accordingly.
Differential Revision: https://reviews.llvm.org/D57013
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351810 91177308-0d34-0410-b5e6-96231b3b80d8
Prior to SSE41 (and sometimes on AVX1), vector select has to be performed as a ((X & C)|(Y & ~C)) bit select.
Exposes a couple of issues with the min/max reduction costs (which only go down to SSE42 for some reason).
The increase pre-SSE41 selection costs also prevent a couple of tests from firing any longer, so I've either tweaked the target or added AVX tests as well to the existing SSE2 tests.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351685 91177308-0d34-0410-b5e6-96231b3b80d8
This reverts commit r351125.
I missed test changes in an SLPVectorizer test, due to the cost model
changes. Reverting for now.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351129 91177308-0d34-0410-b5e6-96231b3b80d8
Noticed in D56011 - handle the case that scalar fp ops are quicker on P3 than P4
Add the other costs so that we're not relying on the default "is legal/custom" cost logic.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@350403 91177308-0d34-0410-b5e6-96231b3b80d8
This is split from D55452 with the correct patch this time.
Pairwise reductions require two shuffles on every level but the last. On the last level the two shuffles are <1, u, u, u...> and <0, u, u, u...>, but <0, u, u, u...> will be dropped by InstCombine/DAGCombine as being an identity shuffle.
Differential Revision: https://reviews.llvm.org/D55615
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@349072 91177308-0d34-0410-b5e6-96231b3b80d8
Summary: The comment says we need 3 extracts and a select at the end. But didn't we just account for the select in the vector cost above. Aren't we just extracting the single element after taking the min/max in the vector register?
Reviewers: RKSimon, spatel, ABataev
Reviewed By: RKSimon
Subscribers: javed.absar, kristof.beyls, llvm-commits
Differential Revision: https://reviews.llvm.org/D55480
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348739 91177308-0d34-0410-b5e6-96231b3b80d8
We were overcounting the number of arithmetic operations needed at each level before we reach a legal type. We were using the full vector type for that level, but we are going to split the input vector at that level in half. So the effective arithmetic operation cost at that level is half the width.
So for example on 8i32 on an sse target. Were were calculating the cost of an 8i32 op which is likely 2 for basic integer. Then after the loop we count 2 more v4i32 ops. For a total arith cost of 4. But if you look at the assembly there would only be 3 arithmetic ops.
There are still more bugs in this code that I'm going to work on next. The non pairwise code shouldn't count extract subvectors in the loop. There are no extracts, the types are split in registers. For pairwise we need to use 2 two src permute shuffles.
Differential Revision: https://reviews.llvm.org/D55397
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348621 91177308-0d34-0410-b5e6-96231b3b80d8
I think this has something to do with matching reductions from extractelement, binops, and shuffles. But we're not matching here.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348340 91177308-0d34-0410-b5e6-96231b3b80d8
A loaded value with multiple users compared with 0 will become a load and
test single instruction. The load is not folded in this case (multiple
users), but the compare instruction is eliminated.
This patch returns 0 cost for the icmp in these cases.
Review: Ulrich Weigand
https://reviews.llvm.org/D55111
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348141 91177308-0d34-0410-b5e6-96231b3b80d8
We were adding the entire scalarization extraction cost for reductions, which returns the total cost of extracting every element of a vector type.
For reductions we don't need to do this - we just need to extract the 0'th element after the reduction pattern has completed.
Fixes PR37731
Rebased and reapplied after being reverted in rL347541 due to PR39774 - which was fixed by D54955/rL347759 and D55017/rL347997
Differential Revision: https://reviews.llvm.org/D54585
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@348076 91177308-0d34-0410-b5e6-96231b3b80d8
Three minor changes to these extra costs:
* For ICmp instructions, instead of adding 2 all the time for extending each
operand, this is only done if that operand is neither a load or an
immediate.
* The operands extension costs for divides removed, because we now use a high
cost already for the divide (20).
* The costs for lhsr/ashr extra costs removed as this did not seem useful.
Review: Ulrich Weigand
https://reviews.llvm.org/D55053
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347961 91177308-0d34-0410-b5e6-96231b3b80d8
Unlike most cost model functions this code makes a lot of table lookups without using the results from getTypeLegalizationCost. This means 512-bit vectors can be looked up even when the type isn't legal.
This patch adds a check around the two tables that contain 512-bit types to make sure that neither of the types would be split by type legalization. Meaning 512 bit types are illegal. I wanted to write this in a somewhat generic way that uses type legalization query hooks. But if prefered, I can switch to just using is512BitVector and the subtarget feature.
Differential Revision: https://reviews.llvm.org/D54984
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347786 91177308-0d34-0410-b5e6-96231b3b80d8
This fixes some of scalarization costs reported for sext/zext using avx512bw. This does not fix all scalarization costs being reported. Just the worst.
I've restricted this only to combinations of types that are legal with avx512bw like v32i1/v64i1/v32i16/v64i8 and conversions between vXi1 and vXi8/vXi16 with legal vXi8/vXi16 result types.
Differential Revision: https://reviews.llvm.org/D54979
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347785 91177308-0d34-0410-b5e6-96231b3b80d8
AH, SH and MH costs are already covered in the cases where LHS is 32 bits and
RHS is 16 bits of memory sign-extended to i32.
As these instructions are also used when LHS is i16, this patch recognizes
that the loads will get folded then as well.
Review: Ulrich Weigand
https://reviews.llvm.org/D54940
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347734 91177308-0d34-0410-b5e6-96231b3b80d8
Single instructions exist for i8 and i16 comparisons of memory against a
small immediate.
This patch makes sure that if the load in these cases has a single user (the
ICmp), it gets a 0 cost (folded), and also that the ICmp gets a cost of 1.
Review: Ulrich Weigand
https://reviews.llvm.org/D54897
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347733 91177308-0d34-0410-b5e6-96231b3b80d8
The check lines marked AVX256 in the zext256/sext256 functions should be closer to the AVX values which would take into account a splitting cost.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347722 91177308-0d34-0410-b5e6-96231b3b80d8
Our sext/zext cost modeling was somewhat incomplete. And had no coverage for the fact that avx512bw v32i16/v64i8 types return a scalarization cost.
Truncates are a whole different mess because isTruncateFree is returning true for vectors when it shouldn't and that's the fall back for anything not in the tables.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@347719 91177308-0d34-0410-b5e6-96231b3b80d8
Simon Pilgrim