229 Commits

Author SHA1 Message Date
Austin Kerbow
34e11f6a0a AMDGPU/GlobalISel: Legalize fast unsafe FDIV
Reviewers: arsenm

Reviewed By: arsenm

Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, rovka, dstuttard, tpr, t-tye, hiraditya, Petar.Avramovic, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D69231

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@375460 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-21 22:18:26 +00:00
Matt Arsenault
2097552a47 GlobalISel: Implement lower for G_SADDO/G_SSUBO
Port directly from SelectionDAG, minus the path using
ISD::SADDSAT/ISD::SSUBSAT.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@375042 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-16 20:46:32 +00:00
Matt Arsenault
64f5ca7e80 GlobalISel: Implement fewerElementsVector for G_BUILD_VECTOR
Turn it into a G_CONCAT_VECTORS of G_BUILD_VECTOR.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@374252 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-09 22:44:43 +00:00
Matt Arsenault
e6f17ad250 AMDGPU/GlobalISel: Clamp G_SITOFP/G_UITOFP sources
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373989 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-07 23:33:08 +00:00
Matt Arsenault
0401164ba7 GlobalISel: Partially implement lower for G_INSERT
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373946 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-07 19:13:27 +00:00
Matt Arsenault
d445793c39 AMDGPU/GlobalISel: Widen 16-bit G_MERGE_VALUEs sources
Continue making a mess of merge/unmerge legality.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373942 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-07 19:05:58 +00:00
Matt Arsenault
9ab5b297df AMDGPU/GlobalISel: Lower G_ATOMIC_CMPXCHG_WITH_SUCCESS
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373839 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-06 01:37:37 +00:00
Matt Arsenault
97454bf24b GlobalISel: Partially implement lower for G_EXTRACT
Turn into shift and truncate. Doesn't yet handle pointers.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373838 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-06 01:37:35 +00:00
Matt Arsenault
c15d04b5f6 AMDGPU/GlobalISel: Fix using wrong addrspace for aperture
This was always passing the destination flat address space, when it
should be picking between the two valid source options.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373716 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-04 08:35:38 +00:00
Matt Arsenault
0e53d95ce4 AMDGPU/GlobalISel: Fix mutationIsSane assert v8s8 and
This would try to do FewerElements to v9s8

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373635 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-03 17:50:29 +00:00
Matt Arsenault
118db2620f AMDGPU/GlobalISel: Expand G_BITCAST legality
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373567 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-03 05:46:08 +00:00
Matt Arsenault
11514b8c13 AMDGPU/GlobalISel: Use getIntrinsicID helper
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373417 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-02 01:02:27 +00:00
Matt Arsenault
8899bbd753 AMDGPU/GlobalISel: Legalize 1024-bit G_BUILD_VECTOR
This will be needed to support AGPR operations.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373413 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-02 01:02:18 +00:00
Matt Arsenault
8ec8c66e71 AMDGPU/GlobalISel: Increase max legal size to 1024
There are 1024 bit register classes defined for AGPRs. Additionally
OpenCL defines vectors up to 16 x i64, and this helps those tests
legalize.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373350 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 16:35:06 +00:00
Matt Arsenault
1346fe80b5 AMDGPU/GlobalISel: Select s1 src G_SITOFP/G_UITOFP
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373298 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 02:23:20 +00:00
Matt Arsenault
900bd7250e AMDGPU/GlobalISel: Avoid creating shift of 0 in arg lowering
This is sort of papering over the fact that we don't run a combiner
anywhere, but avoiding creating 2 instructions in the first place is
easy.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373293 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 01:44:46 +00:00
Matt Arsenault
1c6b68965a AMDGPU/GlobalISel: Select G_UADDO/G_USUBO
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373288 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 01:23:13 +00:00
Matt Arsenault
fa3f3e76a0 GlobalISel: Implement widenScalar for G_SITOFP/G_UITOFP sources
Legalize 16-bit G_SITOFP/G_UITOFP for AMDGPU.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373287 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 01:06:48 +00:00
Matt Arsenault
c58403f341 AMDGPU/GlobalISel: Legalize G_GLOBAL_VALUE
Handle other cases besides LDS. Mostly a straight port of the existing
handling, without the intermediate custom nodes.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373286 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 01:06:43 +00:00
Matt Arsenault
db7d9c2217 Reapply r372285 "GlobalISel: Don't materialize immarg arguments to intrinsics"
This reverts r372314, reapplying r372285 and the commits which depend
on it (r372286-r372293, and r372296-r372297)

This was missing one switch to getTargetConstant in an untested case.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372338 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 16:26:14 +00:00
Hans Wennborg
0c2a34c502 Revert r372285 "GlobalISel: Don't materialize immarg arguments to intrinsics"
This broke the Chromium build, causing it to fail with e.g.

  fatal error: error in backend: Cannot select: t362: v4i32 = X86ISD::VSHLI t392, Constant:i8<15>

See llvm-commits thread of r372285 for details.

This also reverts r372286, r372287, r372288, r372289, r372290, r372291,
r372292, r372293, r372296, and r372297, which seemed to depend on the
main commit.

> Encode them directly as an imm argument to G_INTRINSIC*.
>
> Since now intrinsics can now define what parameters are required to be
> immediates, avoid using registers for them. Intrinsics could
> potentially want a constant that isn't a legal register type. Also,
> since G_CONSTANT is subject to CSE and legalization, transforms could
> potentially obscure the value (and create extra work for the
> selector). The register bank of a G_CONSTANT is also meaningful, so
> this could throw off future folding and legalization logic for AMDGPU.
>
> This will be much more convenient to work with than needing to call
> getConstantVRegVal and checking if it may have failed for every
> constant intrinsic parameter. AMDGPU has quite a lot of intrinsics wth
> immarg operands, many of which need inspection during lowering. Having
> to find the value in a register is going to add a lot of boilerplate
> and waste compile time.
>
> SelectionDAG has always provided TargetConstant for constants which
> should not be legalized or materialized in a register. The distinction
> between Constant and TargetConstant was somewhat fuzzy, and there was
> no automatic way to force usage of TargetConstant for certain
> intrinsic parameters. They were both ultimately ConstantSDNode, and it
> was inconsistently used. It was quite easy to mis-select an
> instruction requiring an immediate. For SelectionDAG, start emitting
> TargetConstant for these arguments, and using timm to match them.
>
> Most of the work here is to cleanup target handling of constants. Some
> targets process intrinsics through intermediate custom nodes, which
> need to preserve TargetConstant usage to match the intrinsic
> expectation. Pattern inputs now need to distinguish whether a constant
> is merely compatible with an operand or whether it is mandatory.
>
> The GlobalISelEmitter needs to treat timm as a special case of a leaf
> node, simlar to MachineBasicBlock operands. This should also enable
> handling of patterns for some G_* instructions with immediates, like
> G_FENCE or G_EXTRACT.
>
> This does include a workaround for a crash in GlobalISelEmitter when
> ARM tries to uses "imm" in an output with a "timm" pattern source.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372314 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 12:33:07 +00:00
Matt Arsenault
2fc5dfba37 AMDGPU/GlobalISel: Select llvm.amdgcn.raw.buffer.store.format
This needs special handling due to some subtargets that have a
nonstandard register layout for f16 vectors

Also reject some illegal types on other targets.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372293 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:35:08 +00:00
Matt Arsenault
ca2672acce AMDGPU/GlobalISel: Select llvm.amdgcn.raw.buffer.store
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372292 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:30:27 +00:00
Matt Arsenault
54dea4c5ee AMDGPU/GlobalISel: Legalize s1 source G_[SU]ITOFP
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371952 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 00:37:10 +00:00
Matt Arsenault
83c97ac441 AMDGPU/GlobalISel: Select S16->S32 fptoint
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371950 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 00:32:56 +00:00
Matt Arsenault
f00688b213 AMDGPU/GlobalISel: Legalize s32->s16 G_SITOFP/G_UITOFP
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371811 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-13 04:04:55 +00:00
Matt Arsenault
8fc9eed989 AMDGPU/GlobalISel: Legalize G_FFLOOR
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371803 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-13 01:48:15 +00:00
Matt Arsenault
bf6eeaf07e AMDGPU/GlobalISel: Legalize G_FMAD
Unlike SelectionDAG, treat this as a normally legalizable operation.
In SelectionDAG this is supposed to only ever formed if it's legal,
but I've found that to be restricting. For AMDGPU this is contextually
legal depending on whether denormal flushing is allowed in the use
function.

Technically we currently treat the denormal mode as a subtarget
feature, so custom lowering could be avoided. However I consider this
to be a defect, and this should be contextually dependent on the
controllable rounding mode of the parent function.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371800 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-13 00:44:35 +00:00
Matt Arsenault
cae6fe6cdb AMDGPU/GlobalISel: Select G_FABS/G_FNEG
f64 doesn't work yet because tablegen currently doesn't handlde
REG_SEQUENCE.

This does regress some multi use VALU fneg cases since now the
immediate remains in an SGPR, and more moves are used for legalizing
the xor. This is a SIFixSGPRCopies deficiency.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371540 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-10 17:19:46 +00:00
Matt Arsenault
6c1da931e7 AMDGPU/GlobalISel: RegBankSelect for G_ZEXTLOAD/G_SEXTLOAD
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371536 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-10 16:42:37 +00:00
Matt Arsenault
f37f483128 AMDGPU/GlobalISel: Legalize constant 32-bit loads
Legalize by casting to a 64-bit constant address. This isn't how the
DAG implements it, but it should.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371535 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-10 16:42:31 +00:00
Matt Arsenault
7b1d1c328c AMDGPU/GlobalISel: First pass at attempting to legalize load/stores
There's still a lot more to do, but this handles decomposing due to
alignment. I've gotten it to the point where nothing crashes or
infinite loops the legalizer.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371533 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-10 16:20:14 +00:00
Matt Arsenault
989241eebf AMDGPU/GlobalISel: Fix insert point when lowering fminnum/fmaxnum
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371471 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-09 23:30:11 +00:00
Austin Kerbow
0bbba0846f AMDGPU/GlobalISel: Rename MIRBuilder to B. NFC
Reviewers: arsenm

Reviewed By: arsenm

Subscribers: kzhuravl, jvesely, wdng, nhaehnle, yaxunl, rovka, dstuttard, tpr, t-tye, hiraditya, Petar.Avramovic, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D67374

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371467 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-09 23:06:13 +00:00
Matt Arsenault
2cfa868dbf AMDGPU/GlobalISel: Legalize G_BUILD_VECTOR v2s16
Handle it the same way as G_BUILD_VECTOR_TRUNC. Arguably only
G_BUILD_VECTOR_TRUNC should be legal for this, but G_BUILD_VECTOR will
probably be more convenient in most cases.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371440 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-09 18:57:51 +00:00
Matt Arsenault
a81f890ca1 AMDGPU/GlobalISel: Implement LDS G_GLOBAL_VALUE
Handle the simple case that lowers to a constant.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371424 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-09 17:13:44 +00:00
Matt Arsenault
93eb1d1364 AMDGPU/GlobalISel: Legalize G_BUILD_VECTOR_TRUNC
Treat this as legal on gfx9 since it can use S_PACK_* instructions for
this.

This isn't used by anything yet. The same will probably apply to
16-bit G_BUILD_VECTOR without the trunc.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371423 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-09 17:04:18 +00:00
Matt Arsenault
ec69d2c9e6 AMDGPU/GlobalISel: Select G_PTR_MASK
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371412 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-09 15:46:13 +00:00
Matt Arsenault
cedf41585f AMDGPU/GlobalISel: Legalize wavefrontsize intrinsic
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371407 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-09 15:20:49 +00:00
Matt Arsenault
9f717fa86d AMDGPU: Add intrinsics for address space identification
The library currently uses ptrtoint and directly checks the queue ptr
for this, which counts as a pointer capture.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371009 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-05 02:20:39 +00:00
Matt Arsenault
ef8535baee AMDGPU/GlobalISel: Restore insert point when getting aperture
Avoids SSA violations in a future patch.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371008 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-05 02:20:32 +00:00
Matt Arsenault
6c2a25a8f6 AMDGPU/GlobalISel: Fix placeholder value used for addrspacecast
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371007 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-05 02:20:29 +00:00
Matt Arsenault
b171b8eace GlobalISel: Add basic legalization for G_BITREVERSE
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@370979 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-04 20:46:15 +00:00
Matt Arsenault
d6d1c46508 AMDGPU/GlobalISel: Make 16-bit constants legal
This is mostly for the benefit of patterns which use 16-bit constants.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@370921 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-04 16:19:45 +00:00
Reid Kleckner
9a699c8345 Fix the build for MSVC builds using M_PI
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@370405 91177308-0d34-0410-b5e6-96231b3b80d8
2019-08-29 20:32:53 +00:00
Matt Arsenault
b11bd36479 AMDGPU/GlobalISel: Legalize sin/cos
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@370402 91177308-0d34-0410-b5e6-96231b3b80d8
2019-08-29 20:06:48 +00:00
Matt Arsenault
6881419669 AMDGPU/GlobalISel: Implement addrspacecast for 32-bit constant addrspace
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@370140 91177308-0d34-0410-b5e6-96231b3b80d8
2019-08-28 00:58:24 +00:00
Matt Arsenault
7c120abd60 GlobalISel: Implement moreElementsVector for G_UNMERGE_VALUES sources
This is necessary for handling <3 x s16> on AMDGPU, assuming this
should be handled as 2 separate legalization actions. The alternative
would be for fewerElementsVector to handle 3->2.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@369547 91177308-0d34-0410-b5e6-96231b3b80d8
2019-08-21 16:59:10 +00:00
Matt Arsenault
5508afab5c GlobalISel: Partially implement fewerElementsVector G_UNMERGE_VALUES
Odd sized vectors aren't handled yet.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@368713 91177308-0d34-0410-b5e6-96231b3b80d8
2019-08-13 16:26:28 +00:00
Matt Arsenault
b5027beff2 GlobalISel: Implement lower for G_SHUFFLE_VECTOR
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@368709 91177308-0d34-0410-b5e6-96231b3b80d8
2019-08-13 16:09:07 +00:00