Commit Graph

2753 Commits

Author SHA1 Message Date
Matt Arsenault
65c0907e68 AMDGPU/GlobalISel: Assume VGPR for G_FRAME_INDEX
In principle this should behave as any other constant. However
eliminateFrameIndex currently assumes a VALU use and uses a vector
shift. Work around this by selecting to VGPR for now until
eliminateFrameIndex is fixed.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373415 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-02 01:02:24 +00:00
Matt Arsenault
addd7f5bb6 AMDGPU/GlobalISel: Private loads always use VGPRs
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373414 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-02 01:02:21 +00:00
Matt Arsenault
8899bbd753 AMDGPU/GlobalISel: Legalize 1024-bit G_BUILD_VECTOR
This will be needed to support AGPR operations.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373413 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-02 01:02:18 +00:00
Matt Arsenault
db73d795b0 AMDGPU/GlobalISel: Fix RegBankSelect for 1024-bit values
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373412 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-02 01:02:14 +00:00
Stanislav Mekhanoshin
e2995cfed4 [AMDGPU] separate accounting for agprs
Account and report agprs separately on gfx908. Other targets
do not change the reporting.

Differential Revision: https://reviews.llvm.org/D68307

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373411 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-02 00:26:58 +00:00
Changpeng Fang
5165c118c2 AMDGPU: Fix an out of date assert in addressing FrameIndex
Reviewers:
  arsenm

Differential Revision:
  https://reviews.llvm.org/D67574

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373404 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 23:07:14 +00:00
Matt Arsenault
8ec8c66e71 AMDGPU/GlobalISel: Increase max legal size to 1024
There are 1024 bit register classes defined for AGPRs. Additionally
OpenCL defines vectors up to 16 x i64, and this helps those tests
legalize.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373350 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 16:35:06 +00:00
Dmitri Gribenko
30e86caa76 Revert "GlobalISel: Handle llvm.read_register"
This reverts commit r373294. It broke Clang's
CodeGen/arm64-microsoft-status-reg.cpp:
http://lab.llvm.org:8011/builders/clang-x86_64-debian-fast/builds/18483

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373310 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 08:24:01 +00:00
Matt Arsenault
1346fe80b5 AMDGPU/GlobalISel: Select s1 src G_SITOFP/G_UITOFP
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373298 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 02:23:20 +00:00
Matt Arsenault
cbd0775331 AMDGPU/GlobalISel: Add support for init.exec intrinsics
TThe existing wave32 behavior seems broken and incomplete, but this
reproduces it.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373296 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 02:07:25 +00:00
Matt Arsenault
af5c54f584 GlobalISel: Handle llvm.read_register
SelectionDAG has a bunch of machinery to defer this to selection time
for some reason. Just directly emit a copy during IRTranslator. The
x86 usage does somewhat questionably check hasFP, which could depend
on the whole function being at minimum translated.

This does lose the convergent bit if the callsite had it, which may be
a problem. We also lose that in general for intrinsics, which may also
be a problem.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373294 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 02:07:16 +00:00
Matt Arsenault
900bd7250e AMDGPU/GlobalISel: Avoid creating shift of 0 in arg lowering
This is sort of papering over the fact that we don't run a combiner
anywhere, but avoiding creating 2 instructions in the first place is
easy.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373293 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 01:44:46 +00:00
Matt Arsenault
1c6b68965a AMDGPU/GlobalISel: Select G_UADDO/G_USUBO
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373288 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 01:23:13 +00:00
Matt Arsenault
fa3f3e76a0 GlobalISel: Implement widenScalar for G_SITOFP/G_UITOFP sources
Legalize 16-bit G_SITOFP/G_UITOFP for AMDGPU.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373287 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 01:06:48 +00:00
Matt Arsenault
c58403f341 AMDGPU/GlobalISel: Legalize G_GLOBAL_VALUE
Handle other cases besides LDS. Mostly a straight port of the existing
handling, without the intermediate custom nodes.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373286 91177308-0d34-0410-b5e6-96231b3b80d8
2019-10-01 01:06:43 +00:00
Alexander Timofeev
864a620db8 [AMDGPU] SIFoldOperands should not fold register acrocc the EXEC definition
Reviewers: rampitec

      Differential Revision: https://reviews.llvm.org/D67662

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373221 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-30 15:31:17 +00:00
Roger Ferrer Ibanez
ca6e7338d6 [TargetLowering] Simplify expansion of S{ADD,SUB}O
ISD::SADDO uses the suggested sequence described in the section §2.4 of
the RISCV Spec v2.2. ISD::SSUBO uses the dual approach but checking for
(non-zero) positive.

Differential Revision: https://reviews.llvm.org/D47927

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373187 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-30 07:58:50 +00:00
Matt Arsenault
fe518bfae1 AMDGPU/GlobalISel: Fix select for v2s16 and/or/xor
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@373180 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-30 06:31:30 +00:00
Stanislav Mekhanoshin
336dc6d0e8 [AMDGPU] Improve fma.f64 test. NFC.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372908 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-25 18:50:34 +00:00
Stanislav Mekhanoshin
6edc006701 [AMDGPU] gfx10 v_fmac_f16 operand folding
Fold immediates into v_fmac_f16.

Differential Revision: https://reviews.llvm.org/D68037

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372906 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-25 18:40:20 +00:00
Matt Arsenault
f9dc386cbe AMDGPU/GlobalISel: Allow selection of scalar min/max
I believe all of the uniform/divergent pattern predicates are
redundant and can be removed. The uniformity bit already influences
the register class, and nothhing has broken when I've removed this and
others.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372450 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-21 02:37:33 +00:00
Stanislav Mekhanoshin
2e8ee4b459 Remove assert from MachineLoop::getLoopPredecessor()
According to the documentation method returns predecessor
if the given loop's header has exactly one unique predecessor
outside the loop. Otherwise return null.

In reality it asserts if there is no predecessor outside of
the loop.

The testcase has the loop where predecessors outside of the
loop were not identified as analyzeBranch() was unable to
process the mask branch and returned true. That is also not
correct to assert for the truly dead loops.

Differential Revision: https://reviews.llvm.org/D67634

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372405 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-20 15:26:10 +00:00
Nico Weber
7622f82bde Revert r372366 "Use getTargetConstant for BLENDI, and add a test to catch it."
This reverts commit 52621307bcab2013e8833f3317cebd63a6db3885.

Tests have been failing all night with

    [0/2] ACTION //llvm/test:check-llvm(//llvm/utils/gn/build/toolchain:unix)
    -- Testing: 33647 tests, 64 threads --
    Testing: 0 .. 10..
    UNRESOLVED: LLVM :: CodeGen/AMDGPU/GlobalISel/isel-blendi-gettargetconstant.ll (6943 of 33647)
    ******************** TEST 'LLVM :: CodeGen/AMDGPU/GlobalISel/isel-blendi-gettargetconstant.ll' FAILED ********************
    Test has no run line!
    ********************

Since there were other concerns on https://reviews.llvm.org/D67785,
I'm just reverting for now.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372383 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-20 12:05:29 +00:00
Sterling Augustine
6decfde0bb Use getTargetConstant for BLENDI, and add a test to catch it.
Summary: This fixes a crasher introduced by r372338.

Reviewers: echristo, arsenm

Subscribers: jvesely, wdng, nhaehnle, hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D67785

Tighten up the test case.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372366 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-20 02:29:16 +00:00
Matt Arsenault
c62b24f070 MachineScheduler: Fix missing dependency with multiple subreg defs
If an instruction had multiple subregister defs, and one of them was
undef, this would improperly conclude all other lanes are
killed. There could still be other defs of those read-undef lanes in
other operands. This would improperly remove register uses from
CurrentVRegUses, so the visitation of later operands would not find
the necessary register dependency. This would also mean this would
fail or not depending on how different subregister def operands were
ordered.

On an undef subregister def, scan the instruction for other
subregister defs and avoid killing those.

This possibly should be deferring removing anything from
CurrentVRegUses until the entire instruction has been processed
instead.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372362 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-20 00:09:15 +00:00
Alexander Timofeev
0df3165d2b [AMDGPU] Unnecessary -amdgpu-scalarize-global-loads=false flag removed from min/max lit tests.
Reviewers: arsenm

Differential Revision: https://reviews.llvm.org/D67712

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372340 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 16:44:38 +00:00
Matt Arsenault
db7d9c2217 Reapply r372285 "GlobalISel: Don't materialize immarg arguments to intrinsics"
This reverts r372314, reapplying r372285 and the commits which depend
on it (r372286-r372293, and r372296-r372297)

This was missing one switch to getTargetConstant in an untested case.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372338 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 16:26:14 +00:00
Hans Wennborg
0c2a34c502 Revert r372285 "GlobalISel: Don't materialize immarg arguments to intrinsics"
This broke the Chromium build, causing it to fail with e.g.

  fatal error: error in backend: Cannot select: t362: v4i32 = X86ISD::VSHLI t392, Constant:i8<15>

See llvm-commits thread of r372285 for details.

This also reverts r372286, r372287, r372288, r372289, r372290, r372291,
r372292, r372293, r372296, and r372297, which seemed to depend on the
main commit.

> Encode them directly as an imm argument to G_INTRINSIC*.
>
> Since now intrinsics can now define what parameters are required to be
> immediates, avoid using registers for them. Intrinsics could
> potentially want a constant that isn't a legal register type. Also,
> since G_CONSTANT is subject to CSE and legalization, transforms could
> potentially obscure the value (and create extra work for the
> selector). The register bank of a G_CONSTANT is also meaningful, so
> this could throw off future folding and legalization logic for AMDGPU.
>
> This will be much more convenient to work with than needing to call
> getConstantVRegVal and checking if it may have failed for every
> constant intrinsic parameter. AMDGPU has quite a lot of intrinsics wth
> immarg operands, many of which need inspection during lowering. Having
> to find the value in a register is going to add a lot of boilerplate
> and waste compile time.
>
> SelectionDAG has always provided TargetConstant for constants which
> should not be legalized or materialized in a register. The distinction
> between Constant and TargetConstant was somewhat fuzzy, and there was
> no automatic way to force usage of TargetConstant for certain
> intrinsic parameters. They were both ultimately ConstantSDNode, and it
> was inconsistently used. It was quite easy to mis-select an
> instruction requiring an immediate. For SelectionDAG, start emitting
> TargetConstant for these arguments, and using timm to match them.
>
> Most of the work here is to cleanup target handling of constants. Some
> targets process intrinsics through intermediate custom nodes, which
> need to preserve TargetConstant usage to match the intrinsic
> expectation. Pattern inputs now need to distinguish whether a constant
> is merely compatible with an operand or whether it is mandatory.
>
> The GlobalISelEmitter needs to treat timm as a special case of a leaf
> node, simlar to MachineBasicBlock operands. This should also enable
> handling of patterns for some G_* instructions with immediates, like
> G_FENCE or G_EXTRACT.
>
> This does include a workaround for a crash in GlobalISelEmitter when
> ARM tries to uses "imm" in an output with a "timm" pattern source.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372314 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 12:33:07 +00:00
Matt Arsenault
a42eeeae65 AMDGPU/GlobalISel: RegBankSelect llvm.amdgcn.ds.swizzle
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372297 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 04:11:17 +00:00
Matt Arsenault
2fc5dfba37 AMDGPU/GlobalISel: Select llvm.amdgcn.raw.buffer.store.format
This needs special handling due to some subtargets that have a
nonstandard register layout for f16 vectors

Also reject some illegal types on other targets.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372293 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:35:08 +00:00
Matt Arsenault
ca2672acce AMDGPU/GlobalISel: Select llvm.amdgcn.raw.buffer.store
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372292 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:30:27 +00:00
Matt Arsenault
75d94a9663 AMDGPU/GlobalISel: RegBankSelect struct buffer load/store
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372291 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:26:53 +00:00
Matt Arsenault
718a0d334f AMDGPU/GlobalISel: RegBankSelect llvm.amdgcn.raw.buffer.{load|store}
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372290 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:25:09 +00:00
Matt Arsenault
ffe06fbfb1 AMDGPU/GlobalISel: Attempt to RegBankSelect image intrinsics
Images should always have 2 consecutive, mandatory SGPR arguments.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372289 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:23:06 +00:00
Matt Arsenault
dbce81cd0d Fix typo
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372288 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:15:29 +00:00
Matt Arsenault
34f46973b8 MachineScheduler: Fix assert from not checking subregs
The assert would fail if there was a dead def of a subregister if
there was a previous use of a different subregister.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372287 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 02:14:12 +00:00
Matt Arsenault
a50f0c836c AMDGPU/GlobalISel: Fix RegBankSelect G_SMULH/G_UMULH pre-gfx9
The scalar versions were only introduced in gfx9.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372286 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 01:42:34 +00:00
Matt Arsenault
1d003d31da GlobalISel: Don't materialize immarg arguments to intrinsics
Encode them directly as an imm argument to G_INTRINSIC*.

Since now intrinsics can now define what parameters are required to be
immediates, avoid using registers for them. Intrinsics could
potentially want a constant that isn't a legal register type. Also,
since G_CONSTANT is subject to CSE and legalization, transforms could
potentially obscure the value (and create extra work for the
selector). The register bank of a G_CONSTANT is also meaningful, so
this could throw off future folding and legalization logic for AMDGPU.

This will be much more convenient to work with than needing to call
getConstantVRegVal and checking if it may have failed for every
constant intrinsic parameter. AMDGPU has quite a lot of intrinsics wth
immarg operands, many of which need inspection during lowering. Having
to find the value in a register is going to add a lot of boilerplate
and waste compile time.

SelectionDAG has always provided TargetConstant for constants which
should not be legalized or materialized in a register. The distinction
between Constant and TargetConstant was somewhat fuzzy, and there was
no automatic way to force usage of TargetConstant for certain
intrinsic parameters. They were both ultimately ConstantSDNode, and it
was inconsistently used. It was quite easy to mis-select an
instruction requiring an immediate. For SelectionDAG, start emitting
TargetConstant for these arguments, and using timm to match them.

Most of the work here is to cleanup target handling of constants. Some
targets process intrinsics through intermediate custom nodes, which
need to preserve TargetConstant usage to match the intrinsic
expectation. Pattern inputs now need to distinguish whether a constant
is merely compatible with an operand or whether it is mandatory.

The GlobalISelEmitter needs to treat timm as a special case of a leaf
node, simlar to MachineBasicBlock operands. This should also enable
handling of patterns for some G_* instructions with immediates, like
G_FENCE or G_EXTRACT.

This does include a workaround for a crash in GlobalISelEmitter when
ARM tries to uses "imm" in an output with a "timm" pattern source.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372285 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-19 01:33:14 +00:00
Tim Renouf
7e270c6c89 [AMDGPU] Allow FP inline constant in v_madak_f16 and v_fmaak_f16
Differential Revision: https://reviews.llvm.org/D67680

Change-Id: Ic38f47cb2079c2c1070a441b5943854844d80a7c

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372208 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-18 09:32:06 +00:00
Alexander Timofeev
7ae04842e0 [AMDGPU]: PHI Elimination hooks added for custom COPY insertion. Fixed
Defferential Revision: https://reviews.llvm.org/D67101

Reviewers: rampitec, vpykhtin

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372086 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-17 09:08:58 +00:00
Amara Emerson
e5fe899c4b [GlobalISel] Partially revert r371901.
r371901 was overeager and widenScalarDst() and the like in the legalizer
attempt to increment the insert point given in order to add new instructions
after the currently legalizing inst. In cases where the insertion point is not
exactly the current instruction, then callers need to de-compensate for the
behaviour by decrementing the insertion iterator before calling them. It's not
a nice state of affairs, for now just undo the problematic parts of the change.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@372050 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 23:46:03 +00:00
Matt Arsenault
fef53526a6 AMDGPU/GlobalISel: Fix some broken run lines
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371992 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 14:14:40 +00:00
Matt Arsenault
8f86794aa2 AMDGPU/GlobalISel: Fix RegBankSelect for G_FRINT and G_FCEIL
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371991 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 14:14:37 +00:00
Matt Arsenault
4fb941a716 AMDGPU/GlobalISel: Remove another illegal select test
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371990 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 14:14:31 +00:00
Matt Arsenault
7511280318 AMDGPU/GlobalISel: Remove illegal select tests
These fail in a release build.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371955 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 04:21:10 +00:00
Matt Arsenault
4cfc531c99 AMDGPU/GlobalISel: Select SMRD loads for more types
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371954 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 00:54:07 +00:00
Matt Arsenault
796815a826 AMDGPU/GlobalISel: RegBankSelect for kill
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371953 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 00:48:37 +00:00
Matt Arsenault
54dea4c5ee AMDGPU/GlobalISel: Legalize s1 source G_[SU]ITOFP
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371952 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 00:37:10 +00:00
Matt Arsenault
8917673a2f AMDGPU/GlobalISel: Set type on vgpr live in special arguments
Fixes assertion with workitem ID intrinsics used in non-kernel
functions.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371951 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 00:33:00 +00:00
Matt Arsenault
83c97ac441 AMDGPU/GlobalISel: Select S16->S32 fptoint
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371950 91177308-0d34-0410-b5e6-96231b3b80d8
2019-09-16 00:32:56 +00:00