926 Commits

Author SHA1 Message Date
Diego Caballero
dc4af2458e [VPlan][LV] Introduce condition bit in VPBlockBase
This patch introduces a VPValue in VPBlockBase to represent the condition
bit that is used as successor selector when a block has multiple successors.
This information wasn't necessary until now, when we are about to introduce
outer loop vectorization support in VPlan code gen.

Reviewers: fhahn, rengolin, mkuper, hfinkel, mssimpso

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D48814



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@336554 91177308-0d34-0410-b5e6-96231b3b80d8
2018-07-09 15:57:09 +00:00
Anastasis Grammenos
fb31c6481f [DebugInfo][LoopVectorize] Preserve DL in generated phi instruction
When creating `phi` instructions to resume at the scalar part of the loop,
copy the DebugLoc from the original phi over to the new one.

Differential Revision: https://reviews.llvm.org/D48769

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@336256 91177308-0d34-0410-b5e6-96231b3b80d8
2018-07-04 10:16:55 +00:00
Diego Caballero
ff160b6333 [LV] Prevent LV to run cost model twice for VF=2
This is a minor fix for LV cost model, where the cost for VF=2 was
computed twice when the vectorization of the loop was forced without
specifying a VF.

Reviewers: xusx595, hsaito, fhahn, mkuper

Reviewed By: hsaito, xusx595

Differential Revision: https://reviews.llvm.org/D48048


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334840 91177308-0d34-0410-b5e6-96231b3b80d8
2018-06-15 16:21:35 +00:00
Florian Hahn
b2621b34b7 Use SmallPtrSet explicitly for SmallSets with pointer types (NFC).
Currently SmallSet<PointerTy> inherits from SmallPtrSet<PointerTy>. This
patch replaces such types with SmallPtrSet, because IMO it is slightly
clearer and allows us to get rid of unnecessarily including SmallSet.h

Reviewers: dblaikie, craig.topper

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D47836


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334492 91177308-0d34-0410-b5e6-96231b3b80d8
2018-06-12 11:16:56 +00:00
Craig Topper
74eba10280 Use SmallPtrSet instead of SmallSet in places where we iterate over the set.
SmallSet forwards to SmallPtrSet for pointer types. SmallPtrSet supports iteration, but a normal SmallSet doesn't. So if it wasn't for the forwarding, this wouldn't work.

These places were found by hiding the begin/end methods in the SmallSet forwarding

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334343 91177308-0d34-0410-b5e6-96231b3b80d8
2018-06-09 05:04:20 +00:00
Florian Hahn
da19974802 [VPlan] Move recipe construction to VPRecipeBuilder.
This patch moves the recipe-creation functions out of
LoopVectorizationPlanner, which should do the high-level
orchestration of the transformations.

Reviewers: dcaballe, rengolin, hsaito, Ayal

Reviewed By: dcaballe

Differential Revision: https://reviews.llvm.org/D47595


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334305 91177308-0d34-0410-b5e6-96231b3b80d8
2018-06-08 17:30:45 +00:00
Florian Hahn
1ec0255159 [VPlan] Move recipe based VPlan generation to separate function.
This first step separates VPInstruction-based and VPRecipe-based
VPlan creation, which should make it easier to migrate to VPInstruction
based code-gen step by step.

Reviewers: Ayal, rengolin, dcaballe, hsaito, mkuper, mzolotukhin

Reviewed By: dcaballe

Subscribers: bollu, tschuett, rkruppe, llvm-commits

Differential Revision: https://reviews.llvm.org/D47477


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334284 91177308-0d34-0410-b5e6-96231b3b80d8
2018-06-08 12:53:51 +00:00
Roman Shirokiy
42f7ad099a [LV] Fix PR36983. For a given recurrence, fix all phis in exit block
There could be more than one PHIs in exit block using same loop recurrence.
Don't assume there is only one and fix each user.

Differential Revision: https://reviews.llvm.org/D47788


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334271 91177308-0d34-0410-b5e6-96231b3b80d8
2018-06-08 08:21:20 +00:00
Diego Caballero
6152bb94fd [VPlan] Reland r332654 and silence unused func warning
r332654 was reverted due to an unused function warning in
release build. This commit includes the same code with the
warning silenced.

Differential Revision: https://reviews.llvm.org/D44338



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@332860 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-21 18:14:23 +00:00
Galina Kistanova
a3fda62139 Reverted r332654 as it has broken some buildbots and left unfixed for a long time.
The introduced problem is:
llvm.src/lib/Transforms/Vectorize/VPlanVerifier.cpp:29:13: error: unused function 'hasDuplicates' [-Werror,-Wunused-function]
static bool hasDuplicates(const SmallVectorImpl<VPBlockBase *> &VPBlockVec) {
            ^


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@332747 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-18 18:14:06 +00:00
Diego Caballero
1f4de6bc0a [LV][VPlan] Build plain CFG with simple VPInstructions for outer loops.
Patch #3 from VPlan Outer Loop Vectorization Patch Series #1
(RFC: http://lists.llvm.org/pipermail/llvm-dev/2017-December/119523.html).

Expected to be NFC for the current inner loop vectorization path. It
introduces the basic algorithm to build the VPlan plain CFG (single-level
CFG, no hierarchical CFG (H-CFG), yet) in the VPlan-native vectorization
path using VPInstructions. It includes:
  - VPlanHCFGBuilder: Main class to build the VPlan H-CFG (plain CFG without nested regions, for now).
  - VPlanVerifier: Main class with utilities to check the consistency of a H-CFG.
  - VPlanBlockUtils: Main class with utilities to manipulate VPBlockBases in VPlan.

Reviewers: rengolin, fhahn, mkuper, mssimpso, a.elovikov, hfinkel, aprantl.

Differential Revision: https://reviews.llvm.org/D44338



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@332654 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-17 19:24:47 +00:00
Nicola Zaghen
0818e789cb Rename DEBUG macro to LLVM_DEBUG.
The DEBUG() macro is very generic so it might clash with other projects.
The renaming was done as follows:
- git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g'
- git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM
- Manual change to APInt
- Manually chage DOCS as regex doesn't match it.

In the transition period the DEBUG() macro is still present and aliased
to the LLVM_DEBUG() one.

Differential Revision: https://reviews.llvm.org/D43624



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@332240 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-14 12:53:11 +00:00
Krzysztof Parzyszek
6b025ac685 [LV] Change MaxVectorSize bound to 256 in assertion, NFC otherwise
It's possible to have a vector of 256 bytes in HVX code on Hexagon
(vector pair in 128-byte mode).


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331885 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-09 15:18:12 +00:00
Hideki Saito
3ce25b339f [LV] Fix for PR37248, Broadcast codegen incorrectly assumed vector loop body is single basic block
Summary:
Broadcast code generation emitted instructions in pre-header, while the instruction they are dependent on in the vector loop body.
This resulted in an IL verification error ---- value used before defined.


Reviewers: rengolin, fhahn, hfinkel

Reviewed By: rengolin, fhahn

Subscribers: dcaballe, Ka-Ka, llvm-commits

Differential Revision: https://reviews.llvm.org/D46302

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331799 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-08 18:57:34 +00:00
Adrian Prantl
26b584c691 Remove \brief commands from doxygen comments.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.

Patch produced by

  for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done

Differential Revision: https://reviews.llvm.org/D46290

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331272 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-01 15:54:18 +00:00
Daniel Neilson
974321bd67 [LV] Preserve inbounds on created GEPs
Summary:
This is a fix for PR23997.

The loop vectorizer is not preserving the inbounds property of GEPs that it creates.
This is inhibiting some optimizations. This patch preserves the inbounds property in
the case where a load/store is being fed by an inbounds GEP.

Reviewers: mkuper, javed.absar, hsaito

Reviewed By: hsaito

Subscribers: dcaballe, hsaito, llvm-commits

Differential Revision: https://reviews.llvm.org/D46191

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331269 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-01 15:35:08 +00:00
Florian Hahn
051bfd9f17 [LV] Use BB::instructionsWithoutDebug to skip DbgInfo (NFC).
This patch updates some code responsible the skip debug info to use
BasicBlock::instructionsWithoutDebug. I think this makes things
slightly simpler and more direct.

Reviewers: mkuper, rengolin, dcaballe, aprantl, vsk

Reviewed By: rengolin

Differential Revision: https://reviews.llvm.org/D46254


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331174 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-30 13:28:08 +00:00
Hideki Saito
8784c51c37 [NFC][LV][LoopUtil] Move LoopVectorizationLegality to its own file
Summary:
This is a follow up to D45420 (included here since it is still under review and this change is dependent on that) and D45072 (committed).
Actual change for this patch is LoopVectorize* and cmakefile. All others are all from D45420.

LoopVectorizationLegality is an analysis and thus really belongs to Analysis tree. It is modular enough and it is reusable enough ---- we can further improve those aspects once uses outside of LV picks up.

Hopefully, this will make it easier for people familiar with vectorization theory, but not necessarily LV itself to contribute, by lowering the volume of code they should deal with. We probably should start adding some code in LV to check its own capability (i.e., vectorization is legal but LV is not ready to handle it) and then bail out.


Reviewers: rengolin, fhahn, hfinkel, mkuper, aemerson, mssimpso, dcaballe, sguggill

Reviewed By: rengolin, dcaballe

Subscribers: egarcia, rogfer01, mgorny, llvm-commits

Differential Revision: https://reviews.llvm.org/D45552

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331139 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-29 07:26:18 +00:00
Daniel Neilson
1c0f86d384 [LV] Common duplicate vector load/store address calculation (NFC)
Summary:
Commoning some obviously copy/paste code in
InnerLoopVectorizer::vectorizeMemoryInstruction

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331076 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-27 20:29:18 +00:00
Diego Caballero
e8b06de378 [LV][VPlan] Detect outer loops for explicit vectorization.
Patch #2 from VPlan Outer Loop Vectorization Patch Series #1
(RFC: http://lists.llvm.org/pipermail/llvm-dev/2017-December/119523.html).

This patch introduces the basic infrastructure to detect, legality check
and process outer loops annotated with hints for explicit vectorization.
All these changes are protected under the feature flag
-enable-vplan-native-path. This should make this patch NFC for the existing
inner loop vectorizer.

Reviewers: hfinkel, mkuper, rengolin, fhahn, aemerson, mssimpso.

Differential Revision: https://reviews.llvm.org/D42447


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@330739 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-24 17:04:17 +00:00
Krzysztof Parzyszek
c6388fd9c7 [LV] Introduce TTI::getMinimumVF
The function getMinimumVF(ElemWidth) will return the minimum VF for
a vector with elements of size ElemWidth bits. This value will only
apply to targets for which TTI::shouldMaximizeVectorBandwidth returns
true. The value of 0 indicates that there is no minimum VF.

Differential Revision: https://reviews.llvm.org/D45271


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@330062 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-13 20:16:32 +00:00
Hideki Saito
536a204251 Fix for the buildbot failure. Now-unused private field TTI deleted.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@329649 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-10 00:38:36 +00:00
Hideki Saito
e29ad6ee1f [NFC][LV] Move InterleaveInfo from Legal to CostModel
Summary:
Another clean up, following D43208.

Interleaved memory access analysis/optimization has nothing to do with vectorization legality. It doesn't really belong there. On the other hand, cost model certainly has to know about it.

In principle, vectorization should proceed like Legality ==> Optimization ==> CostModel ==> CodeGen, and this change just does that,
by moving the interleaved access analysis/decision out of Legal, and run it just before CostModel object is created.

After this, I can move LoopVectorizationLegality and Hints/Requirements classes into it's own header file, making it shareable within Transform tree. I have the patch already but I don't want to mix with this change. Eventual goal is to move to Analysis tree, but I first need to move RecurrenceDescriptor/InductionDescriptor from Transform/Util/LoopUtil.* to Analysis.

Reviewers: rengolin, hfinkel, mkuper, dcaballe, sguggill, fhahn, aemerson

Reviewed By: rengolin

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D45072

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@329645 91177308-0d34-0410-b5e6-96231b3b80d8
2018-04-09 23:45:40 +00:00
Krzysztof Parzyszek
4153b00c21 [LV] Add TTI::shouldMaximizeVectorBandwidth to allow enabling it per target
The default implementation returns false and keeps the current behavior.

Differential Revision: https://reviews.llvm.org/D44735


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@328632 91177308-0d34-0410-b5e6-96231b3b80d8
2018-03-27 16:14:11 +00:00
Andrei Elovikov
afd766bf9c [LV] Let recordVectorLoopValueForInductionCast to check if IV was created from the cast.
Summary:
It turned out to be error-prone to expect the callers to handle that - better to
leave the decision to this routine and make the required data to be explicitly
passed to the function.

This handles the case that was missed in the r322473 and fixes the assert
mentioned in PR36524.

Reviewers: dorit, mssimpso, Ayal, dcaballe

Reviewed By: dcaballe

Subscribers: Ka-Ka, hiraditya, dneilson, hsaito, llvm-commits

Differential Revision: https://reviews.llvm.org/D43812

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@327960 91177308-0d34-0410-b5e6-96231b3b80d8
2018-03-20 09:04:39 +00:00
Diego Caballero
ed3ec2499c [LV] Test commit. Removing white space.
This is just to check that I have commit access privilege.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@327656 91177308-0d34-0410-b5e6-96231b3b80d8
2018-03-15 19:34:27 +00:00
Matt Davis
54cff77be9 [CleanUp] Remove NumInstructions field from LoopVectorizer's RegisterUsage struct.
Summary:
This variable is largely going unused; aside from reporting number of instructions for in DEBUG builds.

The only use of NumInstructions is in debug output to represent the LoopSize.  That value can be can be misleading as it also includes metadata instructions (e.g., DBG_VALUE) which have no real impact.  If we do choose to keep this around, we probably should guard it by a DEBUG macro, as it's not used in production builds.



Reviewers: majnemer, congh, rengolin

Reviewed By: rengolin

Subscribers: llvm-commits, rengolin

Differential Revision: https://reviews.llvm.org/D44495

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@327589 91177308-0d34-0410-b5e6-96231b3b80d8
2018-03-14 23:30:31 +00:00
Renato Golin
658734384f [NFC] Consolidate six getPointerOperand() utility functions into one place
There are six separate instances of getPointerOperand() utility.
LoopVectorize.cpp has one of them,
and I don't want to create a 7th one while I'm trying to move
LoopVectorizationLegality into a separate file
(eventual objective is to move it to Analysis tree).

See http://lists.llvm.org/pipermail/llvm-dev/2018-February/120999.html
for llvm-dev discussions

Closes D43323.

Patch by Hideki Saito <hideki.saito@intel.com>.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@327173 91177308-0d34-0410-b5e6-96231b3b80d8
2018-03-09 21:05:58 +00:00
Renato Golin
2e97c7a391 [LV] Fix vectorizer's isUniform() abuse triggers assert in SCEV
Fixes PR36311.

See more detailed analysis in
https://bugs.llvm.org/show_bug.cgi?id=36311.

isUniform() information is recomputed after LV started transforming the
underlying IR and that triggered an assert in SCEV.

From vectorizer's architectural perspective, such information, while
still useful in vector code gen, should not be recomputed after the
start of transforming the LLVM IR. Instead, we should collect and cache
such information during the analysis phase of LV and use the cached info
during code gen.

From the symptom perspective, this assert as it stands right now is not
very useful. Legality already rejected loops that would trigger the
assert. As such, commenting out the assert is NFC from vectorizer's
functionality perspective. On top of that, just above the assertion, we
check for unit-strided load/store or
gather scatter. Addresses can't be uniform below that check.

From vectorization theory point of view, we don't have to reject all
cases of stores to uniform addresses. Eventually, we should support
safe/profitable cases.

This patch resolves the issue by removing the useless assertion that is
invoking LAA's isUniform() that requires up-to-date DomTree ---- once
vector code gen starts modifying CFG, we don't have an up-to-date
DomTree.

Patch by Hideki Saito <hideki.saito@intel.com>.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@327109 91177308-0d34-0410-b5e6-96231b3b80d8
2018-03-09 10:31:31 +00:00
Florian Hahn
9b9abcabb2 [LV][CFG] Add irreducible CFG detection for outer loops
This patch adds support for detecting outer loops with irreducible control
flow in LV. Current detection uses SCCs and only works for innermost loops.
This patch adds a utility function that works on any CFG, given its RPO
traversal and its LoopInfoBase. This function is a generalization
of isIrreducibleCFG  from lib/CodeGen/ShrinkWrap.cpp. The code in
lib/CodeGen/ShrinkWrap.cpp is also updated to use the new generic utility
function.

Patch by Diego Caballero <diego.caballero@intel.com>

Differential Revision: https://reviews.llvm.org/D40874


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@326568 91177308-0d34-0410-b5e6-96231b3b80d8
2018-03-02 12:24:25 +00:00
David Green
dfc9aa7f4e [Dominators] Remove verifyDomTree and add some verifying for Post Dom Trees
Removes verifyDomTree, using assert(verify()) everywhere instead, and
changes verify a little to always run IsSameAsFreshTree first in order
to print good output when we find errors. Also adds verifyAnalysis for
PostDomTrees, which will allow checking of PostDomTrees it the same way
we check DomTrees and MachineDomTrees.

Differential Revision: https://reviews.llvm.org/D41298



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@326315 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-28 11:00:08 +00:00
Renato Golin
3f10059feb [LV] Move isLegalMasked* functions from Legality to CostModel
All SIMD architectures can emulate masked load/store/gather/scatter
through element-wise condition check, scalar load/store, and
insert/extract. Therefore, bailing out of vectorization as legality
failure, when they return false, is incorrect. We should proceed to cost
model and determine profitability.

This patch is to address the vectorizer's architectural limitation
described above. As such, I tried to keep the cost model and
vectorize/don't-vectorize behavior nearly unchanged. Cost model tuning
should be done separately.

Please see
http://lists.llvm.org/pipermail/llvm-dev/2018-January/120164.html for
RFC and the discussions.

Closes D43208.

Patch by: Hideki Saito <hideki.saito@intel.com>

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@326079 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-26 11:06:36 +00:00
Mircea Trofin
757ef7ca81 [LV] Fix analyzeInterleaving when -pass-remarks enabled
Summary:
If -pass-remarks=loop-vectorize, atomic ops will be seen by
analyzeInterleaving(), even though canVectorizeMemory() == false. This
is because we are requesting extra analysis instead of bailing out.

In such a case, we end up with a Group in both Load- and StoreGroups,
and then we'll try to access freed memory when traversing LoadGroups after having had released the Group when  iterating over StoreGroups.

The fix is to include mayWriteToMemory() when validating that two
instructions are the same kind of memory operation.

Reviewers: mssimpso, davidxl

Reviewed By: davidxl

Subscribers: hsaito, fhahn, llvm-commits

Differential Revision: https://reviews.llvm.org/D43064

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@324786 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-10 00:07:45 +00:00
Mircea Trofin
19265c931f Verify profile data confirms large loop trip counts.
Summary:
Loops with inequality comparers, such as:

   // unsigned bound
   for (unsigned i = 1; i < bound; ++i) {...}

have getSmallConstantMaxTripCount report a large maximum static
trip count - in this case, 0xffff fffe. However, profiling info
may show that the trip count is much smaller, and thus
counter-recommend vectorization.

This change:
- flips loop-vectorize-with-block-frequency on by default.
- validates profiled loop frequency data supports vectorization,
  when static info appears to not counter-recommend it. Absence
  of profile data means we rely on static data, just as we've
  done so far.

Reviewers: twoh, mkuper, davidxl, tejohnson, Ayal

Reviewed By: davidxl

Subscribers: bkramer, llvm-commits

Differential Revision: https://reviews.llvm.org/D42946

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@324543 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-07 23:29:52 +00:00
Chad Rosier
451a188a25 [LV] Use Demanded Bits and ValueTracking for reduction type-shrinking
The type-shrinking logic in reduction detection, although narrow in scope, is
also rather ad-hoc, which has led to bugs (e.g., PR35734). This patch modifies
the approach to rely on the demanded bits and value tracking analyses, if
available. We currently perform type-shrinking separately for reductions and
other instructions in the loop. Long-term, we should probably think about
computing minimal bit widths in a more complete way for the loops we want to
vectorize.

PR35734
Differential Revision: https://reviews.llvm.org/D42309

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@324195 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-04 15:42:24 +00:00
Hiroshi Inoue
8040eab589 [NFC] fix trivial typos in comments and documents
"in in" -> "in", "on on" -> "on" etc.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@323508 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-26 08:15:29 +00:00
Andrei Elovikov
9bfe6e5ae2 [LV] Don't call recordVectorLoopValueForInductionCast for newly-created IV from a trunc.
Summary:
This method is supposed to be called for IVs that have casts in their use-def
chains that are completely ignored after vectorization under PSE. However, for
truncates of such IVs the same InductionDescriptor is used during
creation/widening of both original IV based on PHINode and new IV based on
TruncInst.

This leads to unintended second call to recordVectorLoopValueForInductionCast
with a VectorLoopVal set to the newly created IV for a trunc and causes an
assert due to attempt to store new information for already existing entry in the
map. This is wrong and should not be done.

Fixes PR35773.

Reviewers: dorit, Ayal, mssimpso

Reviewed By: dorit

Subscribers: RKSimon, dim, dcaballe, hsaito, llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D41913

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@322473 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-15 10:56:07 +00:00
Hal Finkel
f0c5b1c6c7 [LV][VPlan] NFC patch to move LoopVectorizationPlanner class out of LoopVectorize.cpp
Another small step forward to move VPlan stuff outside of LoopVectorize.cpp.

VPlanBuilder.h is renamed to LoopVectorizationPlanner.h
LoopVectorizationPlanner class is moved from LoopVectorize.cpp to
LoopVectorizationPlanner.h LoopVectorizationCostModel::VectorizationFactor
class is moved to LoopVectorizationPlanner.h (used by the planner class) ---
this needs further streamlining work in later patches and thus all I did was
take it out of the CostModel class and moved to the header file.  The callback
function had to stay inside LoopVectorize.cpp since it calls an
InnerLoopVectorizer member function declared in it.  Next Steps: Make
InnerLoopVectorizer, LoopVectorizationCostModel, and other classes more modular
and more aligned with VPlan direction, in small increments.

Previous step was: r320900 (https://reviews.llvm.org/D41045)

Patch by Hideki Saito, thanks!

Differential Revision: https://reviews.llvm.org/D41420

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321962 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-07 16:02:58 +00:00
Benjamin Kramer
66f3fb9fac Use phi ranges to simplify code. No functionality change intended.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321585 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-30 15:27:33 +00:00
Florian Hahn
f605692d39 [LV] Remove unnecessary DoExtraAnalysis guard (silent bug)
canVectorize is only checking if the loop has a normalized pre-header if DoExtraAnalysis is true.
This doesn't make sense to me because reporting analysis information shouldn't alter legality
checks. This is probably the result of a last minute minor change before committing (?).

Patch by Diego Caballero.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D40973


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321172 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-20 13:28:38 +00:00
Hal Finkel
3f92210a79 [LV] Extend InstWidening with CM_Widen_Recursive
Changes to the original scalar loop during LV code gen cause the return value
of Legal->isConsecutivePtr() to be inconsistent with the return value during
legal/cost phases (further analysis and information of the bug is in D39346).
This patch is an alternative fix to PR34965 following the CM_Widen approach
proposed by Ayal and Gil in D39346. It extends InstWidening enum with
CM_Widen_Reverse to properly record the widening decision for consecutive
reverse memory accesses and, consequently, get rid of the
Legal->isConsetuviePtr() call in LV code gen. I think this is a simpler/cleaner
solution to PR34965 than the one in D39346.

Fixes PR34965.

Patch by Diego Caballero, thanks!

Differential Revision: https://reviews.llvm.org/D40742

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320913 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-16 02:55:24 +00:00
Hal Finkel
625a9ef4f3 [LV] NFC patch for moving VP*Recipe class definitions from LoopVectorize.cpp to VPlan.h
This is a small step forward to move VPlan stuff to where it should belong (i.e., VPlan.*):

  1. VP*Recipe classes in LoopVectorize.cpp are moved to VPlan.h.
  2. Many of VP*Recipe::print() and execute() definitions are still left in
     LoopVectorize.cpp since they refer to things declared in LoopVectorize.cpp. To
     be moved to VPlan.cpp at a later time.
  3. InterleaveGroup class is moved from anonymous namespace to llvm namespace.
     Referencing it in anonymous namespace from VPlan.h ended up in warning.

Patch by Hideki Saito, thanks!

Differential Revision: https://reviews.llvm.org/D41045

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320900 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-16 01:12:50 +00:00
Dorit Nuzman
3ce8b664b3 [LV] Support efficient vectorization of an induction with redundant casts
D30041 extended SCEVPredicateRewriter to improve handling of Phi nodes whose
update chain involves casts; PSCEV can now build an AddRecurrence for some
forms of such phi nodes, under the proper runtime overflow test. This means
that we can identify such phi nodes as an induction, and the loop-vectorizer
can now vectorize such inductions, however inefficiently. The vectorizer
doesn't know that it can ignore the casts, and so it vectorizes them.

This patch records the casts in the InductionDescriptor, so that they could
be marked to be ignored for cost calculation (we use VecValuesToIgnore for
that) and ignored for vectorization/widening/scalarization (i.e. treated as
TriviallyDead).

In addition to marking all these casts to be ignored, we also need to make
sure that each cast is mapped to the right vector value in the vector loop body
(be it a widened, vectorized, or scalarized induction). So whenever an
induction phi is mapped to a vector value (during vectorization/widening/
scalarization), we also map the respective cast instruction (if exists) to that
vector value. (If the phi-update sequence of an induction involves more than one
cast, then the above mapping to vector value is relevant only for the last cast
of the sequence as we allow only the "last cast" to be used outside the
induction update chain itself).

This is the last step in addressing PR30654.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320672 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-14 07:56:31 +00:00
Dorit Nuzman
330c5d954f [LV] Ignore the cost of values that will not appear in the vectorized loop
VecValuesToIgnore holds values that will not appear in the vectorized loop.
We should therefore ignore their cost when VF > 1.

Differential Revision: https://reviews.llvm.org/D40883



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320463 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-12 08:57:43 +00:00
Adam Nemet
d62c5bdaf9 [LV] Interleaved access vectorization: fix computing new alias info
As a new access is generated spanning across multiple fields, we need to
propagate alias info from all the fields to form the most generic alias info.

rdar://35602528

Differential Revision: https://reviews.llvm.org/D40617

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@319979 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-06 22:42:24 +00:00
Alina Sbirlea
a694e228c2 Add MemorySSA as loop dependency, disabled by default [NFC].
Summary:
First step in adding MemorySSA as dependency for loop pass manager.
Adding the dependency under a flag.

New pass manager: MSSA pointer in LoopStandardAnalysisResults can be null.
Legacy and new pass manager: Use cl::opt EnableMSSALoopDependency. Disabled by default.

Reviewers: sanjoy, davide, gberry

Subscribers: mehdi_amini, Prazek, llvm-commits

Differential Revision: https://reviews.llvm.org/D40274

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318772 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-21 15:45:46 +00:00
Gil Rapaport
a36486e33b [LV] Model masking in VPlan, introducing VPInstructions
This patch adds a new abstraction layer to VPlan and leverages it to model the planned
instructions that manipulate masks (AND, OR, NOT), introduced during predication.

The new VPValue and VPUser classes model how data flows into, through and out
of a VPlan, forming the vertices of a planned Def-Use graph. The new
VPInstruction class is a generic single-instruction Recipe that models a
planned instruction along with its opcode, operands and users. See
VectorizationPlan.rst for more details.

Differential Revision: https://reviews.llvm.org/D38676


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318645 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-20 12:01:47 +00:00
Gil Rapaport
c07df42b1a [LV] Introduce VPBlendRecipe, VPWidenMemoryInstructionRecipe
This patch is part of D38676.

The patch introduces two new Recipes to handle instructions whose vectorization
involves masking. These Recipes take VPlan-level masks in D38676, but still rely
on ILV's existing createEdgeMask(), createBlockInMask() in this patch.

VPBlendRecipe handles intra-loop phi nodes, which are vectorized as a sequence
of SELECTs. Its execute() code is refactored out of ILV::widenPHIInstruction(),
which now handles only loop-header phi nodes.

VPWidenMemoryInstructionRecipe handles load/store which are to be widened
(but are not part of an Interleave Group). In this patch it simply calls
ILV::vectorizeMemoryInstruction on execute().

Differential Revision: https://reviews.llvm.org/D39068


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318149 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-14 12:09:30 +00:00
Dan Gohman
b5e0bec282 Add an @llvm.sideeffect intrinsic
This patch implements Chandler's idea [0] for supporting languages that
require support for infinite loops with side effects, such as Rust, providing
part of a solution to bug 965 [1].

Specifically, it adds an `llvm.sideeffect()` intrinsic, which has no actual
effect, but which appears to optimization passes to have obscure side effects,
such that they don't optimize away loops containing it. It also teaches
several optimization passes to ignore this intrinsic, so that it doesn't
significantly impact optimization in most cases.

As discussed on llvm-dev [2], this patch is the first of two major parts.
The second part, to change LLVM's semantics to have defined behavior
on infinite loops by default, with a function attribute for opting into
potential-undefined-behavior, will be implemented and posted for review in
a separate patch.

[0] http://lists.llvm.org/pipermail/llvm-dev/2015-July/088103.html
[1] https://bugs.llvm.org/show_bug.cgi?id=965
[2] http://lists.llvm.org/pipermail/llvm-dev/2017-October/118632.html

Differential Revision: https://reviews.llvm.org/D38336


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@317729 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-08 21:59:51 +00:00
Sanjay Patel
00e900afdb [IR] redefine 'UnsafeAlgebra' / 'reassoc' fast-math-flags and add 'trans' fast-math-flag
As discussed on llvm-dev:
http://lists.llvm.org/pipermail/llvm-dev/2016-November/107104.html
and again more recently:
http://lists.llvm.org/pipermail/llvm-dev/2017-October/118118.html

...this is a step in cleaning up our fast-math-flags implementation in IR to better match
the capabilities of both clang's user-visible flags and the backend's flags for SDNode.

As proposed in the above threads, we're replacing the 'UnsafeAlgebra' bit (which had the 
'umbrella' meaning that all flags are set) with a new bit that only applies to algebraic 
reassociation - 'AllowReassoc'.

We're also adding a bit to allow approximations for library functions called 'ApproxFunc' 
(this was initially proposed as 'libm' or similar).

...and we're out of bits. 7 bits ought to be enough for anyone, right? :) FWIW, I did 
look at getting this out of SubclassOptionalData via SubclassData (spacious 16-bits), 
but that's apparently already used for other purposes. Also, I don't think we can just 
add a field to FPMathOperator because Operator is not intended to be instantiated. 
We'll defer movement of FMF to another day.

We keep the 'fast' keyword. I thought about removing that, but seeing IR like this:
%f.fast = fadd reassoc nnan ninf nsz arcp contract afn float %op1, %op2
...made me think we want to keep the shortcut synonym.

Finally, this change is binary incompatible with existing IR as seen in the 
compatibility tests. This statement:
"Newer releases can ignore features from older releases, but they cannot miscompile 
them. For example, if nsw is ever replaced with something else, dropping it would be 
a valid way to upgrade the IR." 
( http://llvm.org/docs/DeveloperPolicy.html#ir-backwards-compatibility )
...provides the flexibility we want to make this change without requiring a new IR 
version. Ie, we're not loosening the FP strictness of existing IR. At worst, we will 
fail to optimize some previously 'fast' code because it's no longer recognized as 
'fast'. This should get fixed as we audit/squash all of the uses of 'isFast()'.

Note: an inter-dependent clang commit to use the new API name should closely follow 
commit.

Differential Revision: https://reviews.llvm.org/D39304



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@317488 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-06 16:27:15 +00:00