1435 Commits

Author SHA1 Message Date
Hans Wennborg
71667f5037 Merging r324195:
------------------------------------------------------------------------
r324195 | mcrosier | 2018-02-04 16:42:24 +0100 (Sun, 04 Feb 2018) | 12 lines

[LV] Use Demanded Bits and ValueTracking for reduction type-shrinking

The type-shrinking logic in reduction detection, although narrow in scope, is
also rather ad-hoc, which has led to bugs (e.g., PR35734). This patch modifies
the approach to rely on the demanded bits and value tracking analyses, if
available. We currently perform type-shrinking separately for reductions and
other instructions in the loop. Long-term, we should probably think about
computing minimal bit widths in a more complete way for the loops we want to
vectorize.

PR35734
Differential Revision: https://reviews.llvm.org/D42309
------------------------------------------------------------------------


git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@325508 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-19 15:24:45 +00:00
Hans Wennborg
9a1cf21b57 Merging r321870, r321872, and r321994:
------------------------------------------------------------------------
r321870 | abataev | 2018-01-05 07:20:40 -0800 (Fri, 05 Jan 2018) | 1 line

[SLP] Update test checks, NFC.
------------------------------------------------------------------------

------------------------------------------------------------------------
r321872 | abataev | 2018-01-05 08:15:17 -0800 (Fri, 05 Jan 2018) | 1 line

[SLP] Update more test checks, NFC.
------------------------------------------------------------------------

------------------------------------------------------------------------
r321994 | abataev | 2018-01-08 06:43:06 -0800 (Mon, 08 Jan 2018) | 13 lines

[SLP] Fix PR35777: Incorrect handling of aggregate values.

Summary:
Fixes the bug with incorrect handling of InsertValue|InsertElement
instrucions in SLP vectorizer. Currently, we may use incorrect
ExtractElement instructions as the operands of the original
InsertValue|InsertElement instructions.

Reviewers: mkuper, hfinkel, RKSimon, spatel

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D41767
------------------------------------------------------------------------


git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322675 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-17 16:04:05 +00:00
Hans Wennborg
a1e0ced878 Merging r322473:
------------------------------------------------------------------------
r322473 | a.elovikov | 2018-01-15 02:56:07 -0800 (Mon, 15 Jan 2018) | 23 lines

[LV] Don't call recordVectorLoopValueForInductionCast for newly-created IV from a trunc.

Summary:
This method is supposed to be called for IVs that have casts in their use-def
chains that are completely ignored after vectorization under PSE. However, for
truncates of such IVs the same InductionDescriptor is used during
creation/widening of both original IV based on PHINode and new IV based on
TruncInst.

This leads to unintended second call to recordVectorLoopValueForInductionCast
with a VectorLoopVal set to the newly created IV for a trunc and causes an
assert due to attempt to store new information for already existing entry in the
map. This is wrong and should not be done.

Fixes PR35773.

Reviewers: dorit, Ayal, mssimpso

Reviewed By: dorit

Subscribers: RKSimon, dim, dcaballe, hsaito, llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D41913
------------------------------------------------------------------------


git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322673 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-17 15:57:43 +00:00
Hans Wennborg
c51141594f Merging r321993:
------------------------------------------------------------------------
r321993 | abataev | 2018-01-08 06:33:11 -0800 (Mon, 08 Jan 2018) | 11 lines

[SLP] Fix PR35628: Count external uses on extra reduction arguments.

Summary:
If the vectorized value is marked as extra reduction argument, its users
are not considered as external users. Patch fixes this.

Reviewers: mkuper, hfinkel, RKSimon, spatel

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D41786
------------------------------------------------------------------------


git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/release_60@322669 91177308-0d34-0410-b5e6-96231b3b80d8
2018-01-17 15:47:38 +00:00
Benjamin Kramer
66f3fb9fac Use phi ranges to simplify code. No functionality change intended.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321585 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-30 15:27:33 +00:00
Mohammad Shahid
d73abb30a4 Revert r320548:[SLP] Vectorize jumbled memory loads
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321181 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-20 15:26:59 +00:00
Florian Hahn
f605692d39 [LV] Remove unnecessary DoExtraAnalysis guard (silent bug)
canVectorize is only checking if the loop has a normalized pre-header if DoExtraAnalysis is true.
This doesn't make sense to me because reporting analysis information shouldn't alter legality
checks. This is probably the result of a last minute minor change before committing (?).

Patch by Diego Caballero.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D40973


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321172 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-20 13:28:38 +00:00
Adrian Prantl
c0ade050e0 Silence a bunch of implicit fallthrough warnings
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@321114 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-19 22:05:25 +00:00
Hal Finkel
3f92210a79 [LV] Extend InstWidening with CM_Widen_Recursive
Changes to the original scalar loop during LV code gen cause the return value
of Legal->isConsecutivePtr() to be inconsistent with the return value during
legal/cost phases (further analysis and information of the bug is in D39346).
This patch is an alternative fix to PR34965 following the CM_Widen approach
proposed by Ayal and Gil in D39346. It extends InstWidening enum with
CM_Widen_Reverse to properly record the widening decision for consecutive
reverse memory accesses and, consequently, get rid of the
Legal->isConsetuviePtr() call in LV code gen. I think this is a simpler/cleaner
solution to PR34965 than the one in D39346.

Fixes PR34965.

Patch by Diego Caballero, thanks!

Differential Revision: https://reviews.llvm.org/D40742

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320913 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-16 02:55:24 +00:00
Hal Finkel
625a9ef4f3 [LV] NFC patch for moving VP*Recipe class definitions from LoopVectorize.cpp to VPlan.h
This is a small step forward to move VPlan stuff to where it should belong (i.e., VPlan.*):

  1. VP*Recipe classes in LoopVectorize.cpp are moved to VPlan.h.
  2. Many of VP*Recipe::print() and execute() definitions are still left in
     LoopVectorize.cpp since they refer to things declared in LoopVectorize.cpp. To
     be moved to VPlan.cpp at a later time.
  3. InterleaveGroup class is moved from anonymous namespace to llvm namespace.
     Referencing it in anonymous namespace from VPlan.h ended up in warning.

Patch by Hideki Saito, thanks!

Differential Revision: https://reviews.llvm.org/D41045

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320900 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-16 01:12:50 +00:00
Guozhi Wei
44388667e9 [SLPVectorizer] Don't ignore scalar extraction instructions of aggregate value
In SLPVectorizer, the vector build instructions (insertvalue for aggregate type) is passed to BoUpSLP.buildTree, it is treated as UserIgnoreList, so later in cost estimation, the cost of these instructions are not counted. 
For aggregate value, later usage are more likely to be done in scalar registers, either used as individual scalars or used as a whole for function call or return value. Ignore scalar extraction instructions may cause too aggressive vectorization for aggregate values, and slow down performance. So for vectorization of aggregate value, the scalar extraction instructions are required in cost estimation.

Differential Revision: https://reviews.llvm.org/D41139



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320736 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-14 19:35:43 +00:00
Dorit Nuzman
3ce8b664b3 [LV] Support efficient vectorization of an induction with redundant casts
D30041 extended SCEVPredicateRewriter to improve handling of Phi nodes whose
update chain involves casts; PSCEV can now build an AddRecurrence for some
forms of such phi nodes, under the proper runtime overflow test. This means
that we can identify such phi nodes as an induction, and the loop-vectorizer
can now vectorize such inductions, however inefficiently. The vectorizer
doesn't know that it can ignore the casts, and so it vectorizes them.

This patch records the casts in the InductionDescriptor, so that they could
be marked to be ignored for cost calculation (we use VecValuesToIgnore for
that) and ignored for vectorization/widening/scalarization (i.e. treated as
TriviallyDead).

In addition to marking all these casts to be ignored, we also need to make
sure that each cast is mapped to the right vector value in the vector loop body
(be it a widened, vectorized, or scalarized induction). So whenever an
induction phi is mapped to a vector value (during vectorization/widening/
scalarization), we also map the respective cast instruction (if exists) to that
vector value. (If the phi-update sequence of an induction involves more than one
cast, then the above mapping to vector value is relevant only for the last cast
of the sequence as we allow only the "last cast" to be used outside the
induction update chain itself).

This is the last step in addressing PR30654.



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320672 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-14 07:56:31 +00:00
Michael Zolotukhin
d5c7271167 Remove redundant includes from lib/Transforms.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320628 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-13 21:31:01 +00:00
Mohammad Shahid
fb6004239d [SLP] Vectorize jumbled memory loads.
Summary:
This patch tries to vectorize loads of consecutive memory accesses, accessed
in non-consecutive or jumbled way. An earlier attempt was made with patch D26905
which was reverted back due to some basic issue with representing the 'use mask' of
jumbled accesses.

This patch fixes the mask representation by recording the 'use mask' in the usertree entry.

Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df

Reviewers: mkuper, loladiro, Ayal, zvi, danielcdh

Reviewed By: Ayal

Subscribers: mgrang, dcaballe, hans, mzolotukhin

Differential Revision: https://reviews.llvm.org/D36130

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320548 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-13 03:08:29 +00:00
Dorit Nuzman
330c5d954f [LV] Ignore the cost of values that will not appear in the vectorized loop
VecValuesToIgnore holds values that will not appear in the vectorized loop.
We should therefore ignore their cost when VF > 1.

Differential Revision: https://reviews.llvm.org/D40883



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@320463 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-12 08:57:43 +00:00
Adam Nemet
d62c5bdaf9 [LV] Interleaved access vectorization: fix computing new alias info
As a new access is generated spanning across multiple fields, we need to
propagate alias info from all the fields to form the most generic alias info.

rdar://35602528

Differential Revision: https://reviews.llvm.org/D40617

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@319979 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-06 22:42:24 +00:00
Hans Wennborg
faed772a25 Revert r319531 "[SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in integer binary ops."
It causes builds to fail with "Instruction does not dominate all uses" (PR35497).

> Patch tries to improve vectorization of the following code:
>
> void add1(int * __restrict dst, const int * __restrict src) {
>   *dst++ = *src++;
>   *dst++ = *src++ + 1;
>   *dst++ = *src++ + 2;
>   *dst++ = *src++ + 3;
> }
> Allows to vectorize even if the very first operation is not a binary add, but just a load.
>
> Fixed issues related to previous commit.
>
> Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev
>
> Reviewed By: ABataev, RKSimon
>
> Subscribers: llvm-commits, RKSimon
>
> Differential Revision: https://reviews.llvm.org/D28907

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@319550 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-01 16:17:24 +00:00
Dinar Temirbulatov
8caaeced90 [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in integer binary ops.
Patch tries to improve vectorization of the following code:
    
            void add1(int * __restrict dst, const int * __restrict src) {
              *dst++ = *src++;
              *dst++ = *src++ + 1;
              *dst++ = *src++ + 2;
              *dst++ = *src++ + 3;
            }
            Allows to vectorize even if the very first operation is not a binary add, but just a load.
    
            Fixed issues related to previous commit.
    
            Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev
    
            Reviewed By: ABataev, RKSimon
    
            Subscribers: llvm-commits, RKSimon
    
            Differential Revision: https://reviews.llvm.org/D28907


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@319531 91177308-0d34-0410-b5e6-96231b3b80d8
2017-12-01 11:10:47 +00:00
Alina Sbirlea
a694e228c2 Add MemorySSA as loop dependency, disabled by default [NFC].
Summary:
First step in adding MemorySSA as dependency for loop pass manager.
Adding the dependency under a flag.

New pass manager: MSSA pointer in LoopStandardAnalysisResults can be null.
Legacy and new pass manager: Use cl::opt EnableMSSALoopDependency. Disabled by default.

Reviewers: sanjoy, davide, gberry

Subscribers: mehdi_amini, Prazek, llvm-commits

Differential Revision: https://reviews.llvm.org/D40274

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318772 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-21 15:45:46 +00:00
NAKAMURA Takumi
6132334604 SLPVectorizer.cpp: Avoid std::stable_sort(properlyDominates()).
properlyDominates() shouldn't be used as sort key. It causes different output between stdlibc++ and libc++.
Instead, I introduced RPOT. In most cases, it works for CSE.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318743 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-21 09:41:01 +00:00
Gil Rapaport
a36486e33b [LV] Model masking in VPlan, introducing VPInstructions
This patch adds a new abstraction layer to VPlan and leverages it to model the planned
instructions that manipulate masks (AND, OR, NOT), introduced during predication.

The new VPValue and VPUser classes model how data flows into, through and out
of a VPlan, forming the vertices of a planned Def-Use graph. The new
VPInstruction class is a generic single-instruction Recipe that models a
planned instruction along with its opcode, operands and users. See
VectorizationPlan.rst for more details.

Differential Revision: https://reviews.llvm.org/D38676


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318645 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-20 12:01:47 +00:00
Adam Nemet
e074ad4a0c [SLP] Added more missed optimization remarks
Summary:
Added more remarks to SLP pass, in particular "missed" optimization remarks.
Also proposed several tests for new functionality.

Patch by Vladimir Miloserdov!

For reference you may look at: https://reviews.llvm.org/rL302811

Reviewers: anemet, fhahn

Reviewed By: anemet

Subscribers: javed.absar, lattner, petecoup, yakush, llvm-commits

Differential Revision: https://reviews.llvm.org/D38367

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318307 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-15 17:04:53 +00:00
Hans Wennborg
2c21c88a19 Revert r318193 "[SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in integer binary ops."
It crashes building sqlite; see reply on the llvm-commits thread.

> [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in integer binary ops.
>
>         Patch tries to improve vectorization of the following code:
>
>         void add1(int * __restrict dst, const int * __restrict src) {
>           *dst++ = *src++;
>           *dst++ = *src++ + 1;
>           *dst++ = *src++ + 2;
>           *dst++ = *src++ + 3;
>         }
>         Allows to vectorize even if the very first operation is not a binary add, but just a load.
>
>         Fixed issues related to previous commit.
>
>         Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev
>
>         Reviewed By: ABataev, RKSimon
>
>         Subscribers: llvm-commits, RKSimon
>
>         Differential Revision: https://reviews.llvm.org/D28907

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318239 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-15 00:38:13 +00:00
Dinar Temirbulatov
9c5d8478b1 [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in integer binary ops.
Patch tries to improve vectorization of the following code:
    
        void add1(int * __restrict dst, const int * __restrict src) {
          *dst++ = *src++;
          *dst++ = *src++ + 1;
          *dst++ = *src++ + 2;
          *dst++ = *src++ + 3;
        }
        Allows to vectorize even if the very first operation is not a binary add, but just a load.
    
        Fixed issues related to previous commit.
    
        Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev
    
        Reviewed By: ABataev, RKSimon
    
        Subscribers: llvm-commits, RKSimon
    
        Differential Revision: https://reviews.llvm.org/D28907


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318193 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-14 20:55:08 +00:00
Gil Rapaport
c07df42b1a [LV] Introduce VPBlendRecipe, VPWidenMemoryInstructionRecipe
This patch is part of D38676.

The patch introduces two new Recipes to handle instructions whose vectorization
involves masking. These Recipes take VPlan-level masks in D38676, but still rely
on ILV's existing createEdgeMask(), createBlockInMask() in this patch.

VPBlendRecipe handles intra-loop phi nodes, which are vectorized as a sequence
of SELECTs. Its execute() code is refactored out of ILV::widenPHIInstruction(),
which now handles only loop-header phi nodes.

VPWidenMemoryInstructionRecipe handles load/store which are to be widened
(but are not part of an Interleave Group). In this patch it simply calls
ILV::vectorizeMemoryInstruction on execute().

Differential Revision: https://reviews.llvm.org/D39068


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318149 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-14 12:09:30 +00:00
Alexey Bataev
9daf9c5c89 [SLP] Fix PR23510: Try to find best possible vectorizable stores.
Summary:
The analysis of the store sequence goes in straight order - from the
first store to the last. Bu the best opportunity for vectorization will
happen if we're going to use reverse order - from last store to the
first. It may be best because usually users have some initialization
part + further processing and this first initialization may confuse
SLP vectorizer.

Reviewers: RKSimon, hfinkel, mkuper, spatel

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D39606

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@317821 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-09 19:07:16 +00:00
Dan Gohman
b5e0bec282 Add an @llvm.sideeffect intrinsic
This patch implements Chandler's idea [0] for supporting languages that
require support for infinite loops with side effects, such as Rust, providing
part of a solution to bug 965 [1].

Specifically, it adds an `llvm.sideeffect()` intrinsic, which has no actual
effect, but which appears to optimization passes to have obscure side effects,
such that they don't optimize away loops containing it. It also teaches
several optimization passes to ignore this intrinsic, so that it doesn't
significantly impact optimization in most cases.

As discussed on llvm-dev [2], this patch is the first of two major parts.
The second part, to change LLVM's semantics to have defined behavior
on infinite loops by default, with a function attribute for opting into
potential-undefined-behavior, will be implemented and posted for review in
a separate patch.

[0] http://lists.llvm.org/pipermail/llvm-dev/2015-July/088103.html
[1] https://bugs.llvm.org/show_bug.cgi?id=965
[2] http://lists.llvm.org/pipermail/llvm-dev/2017-October/118632.html

Differential Revision: https://reviews.llvm.org/D38336


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@317729 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-08 21:59:51 +00:00
Mitch Phillips
cee04762ca Revert rL317618
The implemented pass fails and is breaking a large number of unit tests.
Example:
http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/5777/steps/build-stage3-compiler/logs/stdio

This reverts commit rL317618

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@317641 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-08 00:20:53 +00:00
Dinar Temirbulatov
c7c5ad7774 [SLPVectorizer] Failure to beneficially vectorize 'copyable' elements in integer binary ops.
Patch tries to improve vectorization of the following code:

    void add1(int * __restrict dst, const int * __restrict src) {
      *dst++ = *src++;
      *dst++ = *src++ + 1;
      *dst++ = *src++ + 2;
      *dst++ = *src++ + 3;
    }
    Allows to vectorize even if the very first operation is not a binary add, but just a load.

    Fixed PR34619 and other issues related to previous commit.

    Reviewers: spatel, mzolotukhin, mkuper, hfinkel, RKSimon, filcab, ABataev

    Reviewed By: ABataev, RKSimon

    Subscribers: llvm-commits, RKSimon

    Differential Revision: https://reviews.llvm.org/D28907


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@317618 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-07 21:25:34 +00:00
Sanjay Patel
00e900afdb [IR] redefine 'UnsafeAlgebra' / 'reassoc' fast-math-flags and add 'trans' fast-math-flag
As discussed on llvm-dev:
http://lists.llvm.org/pipermail/llvm-dev/2016-November/107104.html
and again more recently:
http://lists.llvm.org/pipermail/llvm-dev/2017-October/118118.html

...this is a step in cleaning up our fast-math-flags implementation in IR to better match
the capabilities of both clang's user-visible flags and the backend's flags for SDNode.

As proposed in the above threads, we're replacing the 'UnsafeAlgebra' bit (which had the 
'umbrella' meaning that all flags are set) with a new bit that only applies to algebraic 
reassociation - 'AllowReassoc'.

We're also adding a bit to allow approximations for library functions called 'ApproxFunc' 
(this was initially proposed as 'libm' or similar).

...and we're out of bits. 7 bits ought to be enough for anyone, right? :) FWIW, I did 
look at getting this out of SubclassOptionalData via SubclassData (spacious 16-bits), 
but that's apparently already used for other purposes. Also, I don't think we can just 
add a field to FPMathOperator because Operator is not intended to be instantiated. 
We'll defer movement of FMF to another day.

We keep the 'fast' keyword. I thought about removing that, but seeing IR like this:
%f.fast = fadd reassoc nnan ninf nsz arcp contract afn float %op1, %op2
...made me think we want to keep the shortcut synonym.

Finally, this change is binary incompatible with existing IR as seen in the 
compatibility tests. This statement:
"Newer releases can ignore features from older releases, but they cannot miscompile 
them. For example, if nsw is ever replaced with something else, dropping it would be 
a valid way to upgrade the IR." 
( http://llvm.org/docs/DeveloperPolicy.html#ir-backwards-compatibility )
...provides the flexibility we want to make this change without requiring a new IR 
version. Ie, we're not loosening the FP strictness of existing IR. At worst, we will 
fail to optimize some previously 'fast' code because it's no longer recognized as 
'fast'. This should get fixed as we audit/squash all of the uses of 'isFast()'.

Note: an inter-dependent clang commit to use the new API name should closely follow 
commit.

Differential Revision: https://reviews.llvm.org/D39304



git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@317488 91177308-0d34-0410-b5e6-96231b3b80d8
2017-11-06 16:27:15 +00:00
Benjamin Kramer
29c5a7e7b9 [LoopVectorize] Replace manual VPlan memory management with unique_ptr.
No functionality change intended.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@317003 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-31 14:58:22 +00:00
Dehao Chen
b1bfcf247f Do not add discriminator encoding for debug intrinsics.
Summary: There are certain requirements for debug location of debug intrinsics, e.g. the scope of the DILocalVariable should be the same as the scope of its debug location. As a result, we should not add discriminator encoding for debug intrinsics.

Reviewers: dblaikie, aprantl

Reviewed By: aprantl

Subscribers: JDevlieghere, aprantl, bjope, sanjoy, llvm-commits

Differential Revision: https://reviews.llvm.org/D39343

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@316703 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-26 21:20:52 +00:00
Bjorn Pettersson
4ff96e5fbb [LSV] Avoid adding vectors of pointers as candidates
Summary:
We no longer add vectors of pointers as candidates for
load/store vectorization. It does not seem to work anyway,
but without this patch we can end up in asserts when trying
to create casts between an integer type and the pointer of
vectors type.

The test case I've added used to assert like this when trying to
cast between i64 and <2 x i16*>:
opt: ../lib/IR/Instructions.cpp:2565: Assertion `castIsValid(op, S, Ty) && "Invalid cast!"' failed.
#0 PrintStackTraceSignalHandler(void*)
#1 SignalHandler(int)
#2 __restore_rt
#3 __GI_raise
#4 __GI_abort
#5 __GI___assert_fail
#6 llvm::CastInst::Create(llvm::Instruction::CastOps, llvm::Value*, llvm::Type*, llvm::Twine const&, llvm::Instruction*)
#7 llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateBitOrPointerCast(llvm::Value*, llvm::Type*, llvm::Twine const&)
#8 Vectorizer::vectorizeStoreChain(llvm::ArrayRef<llvm::Instruction*>, llvm::SmallPtrSet<llvm::Instruction*, 16u>*)

Reviewers: arsenm

Reviewed By: arsenm

Subscribers: nhaehnle, llvm-commits

Differential Revision: https://reviews.llvm.org/D39296

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@316665 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-26 13:59:15 +00:00
Bjorn Pettersson
941b1f1426 [LSV] Skip all non-byte sizes, not only less than eight bits
Summary:
The code comments indicate that no effort has been spent on
handling load/stores when the size isn't a multiple of the
byte size correctly. However, the code only avoided types
smaller than 8 bits. So for example a load of an i28 could
still be considered as a candidate for vectorization.

This patch adjusts the code to behave according to the code
comment.

The test case used to hit the following assert when
trying to use "cast" an i32 to i28 using CreateBitOrPointerCast:

opt: ../lib/IR/Instructions.cpp:2565: Assertion `castIsValid(op, S, Ty) && "Invalid cast!"' failed.
#0 PrintStackTraceSignalHandler(void*)
#1 SignalHandler(int)
#2 __restore_rt
#3 __GI_raise
#4 __GI_abort
#5 __GI___assert_fail
#6 llvm::CastInst::Create(llvm::Instruction::CastOps, llvm::Value*, llvm::Type*, llvm::Twine const&, llvm::Instruction*)
#7 llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateBitOrPointerCast(llvm::Value*, llvm::Type*, llvm::Twine const&)
#8 (anonymous namespace)::Vectorizer::vectorizeLoadChain(llvm::ArrayRef<llvm::Instruction*>, llvm::SmallPtrSet<llvm::Instruction*, 16u>*)

Reviewers: arsenm

Reviewed By: arsenm

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D39295

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@316663 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-26 13:42:55 +00:00
Eugene Zelenko
1c51faba7a [Transforms] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@316034 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-17 21:27:42 +00:00
NAKAMURA Takumi
904a9e2ef5 Revert rL315894, "SLPVectorizer.cpp: Try to appease stage2-3 difference. (D38586)"
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315896 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-16 09:50:01 +00:00
NAKAMURA Takumi
74cc2953c0 SLPVectorizer.cpp: Try to appease stage2-3 difference. (D38586)
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315894 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-16 09:15:23 +00:00
Eugene Zelenko
8187e7e301 [Transforms] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315640 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-12 23:30:03 +00:00
Vivek Pandya
2540c741d5 [NFC] Convert OptimizationRemarkEmitter old emit() calls to new closure
parameterized emit() calls

Summary: This is not functional change to adopt new emit() API added in r313691.

Reviewed By: anemet

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D38285


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315476 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-11 17:12:59 +00:00
Adam Nemet
3b8950a6d6 Rename OptimizationDiagnosticInfo.* to OptimizationRemarkEmitter.*
Sync it up with the name of the class actually defined here.  This has been
bothering me for a while...

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315249 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-09 23:19:02 +00:00
Ayal Zaks
5eb1a9dfa9 [LV] Fix PR34743 - handle casts that sink after interleaved loads
When ignoring a load that participates in an interleaved group, make sure to
move a cast that needs to sink after it.

Testcase derived from reproducer of PR34743.

Differential Revision: https://reviews.llvm.org/D38338


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314986 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-05 15:45:14 +00:00
Ayal Zaks
0e8a63e67f [LV] Fix PR34711 - widen instruction ranges when sinking casts
Instead of trying to keep LastWidenRecipe updated after creating each recipe,
have tryToWiden() retrieve the last recipe of the current VPBasicBlock and check
if it's a VPWidenRecipe when attempting to extend its range. This ensures that
such extensions, optimized to maintain the original instruction order, do so
only when the instructions are to maintain their relative order. The latter does
not always hold, e.g., when a cast needs to sink to unravel first order
recurrence (r306884).

Testcase derived from reproducer of PR34711.

Differential Revision: https://reviews.llvm.org/D38339


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314981 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-05 12:41:49 +00:00
Hans Wennborg
771c4b9eb5 Revert r314806 "[SLP] Vectorize jumbled memory loads."
All the buildbots are red, e.g.
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/2436/

> Summary:
> This patch tries to vectorize loads of consecutive memory accesses, accessed
> in non-consecutive or jumbled way. An earlier attempt was made with patch D26905
> which was reverted back due to some basic issue with representing the 'use mask' of
> jumbled accesses.
>
> This patch fixes the mask representation by recording the 'use mask' in the usertree entry.
>
> Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df
>
> Reviewers: mkuper, loladiro, Ayal, zvi, danielcdh
>
> Reviewed By: Ayal
>
> Subscribers: hans, mzolotukhin
>
> Differential Revision: https://reviews.llvm.org/D36130

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314824 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-03 18:32:29 +00:00
Mohammad Shahid
0d5a4eca18 [SLP] Vectorize jumbled memory loads.
Summary:
This patch tries to vectorize loads of consecutive memory accesses, accessed
in non-consecutive or jumbled way. An earlier attempt was made with patch D26905
which was reverted back due to some basic issue with representing the 'use mask' of
jumbled accesses.

This patch fixes the mask representation by recording the 'use mask' in the usertree entry.

Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df

Reviewers: mkuper, loladiro, Ayal, zvi, danielcdh

Reviewed By: Ayal

Subscribers: hans, mzolotukhin

Differential Revision: https://reviews.llvm.org/D36130

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314806 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-03 15:28:48 +00:00
Matthew Simpson
e0d97dea2f [LV] Use correct insertion point when type shrinking reductions
When type shrinking reductions, we should insert the truncations and extends at
the end of the loop latch block. Previously, these instructions were inserted
at the end of the loop header block. The difference is only a problem for loops
with predicated instructions (e.g., conditional stores and instructions that
may divide by zero). For these instructions, we create new basic blocks inside
the vectorized loop, which cause the loop header and latch to no longer be the
same block. This should fix PR34687.

Reference: https://bugs.llvm.org/show_bug.cgi?id=34687

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314542 91177308-0d34-0410-b5e6-96231b3b80d8
2017-09-29 18:07:39 +00:00
Sanjoy Das
e87cf87e45 Use a BumpPtrAllocator for Loop objects
Summary:
And now that we no longer have to explicitly free() the Loop instances, we can
(with more ease) use the destructor of LoopBase to do what LoopBase::clear() was
doing.

Reviewers: chandlerc

Subscribers: mehdi_amini, mcrosier, llvm-commits

Differential Revision: https://reviews.llvm.org/D38201

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314375 91177308-0d34-0410-b5e6-96231b3b80d8
2017-09-28 02:45:42 +00:00
Alexey Bataev
463b87bb2b [SLP] Fix crash on propagate IR flags for undef operands of min/max
reductions.

If both operands of the newly created SelectInst are Undefs the
resulting operation is also Undef, not SelectInst. It may cause crashes
when trying to propagate IR flags because function expects exactly
SelectInst instruction, nothing else.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314323 91177308-0d34-0410-b5e6-96231b3b80d8
2017-09-27 17:42:49 +00:00
Sanjay Patel
c4d738472b [SLP] fix typos/formatting; NFC
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314315 91177308-0d34-0410-b5e6-96231b3b80d8
2017-09-27 16:32:56 +00:00
Alexey Bataev
ce5fe1366c [SLP] Support for horizontal min/max reduction.
Summary:
SLP vectorizer supports horizontal reductions for Add/FAdd binary operations. Patch adds support for horizontal min/max reductions.
Function getReductionCost() is split to getArithmeticReductionCost() for binary operation reductions and getMinMaxReductionCost() for min/max reductions.
Patch fixes PR26956.

Reviewers: spatel, mkuper, hfinkel, RKSimon

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D27846

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314101 91177308-0d34-0410-b5e6-96231b3b80d8
2017-09-25 13:34:59 +00:00
Hans Wennborg
1809c66b66 Revert r313771 "[SLP] Vectorize jumbled memory loads."
This broke the buildbots, e.g.
http://bb.pgr.jp/builders/test-llvm-i686-linux-RA/builds/391

> Summary:
> This patch tries to vectorize loads of consecutive memory accesses, accessed
> in non-consecutive or jumbled way. An earlier attempt was made with patch D26905
> which was reverted back due to some basic issue with representing the 'use mask'
> jumbled accesses.
>
> This patch fixes the mask representation by recording the 'use mask' in the usertree entry.
>
> Change-Id: I9fe7f5045f065d84c126fa307ef6ebe0787296df
>
> Subscribers: mzolotukhin
>
> Reviewed By: ayal
>
> Differential Revision: https://reviews.llvm.org/D36130
>
> Review comments updated accordingly
>
> Change-Id: I22ab0a8a9bac9d49d74baa81a08e1e486f5e75f0
>
> Added a TODO for sortLoadAccesses API
>
> Change-Id: I3c679bf1865422d1b45e17ea28f1992bca660b58
>
> Modified the TODO for sortLoadAccesses API
>
> Change-Id: Ie64a66cb5f9e2a7610438abb0e750c6e090f9565
>
> Review comment update for using OpdNum to insert the mask in respective location
>
> Change-Id: I016d0c1b29874e979efc0205bbf078991f92edce
>
> Fixes '-Wsign-compare warning' in LoopAccessAnalysis.cpp and code rebase
>
> Change-Id: I64b2ea5e68c1d7b6a028f5ef8251c5a97333f89b

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@313781 91177308-0d34-0410-b5e6-96231b3b80d8
2017-09-20 18:00:03 +00:00