In -P mode, PrintPPOutputPPCallbacks::MoveToLine started at least one
newline if current and target line number mismatched. The method is also
called when entering a new file, be it the main file or an include file.
In this situation line numbers always almost mismatch, resulting in a
newline for each occurance even if no tokens have been printed
in-between.
Empty lines at the beginning of the output must be trimmed because it
may be parsed by scripts expecting the result to appear on the first
output line, as done by LibreOffice's configure script.
Fix by only emitting a newline if tokens have been printed so far using
the EmittedTokensOnThisLine flag. Also adding a test case of FileChanged
callbacks occuring with empty include files.
This fixes llvm.org/PR51616
The -fms-extensions converts __pragma (and _Pragma) into a #pragma that
has to occur at the beginning of a line and end with a newline. This
patch ensures that the newline after the #pragma is added even if
Token::isAtStartOfLine() indicated that we should not start a newline.
Committing relying post-commit review since the change is small, some
downstream uses might be blocked without this fix, and to make clear the
decision of the new -fminimize-whitespace feature (fix on main, revert
on clang-13.x branch) suggested by @aaron.ballman in D104601.
Differential Revision: https://reviews.llvm.org/D107183
The implementation of -fminimize-whitespace (D104601) revised the logic
when to emit newlines. There was no case to handle when more than
8 lines were skippped in -P (DisableLineMarkers) mode and instead fell
through the case intended for -fminimize-whitespace, i.e. emit nothing.
This patch will emit one newline in this case.
The newline logic is slightly reorganized. The `-P -fminimize-whitespace`
case is handled explicitly and emitting at least one newline is the new
fallback case. The choice between emitting a line marker or up to
7 empty lines is now a choice only with enabled line markers. The up to
8 newlines likely are fewer characters than a line directive, but
in -P mode this had the paradoxic effect that it would print up to
7 empty lines, but none at all if more than 8 lines had to be skipped.
Now with DisableLineMarkers, we don't consider printing empty lines
(just start a new line) which matches gcc's behavior.
The line-directive-output-mincol.c test is replaced with a more
comprehensive test skip-empty-lines.c also testing the more than
8 skipped lines behaviour with all flag combinations.
Reviewed By: aaron.ballman
Differential Revision: https://reviews.llvm.org/D106924
This patch adds the -fminimize-whitespace with the following effects:
* If combined with -E, remove as much non-line-breaking whitespace as
possible.
* If combined with -E -P, removes as much whitespace as possible,
including line-breaks.
The motivation is to reduce the amount of insignificant changes in the
preprocessed output with source files where only whitespace has been
changed (add/remove comments, clang-format, etc.) which is in particular
useful with ccache.
A patch for ccache for using this flag has been proposed to ccache as well:
https://github.com/ccache/ccache/pull/815, which will use
-fnormalize-whitespace when clang-13 has been detected, and additionally
uses -P in "unify_mode". ccache already had a unify_mode in an older
version which was removed because of problems that using the
preprocessor itself does not have (such that the custom tokenizer did
not recognize C++11 raw strings).
This patch slightly reorganizes which part is responsible for adding
newlines that are required for semantics. It is now either
startNewLineIfNeeded() or MoveToLine() but never both; this avoids the
ShouldUpdateCurrentLine workaround and avoids redundant lines being
inserted in some cases. It also fixes a mandatory newline not inserted
after a _Pragma("...") that is expanded into a #pragma.
Reviewed By: aaron.ballman
Differential Revision: https://reviews.llvm.org/D104601
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368942
Currently, a pragma AST node's recorded location starts at the
namespace token (such as `omp` in the case of OpenMP) after the
`#pragma` token, and the `#pragma` location isn't available. However,
the `#pragma` location can be useful when, for example, rewriting a
directive using Clang's Rewrite facility.
This patch makes `#pragma` locations available in any `PragmaHandler`
but it doesn't yet make use of them.
This patch also uses the new `struct PragmaIntroducer` to simplify
`Preprocessor::HandlePragmaDirective`. It doesn't do the same for
`PPCallbacks::PragmaDirective` because that changes the API documented
in `clang-tools-extra/docs/pp-trace.rst`, and I'm not sure about
backward compatibility guarantees there.
Reviewed By: ABataev, lebedev.ri, aaron.ballman
Differential Revision: https://reviews.llvm.org/D61643
llvm-svn: 361335
Summary:
By adding a hook to consume all tokens produced by the preprocessor.
The intention of this change is to make it possible to consume the
expanded tokens without re-runnig the preprocessor with minimal changes
to the preprocessor and minimal performance penalty when preprocessing
without recording the tokens.
The added hook is very low-level and reconstructing the expanded token
stream requires more work in the client code, the actual algorithm to
collect the tokens using this hook can be found in the follow-up change.
Reviewers: rsmith
Reviewed By: rsmith
Subscribers: eraman, nemanjai, kbarton, jsji, riccibruno, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D59885
llvm-svn: 361007
__pragma(execution_character_set(push, "UTF-8")) is used in
TraceLoggingProvider.h. This commit implements a no-op handler for
compatability, similar to how the flag -fexec_charset is handled.
Patch by Matt Gardner!
Differential Revision: https://reviews.llvm.org/D58530
llvm-svn: 356185
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Previously these would be transformed into annotation tokens and the
preprocessor would then assume they were real tokens with source
locations and assert/UB.
Other pragmas that produce annotation tokens aren't a problem because
they aren't handled if the parser isn't hooked up - ParsePragma.cpp
registers those handlers & isn't run for pure preprocessing. So they're
treated as unknown pragmas & printed verbatim by the preprocessor.
Perhaps these pragmas should be treated the same way? But they got mixed
in with other __debug pragmas that do need to be handled during
preprocessing.
The third __debug pragma that produces an annotation token is 'captured'
- which had its own fix for this issue - by not inserting the annotation
token in the first place if it detected that it was in preprocessing
mode. I've removed that fix (from Lex/Pragma.cpp) in favor of the more
general one in Frontend/PrintPreprocessedOutput.cpp.
llvm-svn: 346928
This commit relands r331904.
Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, and updating calls to that function. This will be useful
in https://reviews.llvm.org/D43778 to determine which includes are
system
headers.
Differential Revision: https://reviews.llvm.org/D46614
llvm-svn: 332021
Adding a SrcMgr::CharacteristicKind parameter to the InclusionDirective
in PPCallbacks, and updating calls to that function. This will be useful
in https://reviews.llvm.org/D43778 to determine which includes are system
headers.
Differential Revision: https://reviews.llvm.org/D46614
llvm-svn: 331904
This comes up when pre-processing standalone .s files containing
hash-prefixed comments. The pre-processor should skip the unknown
directive and not emit an extra newline as we were doing.
Fixes PR34950
llvm-svn: 315953
- Extracted the reading of the tokens out into a separate function.
- Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation).
This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible.
I will also directly update some extra clang tooling that is broken by the change from Argument to Parameter.
Hopefully the bots will stay appeased.
Thanks!
llvm-svn: 308190
- Extracted the reading of the tokens out into a separate function.
- Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation).
This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible.
Thanks!
llvm-svn: 308157
These pragmas are intended to simulate the effect of entering or leaving a file
with an associated module. This is not completely implemented yet: declarations
between the pragmas will not be attributed to the correct module, but macro
visibility is already functional.
Modules named by #pragma clang module begin must already be known to clang (in
some module map that's either loaded or on the search path).
llvm-svn: 302098
Many of our supported configurations support modules but do not have any
first-class syntax to perform a module import. This leaves us with a problem:
there is no way to represent the expansion of a #include that imports a module
in the -E output for such languages. (We don't want to just leave it as a
#include because that requires the consumer of the preprocessed source to have
the same file system layout and include paths as the creator.)
This patch adds a new pragma:
#pragma clang module import MODULE.NAME.HERE
that imports a module, and changes -E and -frewrite-includes to use it when
rewriting a #include that maps to a module import. We don't make any attempt
to use a native language syntax import if one exists, to get more consistent
output. (If in the future, @import and #include have different semantics in
some way, the pragma will track the #include semantics.)
llvm-svn: 301725
The UnknownPragmaHandlers added by DoPrintPreprocessedInput conflict with the
real PragmaHandlers from clang::Parser because they try to handle the same
#pragma directives. This makes it impossible to use a Preprocessor (that was
previously passed to DoPrintPreprocessedInput), as an Preprocessor for a
clang::Parser instance which is what we currently do in cling.
This patch removes the added UnknownPragmaHandler to avoid conflicts these
conflicts and leave the PragmaHandlers of the Preprocessors in a the same state
as before calling DoPrintPreprocessedInput.
Patch by Raphael Isemann (D32486)!
llvm-svn: 301563
This reverts commit r301449. It breaks the build with:
MacroPPCallbacks.h:114:50: error: non-virtual member function marked 'override' hides virtual member function
llvm-svn: 301469
Summary:
The PPCallbacks::MacroUndefined callback is currently insufficient for clients that need to track the MacroDirectives.
This patch adds an additional argument to PPCallbacks::MacroUndefined that is the undef MacroDirective.
Reviewers: bruno, manmanren
Reviewed By: bruno
Subscribers: nemanjai, cfe-commits
Differential Revision: https://reviews.llvm.org/D29923
llvm-svn: 301449
Re-introduce r285411.
Implement the -dI as supported by GCC: Output ‘#include’ directives in addition
to the result of preprocessing.
This change aims to add this option, pass it through to the preprocessor via
the options class, and when inclusions occur we output some information (+ test
cases).
Patch by Steve O'Brien!
Differential Revision: https://reviews.llvm.org/D26089
llvm-svn: 287275
Implement the -dI as supported by GCC: Output ‘#include’ directives in addition
to the result of preprocessing.
This change aims to add this option, pass it through to the preprocessor via
the options class, and when inclusions occur we output some information (+ test
cases).
Patch by Steve O'Brien!
Differential Revision: https://reviews.llvm.org/D25153
llvm-svn: 285411
While this won't help fix things like the bug that r260219 addressed, it
seems like good tidy up to have anyway.
(it might be nice if "makeArrayRef" always produced a MutableArrayRef &
let it decay to an ArrayRef when needed - then I'd use that for the
MutableArrayRefs in this patch)
If we had std::dynarray I'd use that instead of unique_ptr+size_t,
ideally (but then it'd have to be threaded down through the Preprocessor
all the way - no idea how painful that would be)
llvm-svn: 260246
Modules builds fundamentally have a non-linear macro history. In the interest
of better source fidelity, represent the macro definition information
faithfully: we have a linear macro directive history within each module, and at
any point we have a unique "latest" local macro directive and a collection of
visible imported directives. This also removes the attendent complexity of
attempting to create a correct MacroDirective history (which we got wrong
in the general case).
No functionality change intended.
llvm-svn: 236176
Previously we'd defer this determination until writing the AST, which doesn't
allow us to use this information when building other submodules of the same
module. This change also allows us to use a uniform mechanism for writing
module macro records, independent of whether they are local or imported.
llvm-svn: 235614
Currently -fms-extensions controls this behavior, which doesn't make
much sense. It means we can't identify what is and isn't a system header
when compiling our own preprocessed output, because #line doesn't
represent this information.
If someone is feeding Clang's preprocessed output to another compiler,
they can use this flag.
Fixes PR20553.
Reviewers: rsmith
Differential Revision: http://reviews.llvm.org/D5217
llvm-svn: 230587
1. We were hitting the NextIsPrevious assertion because we were trying
to merge decl chains that were independent of each other because we had
no Sema object to allow them to find existing decls. This is fixed by
delaying loading the "preloaded" decls until Sema is available.
2. We were trying to get identifier info from an annotation token, which
asserts. The fix is to special-case the module annotations in the
preprocessed output printer.
Fixed in a single commit because when you hit 1 you almost invariably
hit 2 as well.
llvm-svn: 217550
This begins to address cognitive dissonance caused by treating the Note
diagnostic level as a severity in the diagnostic engine.
No change in functionality.
llvm-svn: 210758
