This patch adopts `FileEntryRef` in the `HeaderFileInfo`-writing part of
`ASTWriter`.
First, this patch removes the loop over
`FileManager::VirtualFileEntries`. It's redundant, since all virtual
file entries are also present in `SeenFileEntries` and thus already in
`UIDToFiles`.
Second, since we now no longer rely on
`FileEntry::getLastRef()`/`FileEntry::getName()`, this patch takes care
to establish which path gets used for each UID by picking the
`FileEntryRef` with the most "`<`" name (instead of just relying on the
`StringMap` iteration order).
Note that which `FileEntry`/`FileEntryRef` objects we pick for each UID
for serialization into the `llvm::OnDiskChainedHashTable` doesn't really
matter. The hash function only includes the file size and modification
time. The file name only plays role during resolution of hash
collisions, in which case it goes through `FileManager` and resolves to
a `FileEntry` that gets pointer-compared with the queried `FileEntry`.
(Reincarnation of [D143414](https://reviews.llvm.org/D143414) and
[D142780](https://reviews.llvm.org/D142780).)
Running lit tests on Windows can fail because its use of
`os.path.realpath` expands substitute drives, which are used to keep
paths short and avoid hitting MAX_PATH limitations.
Changes lit logic to:
Use `os.path.abspath` on Windows, where `MAX_PATH` is a concern that we
can work around using substitute drives, which `os.path.realpath` would
resolve.
Use `os.path.realpath` on Unix, where the current directory always has
symlinks resolved, so it is impossible to preserve symlinks in the
presence of relative paths, and so we must make sure that all code paths
use real paths.
Also updates clang's `FileManager::getCanonicalName` and `ExtractAPI`
code to avoid resolving substitute drives (i.e. resolving to a path
under a different root).
How tested: built with `-DLLVM_ENABLE_PROJECTS=clang` and built `check-all` on both Windows
Differential Revision: https://reviews.llvm.org/D154130
Reviewed By: @benlangmuir
Patch by Tristan Labelle <tristan@thebrowser.company>!
This patch removes the last use of deprecated `DirectoryEntry::getName()`.
Depends on D151855.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D151922
For redirected file entries, `FileEntryRef::getDir()` returns the parent directory entry of the target file entry. This differs from `FileEntry::getDir()` that always returns the parent directory that was last used to look up that file.
After switching from `FileEntry` to `FileEntryRef` for umbrella headers in D142113, this discrepancy became observable and caused Clang to emit incorrect diagnostics.
This patch changes Clang so that it always associates `FileEntryRef` with the parent directory that was used to look it up. This brings its behavior closer to `FileEntry`, but without the hacky mutation.
This also ensures that `llvm::sys::path::parent_path(FileRef->getNameAsRequested()) == FileRef->getDir()->getName()`. Previously, `FileRef->getDir()` would fall underneath the redirecting VFS into the world of on-disk paths.
Reviewed By: benlangmuir, rmaz
Differential Revision: https://reviews.llvm.org/D151398
We no longer support GCC 5.1 (D122976), so let's remove the workaround in FileEntry.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D143427
The needed tweaks are mostly trivial, the one nasty bit is Clang's usage
of OptionalStorage. To keep this working old Optional stays around as
clang::CustomizableOptional, with the default Storage removed.
Optional<File/DirectoryEntryRef> is replaced with a typedef.
I tested this with GCC 7.5, the oldest supported GCC I had around.
Differential Revision: https://reviews.llvm.org/D140332
This reverts commit 8f0df9f3bbc6d7f3d5cbfd955c5ee4404c53a75d.
The Optional*RefDegradesTo*EntryPtr types want to keep the same size as
the underlying type, which std::optional doesn't guarantee. For use with
llvm::Optional, they define their own storage class, and there is no way
to do that in std::optional.
On top of that, that commit broke builds with older GCCs, where
std::optional was not trivially copyable (static_assert in the clang
sources was failing).
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
In 6a79e2ff1989b we changed Filemanager::getEntryRef() to return the
redirecting FileEntryRef instead of looking through the redirection.
This commit fixes the case when looking up a cached file path to also
return the redirecting FileEntryRef. This mainly affects the behaviour
of calling getNameAsRequested() on the resulting entry ref.
Differential Revision: https://reviews.llvm.org/D131273
As progress towards having FileManager::getFileRef() return the path
as-requested by default, return a FileEntryRef that can use
getNameAsRequested() to retrieve this path, with the ultimate goal that
this should be the behaviour of getName() and clients should explicitly
request the "external" name if they need to (see comment in
FileManager::getFileRef). For now, getName() continues to return the
external path by looking through the redirects.
For now, the new function is only used in unit tests.
Differential Revision: https://reviews.llvm.org/D131004
As progress towards having FileEntryRef contain the requested name of
the file, this commit narrows the "remap" hack to only apply to paths
that were remapped to an external contents path by a VFS. That was
always the original intent of this code, and the fact it was making
relative paths absolute was an unintended side effect.
Differential Revision: https://reviews.llvm.org/D130935
If the `ExternalFS` has already remapped to an external path then
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).
For now this is accomplished through the use of a new
`ExposesExternalVFSPath` field on `vfs::Status`. This flag is true when
the `Status` has an external path that's different from its virtual
path, ie. the contained path is the external path. See the plan in
`FileManager::getFileRef` for where this is going - eventually we won't
need `IsVFSMapped` any more and all returned paths should be virtual.
Resolves rdar://90578880 and llvm-project#53306.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D123398
- isValid: FileManager only ever returns valid FileEntries (see next point)
- construction from outside FileManager (both FileEntry and DirectoryEntry).
It's not possible to create a useful FileEntry this way, there are no setters.
This was only used in FileEntryTest, added a friend to enable this.
A real constructor is cleaner but requires larger changes to FileManager.
Differential Revision: https://reviews.llvm.org/D123197
This reverts commit 3fda0edc51fd68192a30e302d45db081bb02d7f9, which
breaks crash reproducers in very specific circumstances. Specifically,
since crash reproducers have `UseExternalNames` set to false, the
`File->getFileEntry().getDir()->getName()` call in `DoFrameworkLookup`
would use the *cached* directory name instead of the directory of the
looked-up file.
The plan is to re-commit this patch but to *add*
`ExposesExternalVFSPath` rather than replace `IsVFSMapped`.
Differential Revision: https://reviews.llvm.org/D123103
If the `ExternalFS` has already remapped a path then the
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).
This also renames `IsVFSMapped` to `ExposesExternalVFSPath` and only
sets it if `UseExternalName` is true. This flag then represents that the
`Status` has an external path that's different from its virtual path.
Right now the contained path is still the external path, but further PRs
will change this to *always* be the virtual path. Clients that need the
external can then request it specifically.
Note that even though `ExposesExternalVFSPath` isn't set for all
VFS-mapped paths, `IsVFSMapped` was only being used by a hack in
`FileManager` that was specific to module searching. In that case
`UseExternalNames` is always `true` and so that hack still applies.
Resolves rdar://90578880 and llvm-project#53306.
Differential Revision: https://reviews.llvm.org/D122549
Use the new sys::path::is_style_posix() and is_style_windows() in a few
places that need to detect the system's native path style.
In llvm/lib/Support/Path.cpp, this patch removes most uses of the
private `real_style()`, where is_style_posix() and is_style_windows()
are just a little tidier.
Elsewhere, this removes `_WIN32` macro checks. Added a FIXME to a
FileManagerTest that seemed fishy, but maintained the existing
behaviour.
Differential Revision: https://reviews.llvm.org/D112289
Document 'use-external-name' and the various bits of logic that make it
work, to avoid others having to repeat the archival work (given that I
added getFileRefReturnsCorrectNameForDifferentStatPath to
FileManagerTest, seems possible I understood this once before!).
- b59cf679e81483cbb3a9252056b7528f4c49586c added 'use-external-name' to
RedirectingFileSystem. This causes `stat`s to return the external
name for a redirected file instead of the name it was accessed by,
leaking it through the VFS.
- d066d4c849be06a01c0d17e8dc206913f4e7bfe3 propagated the external name
further through clang::FileManager.
- 4dc5573acc0d2e7c59d8bac2543eb25cb4b32984, which added
clang::FileEntryRef to clang::FileManager, has complicated concession
to account for this as well (since refactored a bit).
The goal of 'use-external-name' is to enable Clang to report "real" file
paths to users (via diagnostics) and to external tools (such as
debuggers reading debug info and build systems reading `.d` files).
I've added FIXMEs to look at other channels for communicating the
external names, since the current implementation adds complexity to
FileManager and exposes an inconsistent interface to clients.
Besides that, the FileManager logic appears to be kicking in outside of
'use-external-name'. Seems that *some* vfs::FileSystem implementations
canonicalize some paths returned by `stat` in *some* cases (the bug
isn't fully understood yet). Volodymyr Sapsai is investigating, this at
least better documents what *is* understood.
C++23 will make these conversions ambiguous - so fix them to make the
codebase forward-compatible with C++23 (& a follow-up change I've made
will make this ambiguous/invalid even in <C++23 so we don't regress
this & it generally improves the code anyway)
This is mostly a mechanical change, but a testcase that contains
parts of the StringRef class (clang/test/Analysis/llvm-conventions.cpp)
isn't touched.
After https://reviews.llvm.org/D90484 libclang is unable to read a serialized diagnostic file
which contains a diagnostic which came from a file with an empty filename. The reason being is
that the serialized diagnostic reader is creating a virtual file for the "" filename, which now
fails after the changes in https://reviews.llvm.org/D90484. This patch restores the previous
behavior in getVirtualFileRef by allowing it to construct a file entry ref with an empty name by
pretending its name is "." so that the directory entry can be created.
Differential Revision: https://reviews.llvm.org/D100428
Add support for stdin to SourceManager and FileManager. Adds
FileManager::getSTDIN, which adds a FileEntryRef for `<stdin>` and reads
the MemoryBuffer, which is stored as `FileEntry::Content`.
Eventually the other buffers in `ContentCache` will sink to here as well
-- we probably usually want to load/save a MemoryBuffer eagerly -- but
it's happening early for stdin to get rid of
CompilerInstance::InitializeSourceManager's final call to
`SourceManager::overrideFileContents`.
clang/test/CXX/modules-ts/dcl.dcl/dcl.module/dcl.module.export/p1.cpp
relies on building a module from stdin; supporting that requires setting
ContentCache::BufferOverridden.
Differential Revision: https://reviews.llvm.org/D93148
Handle named pipes natively in SourceManager and FileManager, removing a
call to `SourceManager::overrideFileContents` in
`CompilerInstance::InitializeSourceManager` (removing a blocker for
sinking the content cache to FileManager (which will incidently sink
this new named pipe logic with it)).
SourceManager usually checks if the file entry's size matches the
eventually loaded buffer, but that's now skipped for named pipes since
the `stat` won't reflect the full size. Since we can't trust
`ContentsEntry->getSize()`, we also need shift the check for files that
are too large until after the buffer is loaded... and load the buffer
immediately in `createFileID` so that no client gets a bad value from
`ContentCache::getSize`. `FileManager::getBufferForFile` also needs to
treat these files as volatile when loading the buffer.
Native support in SourceManager / FileManager means that named pipes can
also be `#include`d, and clang/test/Misc/dev-fd-fs.c was expanded to
check for that.
This is a new version of 3b18a594c7717a328c33b9c1eba675e9f4bd367c, which
was reverted in b34632201987eed369bb7ef4646f341b901c95b8 since it was
missing the `SourceManager` changes.
Differential Revision: https://reviews.llvm.org/D92531
This reverts commit 3b18a594c7717a328c33b9c1eba675e9f4bd367c, since
apparently this doesn't work everywhere. E.g.,
clang-x86_64-debian-fast/3889
(http://lab.llvm.org:8011/#/builders/109/builds/3889) gives me:
```
+ : 'RUN: at line 8'
+ /b/1/clang-x86_64-debian-fast/llvm.obj/bin/clang -x c /dev/fd/0 -E
+ cat /b/1/clang-x86_64-debian-fast/llvm.src/clang/test/Misc/dev-fd-fs.c
fatal error: file '/dev/fd/0' modified since it was first processed
1 error generated.
```
Remove compilicated logic from CompilerInstance::InitializeSourceManager
to deal with named pipes, updating FileManager::getBufferForFile to
handle it in a more straightforward way. The existing test at
clang/test/Misc/dev-fd-fs.c covers the new behaviour (just like it did
the old behaviour).
Differential Revision: https://reviews.llvm.org/D90733
Add `FileEntryRef::getDir`, which returns a `DirectoryEntryRef`. This
includes a few changes:
- Customize `OptionalStorage` so that `Optional<DirectoryEntryRef>` is
pointer-sized (like the change made to `Optional<FileEntryRef>`).
Factored out a common class, `FileMgr::MapEntryOptionalStorage`, to
reduce the code duplication.
- Store an `Optional<DirectoryEntryRef>` in `FileEntryRef::MapValue`.
This is set if and only if `MapValue` has a real `FileEntry`.
- Change `FileManager::getFileRef` and `getVirtualFileRef` to use
`getDirectoryRef` and store it in the `StringMap` for `FileEntryRef`.
Differential Revision: https://reviews.llvm.org/D90484
Change the `InputFile` class to store `Optional<FileEntryRef>` instead
of `FileEntry*`. This paged in a few API changes:
- Added `FileManager::getVirtualFileRef`, and converted `getVirtualFile`
to a wrapper of it.
- Updated `SourceManager::bypassFileContentsOverride` to take
`FileEntryRef` and return `Optional<FileEntryRef>`
(`ASTReader::getInputFile` is the only caller).
Differential Revision: https://reviews.llvm.org/D90053
Shrink `FileEntryRef` to the size of a pointer, by having it directly
reference the `StringMapEntry` the same way that `DirectoryEntryRef`
does. This makes `FileEntryRef::FileEntryRef` private as a side effect
(`FileManager` is a friend!).
There are two helper types added within `FileEntryRef`:
- `FileEntryRef::MapValue` is the type stored in
`FileManager::SeenFileEntries`. It's a replacement for
`SeenFileEntryOrRedirect`, where the second pointer type has been
changed from `StringRef*` to `MapEntry*` (see next bullet).
- `FileEntryRef::MapEntry` is the instantiation of `StringMapEntry<>`
where `MapValue` is stored. This is what `FileEntryRef` has a pointer
to, in order to grab the name in addition to the value.
Differential Revision: https://reviews.llvm.org/D89488
This is needed to fix the reason
0a2be46cfdb698fe (Modules: Invalidate out-of-date PCMs as they're
discovered) and 5b44a4b07fc1d ([modules] Do not cache invalid state for
modules that we attempted to load.) were reverted.
These patches changed Clang to use `isVolatile` when loading modules.
This had the side effect of not using mmap when loading modules, and
thus greatly increased memory usage.
The reason it wasn't using mmap is because `MemoryBuffer` plays some
games with file size when you request null termination, and it has to
disable these when `isVolatile` is set as the size may change by the
time it's mmapped. Clang by default passes
`RequiresNullTerminator = true`, and `shouldUseMmap` ignored if
`RequiresNullTerminator` was even requested.
This patch adds `RequiresNullTerminator` to the `FileManager` interface
so Clang can use it when loading modules, and changes `shouldUseMmap` to
only take volatility into account if `RequiresNullTerminator` is true.
This is fine as both `mmap` and a `read` loop are vulnerable to
modifying the file while reading, but are immune to the rename Clang
does when replacing a module file.
Differential Revision: https://reviews.llvm.org/D77772
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
In the current implementation of clang the canonicalization of paths in
diagnostic messages (when using -fdiagnostics-absolute-paths) only works
if the symbolic link is in the directory part of the filename, not if
the file itself is a symbolic link to another file.
This patch adds support to canonicalize the complete path including the
file.
Reviewers: rsmith, hans, rnk, ikudrin
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D70527
accessed name to the directory entry
This commit introduces a parallel API that returns a DirectoryEntryRef
to the FileManager, similar to the parallel FileEntryRef API. All
uses will have to be update in follow-up patches. The immediate use of the new API in this
patch fixes the issue where a file manager was reused in clang-scan-deps,
but reported an different file path whenever a framework lookup was done through a symlink.
Differential Revision: https://reviews.llvm.org/D67026
llvm-svn: 370562
If contents of a file that is part of a PCM are overridden when reading
it, but weren't overridden when the PCM was being built, the ASTReader
will emit an error. Now it creates a separate FileEntry for recovery,
bypassing the overridden content instead of discarding it. The
pre-existing testcase clang/test/PCH/remap-file-from-pch.cpp confirms
that the new recovery method works correctly.
This resolves a long-standing FIXME to avoid hypothetically invalidating
another precompiled module that's already using the overridden contents.
This also removes ContentCache-related API that would be unsafe to use
across `CompilerInstance`s in an implicit modules build. This helps to
unblock us sinking it from SourceManager into FileManager in the future,
which would allow us to delete `InMemoryModuleCache`.
https://reviews.llvm.org/D66710
llvm-svn: 370546
`FileManager::getFileRef` is a modern API which we expect to convert to
over time. We should modernize the error handling as well, using
`llvm::Expected` instead of `llvm::ErrorOr`, to help clients that care
about errors to ensure nothing is missed.
However, not all clients care. I've also added another path for those
that don't:
- `FileEntryRef` is now copy- and move-assignable (using a pointer
instead of a reference).
- `FileManager::getOptionalFileRef` returns an `llvm::Optional` instead
of `llvm::Expected`.
- Added an `llvm::expectedToOptional` utility in case this is useful
elsewhere.
https://reviews.llvm.org/D66705
llvm-svn: 369943
when the FileManager is reused across invocations
This commit introduces a parallel API to FileManager's getFile: getFileEntryRef, which returns
a reference to the FileEntry, and the name that was used to access the file. In the case of
a VFS with 'use-external-names', the FileEntyRef contains the external name of the file,
not the filename that was used to access it.
The new API is adopted only in the HeaderSearch and Preprocessor for include file lookup, so that the
accessed path can be propagated to SourceManager's FileInfo. SourceManager's FileInfo now can report this accessed path, using
the new getName method. This API is then adopted in the dependency collector, which now correctly reports dependencies when a file
is included both using a symlink and a real path in the case when the FileManager is reused across multiple Preprocessor invocations.
Note that this patch does not fix all dependency collector issues, as the same problem is still present in other cases when dependencies
are obtained using FileSkipped, InclusionDirective, and HasInclude. This will be fixed in follow-up commits.
Differential Revision: https://reviews.llvm.org/D65907
llvm-svn: 369680
