FileOutputBuffer creates a temp file and on commit atomically
renames the temp file to the destination file. Sometimes we
want to modify an existing file in place, but still have the
atomicity guarantee. To do this we can initialize the contents
of the temp file from the destination file (if it exists), that
way the resulting FileOutputBuffer can have only selective
bytes modified. Committing will then atomically replace the
destination file as desired.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@335902 91177308-0d34-0410-b5e6-96231b3b80d8
On Windows we've observed that if you open a file, write to it, map it into
memory and close the file handle, the contents of the memory mapping can
sometimes be incorrect. That was what we did when adding an entry to the
ThinLTO cache using the TempFile and MemoryBuffer classes, and it was causing
intermittent build failures on Chromium's ThinLTO bots on Windows. More
details are in the associated Chromium bug (crbug.com/786127).
We can prevent this from happening by keeping a handle to the file open while
the mapping is active. So this patch changes the mapped_file_region class to
duplicate the file handle when mapping the file and close it upon unmapping it.
One gotcha is that the file handle that we keep open must not have been
created with FILE_FLAG_DELETE_ON_CLOSE, as otherwise the operating system
will prevent other processes from opening the file. We can achieve this
by avoiding the use of FILE_FLAG_DELETE_ON_CLOSE altogether. Instead,
we use SetFileInformationByHandle with FileDispositionInfo to manage the
delete-on-close bit. This lets us remove the hack that we used to use to
clear the delete-on-close bit on a file opened with FILE_FLAG_DELETE_ON_CLOSE.
A downside of using SetFileInformationByHandle/FileDispositionInfo as
opposed to FILE_FLAG_DELETE_ON_CLOSE is that it prevents us from using
CreateFile to open the file while the flag is set, even within the same
process. This doesn't seem to matter for almost every client of TempFile,
except for LockFileManager, which calls sys::fs::create_link to create a
hard link from the lock file, and in the process of doing so tries to open
the file. To prevent this change from breaking LockFileManager I changed it
to stop using TempFile by effectively reverting r318550.
Differential Revision: https://reviews.llvm.org/D48051
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334630 91177308-0d34-0410-b5e6-96231b3b80d8
O_CLOEXEC is the right default, but occasionally you don't
want this. This is especially true for tools like debuggers
where you might need to spawn the child process with specific
files already open, but it's occasionally useful in other
scenarios as well, like when you want to do some IPC between
parent and child.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334293 91177308-0d34-0410-b5e6-96231b3b80d8
This one allows much more flexibility than the standard
openFileForRead / openFileForWrite functions. Since there is now
just one "real" function that does the work, all other implementations
simply delegate to this one.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334246 91177308-0d34-0410-b5e6-96231b3b80d8
This breaks the OpenFlags enumeration into two separate
enumerations: OpenFlags and CreationDisposition. The first
controls the behavior of the API depending on whether or not
the target file already exists, and is not a flags-based
enum. The second controls more flags-like values.
This yields a more easy to understand API, while also allowing
flags to be passed to the openForRead api, where most of the
values didn't make sense before. This also makes the apis more
testable as it becomes easy to enumerate all the configurations
which make sense, so I've added many new tests to exercise all
the different values.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334221 91177308-0d34-0410-b5e6-96231b3b80d8
There was only one place in the entire codebase where a non
default value was being passed, and that place was already hidden
in an implementation file. So we can delete the extra parameter
and all existing clients continue to work as they always have,
while making the interface a bit simpler.
Differential Revision: https://reviews.llvm.org/D47789
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334046 91177308-0d34-0410-b5e6-96231b3b80d8
Windows' CRT has a limit of 512 open file descriptors, and fds which are
generated by converting a HANDLE via _get_osfhandle count towards this
limit as well.
Regardless, often you find yourself marshalling back and forth between
native HANDLE objects and fds anyway. If we know from the getgo that
we're going to need to work directly with the handle, we can cut out the
marshalling layer while also not contributing to filling up the CRT's
very limited handle table.
On Unix these functions just delegate directly to the existing set of
functions since an fd *is* the native file type. It would be nice, very
long term, if we could convert most uses of fds to file_t.
Differential Revision: https://reviews.llvm.org/D47688
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@333945 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Add documentation for the LLVM Support functions `openFileForWrite` and
`openFileForRead`. The `openFileForRead` parameter `RealPath`, in
particular, I think warranted some explanation.
In addition, make the behavior of the functions more consistent across
platforms. Prior to this patch, Windows would set or not set the result
file descriptor based on the nature of the error, whereas Unix would
consistently set it to `-1` if the open failed. Make Windows
consistently set it to `-1` as well.
Test Plan:
1. `ninja check-llvm`
2. `ninja docs-llvm-html`
Reviewers: zturner, rnk, danielmartin, scanon
Reviewed By: danielmartin, scanon
Subscribers: scanon, danielmartin, llvm-commits
Differential Revision: https://reviews.llvm.org/D46499
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@332075 91177308-0d34-0410-b5e6-96231b3b80d8
This is a follow-up to r331272.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done
https://reviews.llvm.org/D46290
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331275 91177308-0d34-0410-b5e6-96231b3b80d8
See r331124 for how I made a list of files missing the include.
I then ran this Python script:
for f in open('filelist.txt'):
f = f.strip()
fl = open(f).readlines()
found = False
for i in xrange(len(fl)):
p = '#include "llvm/'
if not fl[i].startswith(p):
continue
if fl[i][len(p):] > 'Config':
fl.insert(i, '#include "llvm/Config/llvm-config.h"\n')
found = True
break
if not found:
print 'not found', f
else:
open(f, 'w').write(''.join(fl))
and then looked through everything with `svn diff | diffstat -l | xargs -n 1000 gvim -p`
and tried to fix include ordering and whatnot.
No intended behavior change.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331184 91177308-0d34-0410-b5e6-96231b3b80d8
LLVM_ON_WIN32 is set exactly with MSVC and MinGW (but not Cygwin) in
HandleLLVMOptions.cmake, which is where _WIN32 defined too. Just use the
default macro instead of a reinvented one.
See thread "Replacing LLVM_ON_WIN32 with just _WIN32" on llvm-dev and cfe-dev.
No intended behavior change.
This moves over all uses of the macro, but doesn't remove the definition
of it in (llvm-)config.h yet.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@331127 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Previous code hangs indefinitely when trying to iterate through a
symbol link file that points to an non-exist directory. This change
fixes the bug to make the addCollectedPath function exit ealier and
print out correct warning messages.
Patch by Yuke Liao (@liaoyuke).
Reviewers: Dor1s, vsk
Reviewed By: vsk
Subscribers: bruno, mgrang, llvm-commits
Differential Revision: https://reviews.llvm.org/D44960
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@329338 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
This commit changes semantics of createUniqueFile and
createTemporaryFile variants that do not return file descriptors.
Previously they only checked if files exist, therefore being subject
to race conditions. Now they will create an empty file to avoid them.
Functions that do not create a file are now called
getPotentiallyUniqueTempFileName and getPotentiallyUniqueFileName.
Reviewers: klimek, bkramer, krasimir, JDevlieghere, espindola
Reviewed By: klimek
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36827
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@327851 91177308-0d34-0410-b5e6-96231b3b80d8
This is like MemoryBuffer (read-only) and WritableMemoryBuffer
(writable private), but where the underlying file can be modified
after writing. This is useful when you want to open a file, make
some targeted edits, and then write it back out.
Differential Revision: https://reviews.llvm.org/D44230
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@327057 91177308-0d34-0410-b5e6-96231b3b80d8
There is a latent Windows kernel bug, the exact trigger
conditions are not well understood, which can cause a file
to be correctly written, but unable to be correctly read.
The workaround appears to be simply calling FlushFileBuffers.
Differential Revision: https://reviews.llvm.org/D42925
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@325274 91177308-0d34-0410-b5e6-96231b3b80d8
This will allow a future F_Delete flag to be specified when we want
the file to be automatically deleted on close.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@319117 91177308-0d34-0410-b5e6-96231b3b80d8
This one requires a new small feature in TempFile: the ability to keep
the temporary file with the temporary name.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318458 91177308-0d34-0410-b5e6-96231b3b80d8
This just adds a TempFile class and replaces the use in
FileOutputBuffer with it.
The only difference for now is better error handling. Followup work includes:
- Convert other user of temporary files to it.
- Add support for automatically deleting on windows.
- Add a createUnnamed method that returns a potentially unnamed
file. It would be actually unnamed on modern linux and have a
unknown name on windows.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@318069 91177308-0d34-0410-b5e6-96231b3b80d8
This allows clients to avoid an unnecessary fs::status() call on each
directory entry. Because the information returned by FindFirstFileEx
is a subset of the information returned by a regular status() call,
I needed to extract a base class from file_status that contains only
that information.
On my machine, this reduces the time required to enumerate a ThinLTO
cache directory containing 520k files from almost 4 minutes to less
than 2 seconds.
Differential Revision: https://reviews.llvm.org/D38716
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315378 91177308-0d34-0410-b5e6-96231b3b80d8
The current implementation of rename uses ReplaceFile if the
destination file already exists. According to the documentation for
ReplaceFile, the source file is opened without a sharing mode. This
means that there is a short interval of time between when ReplaceFile
renames the file and when it closes the file during which the
destination file cannot be opened.
This behaviour is not POSIX compliant because rename is supposed
to be atomic. It was also causing intermittent link failures when
linking with a ThinLTO cache; the ThinLTO cache implementation expects
all cache files to be openable.
This patch addresses that problem by re-implementing rename
using CreateFile and SetFileInformationByHandle. It is roughly a
reimplementation of ReplaceFile with a better sharing policy as well
as support for renaming in the case where the destination file does
not exist.
This implementation is still not fully POSIX. Specifically in the case
where the destination file is open at the point when rename is called,
there will be a short interval of time during which the destination
file will not exist. It isn't clear whether it is possible to avoid
this using the Windows API.
Differential Revision: https://reviews.llvm.org/D38570
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315079 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Found when testing stage-2 build with D38101.
```
In file included from /build/llvm/lib/Support/Path.cpp:1045:
/build/llvm/lib/Support/Unix/Path.inc:648:14: error: comparison 'uint64_t' (aka 'unsigned long') > 18446744073709551615 is always false [-Werror,-Wtautological-constant-compare]
if (length > std::numeric_limits<size_t>::max()) {
~~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
`size_t` is `uint64_t` here, apparently, thus any `uint64_t` value
always fits into `size_t`.
Initial patch was to use some preprocessor logic to
not check if the size is known to fit at compile time.
But Zachary Turner suggested using this approach.
Reviewers: Bigcheese, rafael, zturner, mehdi_amini
Reviewed by (via email): zturner
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D38132
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@314312 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
It was added to support clang warnings about includes with case
mismatches, but it ended up not being necessary.
Reviewers: twoh, rafael
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D36328
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@310078 91177308-0d34-0410-b5e6-96231b3b80d8
This creates a new library called BinaryFormat that has all of
the headers from llvm/Support containing structure and layout
definitions for various types of binary formats like dwarf, coff,
elf, etc as well as the code for identifying a file from its
magic.
Differential Revision: https://reviews.llvm.org/D33843
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@304864 91177308-0d34-0410-b5e6-96231b3b80d8
This was exposed in r297945 and r301220: the intermediate complement
is a 32-bit value, and casting it to 'perms' invokes UB.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@301373 91177308-0d34-0410-b5e6-96231b3b80d8
In doing so, clean up the MD5 interface a little. Most
existing users only care about the lower 8 bytes of an MD5,
but for some users that care about the upper and lower,
there wasn't a good interface. Furthermore, consumers
of the MD5 checksum were required to handle endianness
details on their own, so it seems reasonable to abstract
this into a nicer interface that just gives you the right
value.
Differential Revision: https://reviews.llvm.org/D31105
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@298322 91177308-0d34-0410-b5e6-96231b3b80d8
This change adds support for functions to set and get file permissions, in a similar manner to the C++17 permissions() function in <filesystem>. The setter uses chmod on Unix systems and SetFileAttributes on Windows, setting the permissions as passed in. The getter simply uses the existing status() function.
Prior to this change, status() would always return an unknown value for the permissions on a Windows file, making it impossible to test the new function on Windows. I have therefore added support for this as well. On Linux, prior to this change, the permissions included the file type, which should actually be accessed via a different member of the file_status class.
Note that on Windows, only the *_write permission bits have any affect - if any are set, the file is writable, and if not, the file is read-only. This is in common with what MSDN describes for their behaviour of std::filesystem::permissions(), and also what boost::filesystem does.
The motivation behind this change is so that we can easily test behaviour on read-only files in LLVM unit tests, but I am sure that others may find it useful in some situations.
Reviewers: zturner, amccarth, aaron.ballman
Differential Revision: https://reviews.llvm.org/D30736
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@297945 91177308-0d34-0410-b5e6-96231b3b80d8
LLVM already has real_path like functionality, but it is
cumbersome to use and involves clean up after (e.g. you have
to call openFileForRead, then close the resulting FD).
Furthermore, on Windows it doesn't work for directories since
opening a directory and opening a file require slightly
different flags.
So I add a simple function `real_path` which works for all
paths on all platforms and has a simple to use interface.
In doing so, I add the ability to opt in to resolving tilde
expressions (e.g. ~/foo), which are normally handled by
the shell.
Differential Revision: https://reviews.llvm.org/D30668
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@297483 91177308-0d34-0410-b5e6-96231b3b80d8
We already have a function create_directories() which can create
an entire tree, and remove() which can remove an empty directory,
but we do not have remove_directories() which can remove an entire
tree. This patch adds such a function.
Because removing a directory tree can have dangerous consequences
when the tree contains a directory symlink, the patch here updates
the existing directory_iterator construct to optionally not follow
symlinks (previously it would always follow symlinks). The delete
algorithm uses this flag so that for symlinks, only the links are
removed, and not the targets.
On Windows this is implemented with SHFileOperation, which also
does not recurse into symbolic links or junctions.
Differential Revision: https://reviews.llvm.org/D30676
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@297314 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
This adds a cross-platform way of setting the current working directory
analogous to the existing current_path() function used for retrieving
it. The function will be used in lldb.
Reviewers: rafael, silvas, zturner
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D29035
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@292907 91177308-0d34-0410-b5e6-96231b3b80d8
If this is a problem for anyone (shared_ptr is two pointers in size,
whereas IntrusiveRefCntPtr is 1 - and the ref count control block that
make_shared adds is probably larger than the one int in RefCountedBase)
I'd prefer to address this by adding a lower-overhead version of
shared_ptr (possibly refactoring IntrusiveRefCntPtr into such a thing)
to avoid the intrusiveness - this allows memory ownership to remain
orthogonal to types and at least to me, seems to make code easier to
understand (since no implicit ownership acquisition can happen).
This recommits 291006, reverted in r291007.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@291016 91177308-0d34-0410-b5e6-96231b3b80d8
Breaks Clang's use of bitcode. Reverting until I have a fix to go with
it there.
This reverts commit r291006.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@291007 91177308-0d34-0410-b5e6-96231b3b80d8
If this is a problem for anyone (shared_ptr is two pointers in size,
whereas IntrusiveRefCntPtr is 1 - and the ref count control block that
make_shared adds is probably larger than the one int in RefCountedBase)
I'd prefer to address this by adding a lower-overhead version of
shared_ptr (possibly refactoring IntrusiveRefCntPtr into such a thing)
to avoid the intrusiveness - this allows memory ownership to remain
orthogonal to types and at least to me, seems to make code easier to
understand (since no implicit ownership acquisition can happen).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@291006 91177308-0d34-0410-b5e6-96231b3b80d8
