After r361885, realPathFromHandle() ends up getting called on the working
directory on each Clang invocation. This unveiled that the code didn't work for
paths on network shares.
For example, if one maps the local dir c:\src\tmp to x:
net use x: \\localhost\c$\tmp
and run e.g. "clang -c foo.cc" in x:\, realPathFromHandle will get
\\?\UNC\localhost\c$\src\tmp\ back from GetFinalPathNameByHandleW, and would
strip off the initial \\?\ prefix, ending up with a path that doesn't work.
This patch makes the prefix stripping a little smarter to handle this case.
Differential revision: https://reviews.llvm.org/D67166
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@371035 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
There was a subtle, but pretty important difference between the Slice
and regular versions of this function. The Slice function was
zero-initializing the rest of the buffer when the read syscall returned
less bytes than expected, while the regular function did not.
This patch removes the inconsistency by making both functions *not*
zero-initialize the buffer. The zeroing code is moved to the
MemoryBuffer class, which is currently the only user of this code. This
makes the API more consistent, and the code shorter.
While in there, I also refactor the functions to return the number of
bytes through the regular return value (via Expected<size_t>) instead of
a separate by-ref argument.
Reviewers: aganea, rnk
Subscribers: kristina, Bigcheese, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D66471
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@369627 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
The windows version implementation of readNativeFileSlice, was trying to
match the POSIX behavior of not treating EOF as an error, but it was
only handling the case of reading from a pipe. Attempting to read past
the end of a regular file returns a slightly different error code, which
needs to be handled too. This patch adds ERROR_HANDLE_EOF to the list of
error codes to be treated as an end of file, and adds some unit tests
for the API.
This issue was found while attempting to land D66224, which caused a bunch of
lldb tests to start failing on windows.
Reviewers: rnk, aganea
Subscribers: kristina, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D66344
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@369269 91177308-0d34-0410-b5e6-96231b3b80d8
This should have the same semantics. We use std::shared_mutex instead on
MSVC and C++17, std::shared_timed_mutex is less efficient than our
custom implementation on Windows, std::shared_mutex should be faster.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@369018 91177308-0d34-0410-b5e6-96231b3b80d8
- Remove support for non-recursive mutexes. This was unused.
- The std::recursive_mutex is now created/destroyed unconditionally.
Locking is still only done if threading is enabled.
- Alias SmartScopedLock to std::lock_guard.
This should make no semantic difference on the existing APIs.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@368158 91177308-0d34-0410-b5e6-96231b3b80d8
Support SIGINFO (and SIGUSR1 for POSIX purposes) to tell what
long-running jobs are doing, as inspired by BSD tools (including on
macOS), by dumping the current PrettyStackTrace.
This adds a new kind of signal handler for non-fatal "info" signals,
similar to the "interrupt" handler that already exists for SIGINT
(Ctrl-C). It then uses that handler to update a "generation count"
managed by the PrettyStackTrace infrastructure, which is then checked
whenever a PrettyStackTraceEntry is pushed or popped on each
thread. If the generation has changed---i.e. if the user has pressed
Ctrl-T---the stack trace is dumped, though unfortunately it can't
include the deepest entry because that one is currently being
constructed/destructed.
https://reviews.llvm.org/D63750
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@365911 91177308-0d34-0410-b5e6-96231b3b80d8
There is currently an EPERM error when a regular user executes `llvm-objcopy a.o /dev/null`.
Worse, root can even change the mode bits of /dev/null.
Fix it by checking if the output file is special.
A new overload of llvm::sys::fs::setPermissions with FD as the parameter
is added. Users should provide `perm & ~umask` as the parameter if they
intend to respect umask.
The existing overload of llvm::sys::fs::setPermissions may be deleted if
we can find an implementation of fchmod() on Windows. fchmod() is
usually better than chmod() because it saves syscalls and can avoid race
condition.
Reviewed By: jakehehrlich, jhenderson
Differential Revision: https://reviews.llvm.org/D64236
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@365753 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
On Windows, Posix integer file descriptors are a compatibility layer
over native file handles provided by the C runtime. There is a hard
limit on the maximum number of file descriptors that a process can open,
and the limit is 8192. LLD typically doesn't run into this limit because
it opens input files, maps them into memory, and then immediately closes
the file descriptor. This prevents it from running out of FDs.
For various reasons, I'd like to open handles to every input file and
keep them open during linking. That requires migrating MemoryBuffer over
to taking open native file handles instead of integer FDs.
Reviewers: aganea, Bigcheese
Reviewed By: aganea
Subscribers: smeenai, silvas, mehdi_amini, hiraditya, steven_wu, dexonsmith, dang, llvm-commits, zturner
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D63453
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@365588 91177308-0d34-0410-b5e6-96231b3b80d8
Summary: This patch changes fs::setPermissions to optionally set permissions while respecting the umask. It also adds the function fs::getUmask() which returns the current umask.
Reviewers: jhenderson, rupprecht, aprantl, lhames
Reviewed By: jhenderson, rupprecht
Subscribers: sanaanajjar231288, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D63583
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@364621 91177308-0d34-0410-b5e6-96231b3b80d8
Rename member 'Size' to 'AllocatedSize' in order to provide a hint that the
allocated size may be different than the requested size. Comments are added to
clarify this point. Updated the InMemoryBuffer in FileOutputBuffer.cpp to track
the requested buffer size.
Patch by Machiel van Hooren. Thanks Machiel!
https://reviews.llvm.org/D61599
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@361195 91177308-0d34-0410-b5e6-96231b3b80d8
This patch changes the return type of sys::Process::getPageSize to
Expected<unsigned> to account for the fact that the underlying syscalls used to
obtain the page size may fail (see below).
For clients who use the page size as an optimization only this patch adds a new
method, getPageSizeEstimate, which calls through to getPageSize but discards
any error returned and substitues a "reasonable" page size estimate estimate
instead. All existing LLVM clients are updated to call getPageSizeEstimate
rather than getPageSize.
On Unix, sys::Process::getPageSize is implemented in terms of getpagesize or
sysconf, depending on which macros are set. The sysconf call is documented to
return -1 on failure. On Darwin getpagesize is implemented in terms of sysconf
and may also fail (though the manpage documentation does not mention this).
These failures have been observed in practice when highly restrictive sandbox
permissions have been applied. Without this patch, the result is that
getPageSize returns -1, which wreaks havoc on any subsequent code that was
assuming a sane page size value.
<rdar://problem/41654857>
Reviewers: dblaikie, echristo
Subscribers: kristina, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D59107
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@360221 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
We have a multi-platform thread priority setting function(last piece
landed with D58683), I wanted to make this available to all llvm community,
there seem to be other users of such functionality with portability fixmes:
lib/Support/CrashRecoveryContext.cpp
tools/clang/tools/libclang/CIndex.cpp
Reviewers: gribozavr, ioeric
Subscribers: krytarowski, jfb, kristina, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D59130
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@358494 91177308-0d34-0410-b5e6-96231b3b80d8
This patch mirrors the change made to the Unix equivalent in
r351916. This in turn fixes bugs related to the use of FileOutputBuffer
to output to "-", i.e. stdout, on Windows.
Differential Revision: https://reviews.llvm.org/D59663
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@357058 91177308-0d34-0410-b5e6-96231b3b80d8
This patch introduces Memory::MF_HUGE_HINT which indicates that allocateMappedMemory() shall return a pointer to a large memory page.
However the flag is a hint because we're not guaranteed in any way that we will get back a large memory page. There are several restrictions:
- Large/huge memory pages aren't enabled by default on modern OSes (Windows 10 and Linux at least), and should be manually enabled/reserved.
- Once enabled, it should be kept in mind that large pages are physical only, they can't be swapped.
- Memory fragmentation can affect the availability of large pages, especially after running the OS for a long time and/or running along many other applications.
Memory::allocateMappedMemory() will fallback to 4KB pages if it can't allocate 2MB large pages (if Memory::MF_HUGE_HINT is provided)
Currently, Memory::MF_HUGE_HINT only works on Windows. The hint will be ignored on Linux, 4KB pages will always be returned.
Differential Revision: https://reviews.llvm.org/D58718
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@355065 91177308-0d34-0410-b5e6-96231b3b80d8
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351636 91177308-0d34-0410-b5e6-96231b3b80d8
This generates -Wnonportable-include-dir warnings, and doesn't need
to be there. It seems this was just checked in on accident.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@350655 91177308-0d34-0410-b5e6-96231b3b80d8
In D54435 there was some discussion about the expand_tilde flag for
real_path that I wanted to expose through the VFS. The consensus is that
these two things should be separate functions. Since we already have the
code for this I went ahead and added a function expand_tilde that does
just that.
Differential revision: https://reviews.llvm.org/D54448
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@346776 91177308-0d34-0410-b5e6-96231b3b80d8
Sink Windows version detection code from WindowsSupport.h to Path.inc.
These functions don't need to be inlined. I randomly picked Process.inc
for the Windows version helpers, since that's the most related file.
Sink MakeErrMsg to Program.inc since it's the main client.
Move those functions into the llvm namespace, and delete the scoped
handle copy and assignment operators.
Reviewers: zturner, aganea
Differential Revision: https://reviews.llvm.org/D54182
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@346280 91177308-0d34-0410-b5e6-96231b3b80d8
This is a follow-up for "r325274: Call FlushFileBuffers on output files."
Previously, FlushFileBuffers() was called in all cases when writing a file. The objective was to go around a bug in the Windows kernel (as described here: https://randomascii.wordpress.com/2018/02/25/compiler-bug-linker-bug-windows-kernel-bug/). However that is required only when writing EXEs, any other file type doesn't need flushing.
This patch calls FlushFileBuffers() only for EXEs. In addition, we completly disable FlushFileBuffers() for known Windows 10 versions that do not exhibit the original kernel bug.
Differential Revision: https://reviews.llvm.org/D53727
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@346152 91177308-0d34-0410-b5e6-96231b3b80d8
This is available on most platforms (Linux/Mac/Win/BSD) with no extra syscalls.
On other platforms (e.g. Solaris) we stat() if this information is requested.
This will allow switching clang's VFS to efficiently expose (path, type) when
traversing a directory. Currently it exposes an entire Status, but does so by
calling fs::status() on all platforms.
Almost all callers only need the path, and all callers only need (path, type).
Patch by sammccall (Sam McCall)
Differential Revision: https://reviews.llvm.org/D51918
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@342089 91177308-0d34-0410-b5e6-96231b3b80d8
Fixes at_file.c test failure caused by r341988. We may want to change
how we treat \n in our tokenizer, but this is probably a good fix
regardless, since we can invoke all kinds of programs with different
interpretations of the command line quoting rules.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@341992 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Shell32.dll depends on gdi32.dll and user32.dll, which are mostly DLLs
for Windows GUI functionality. LLVM's utilities don't typically need GUI
functionality, and loading these DLLs seems to be slowing down startup.
Also, we already have an implementation of Windows command line
tokenization in cl::TokenizeWindowsCommandLine, so we can just use it.
The goal is to get the original argv in UTF-8, so that it can pass
through most LLVM string APIs. A Windows process starts life with a
UTF-16 string for its command line, and it can be retreived with
GetCommandLineW from kernel32.dll.
Previously, we would:
1. Get the wide command line
2. Call CommandLineToArgvW to handle quoting rules and separate it into
arguments.
3. For each wide argument, expand wildcards (* and ?) using
FindFirstFileW.
4. Convert each argument to UTF-8
Now we:
1. Get the wide command line, convert the whole thing to UTF-8
2. Tokenize the UTF-8 command line with cl::TokenizeWindowsCommandLine
3. For each argument, expand wildcards if present
- This requires converting back to UTF-16 to call FindFirstFileW
- Results of FindFirstFileW must be converted back to UTF-8
Reviewers: zturner
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51941
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@341988 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Add an overload to sys::fs::setLastModificationAndAccessTime that allows setting last access and modification times separately. This will allow tools to use this API when they want to preserve both the access and modification times from an input file, which may be different.
Also note that both the POSIX (futimens/futimes) and Windows (SetFileTime) APIs take the two timestamps in the order of (1) access (2) modification time, so this renames the method to "setLastAccessAndModificationTime" to make it clear which timestamp is which.
For existing callers, the 1-arg overload just sets both timestamps to the same thing.
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D50521
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@339628 91177308-0d34-0410-b5e6-96231b3b80d8
In r338216 / D49860 TempFile::keep was extended to allow keeping across
filesystems. The aim on Windows was to have this happen in rename_internal
using the existing system API. However, to fix an issue and preserve the
idea of "renaming" not being a move, put Windows keep-across-filesystem in
TempFile::keep.
Differential Revision: https://reviews.llvm.org/D50048
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@338841 91177308-0d34-0410-b5e6-96231b3b80d8
Dsymutil's update functionality was broken on Windows because we tried
to rename a file while we're holding open handles to that file. TempFile
provides a solution for this through its keep(Twine) method. This patch
changes dsymutil to make use of that functionality.
Differential revision: https://reviews.llvm.org/D49860
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@338216 91177308-0d34-0410-b5e6-96231b3b80d8
On Windows we've observed that if you open a file, write to it, map it into
memory and close the file handle, the contents of the memory mapping can
sometimes be incorrect. That was what we did when adding an entry to the
ThinLTO cache using the TempFile and MemoryBuffer classes, and it was causing
intermittent build failures on Chromium's ThinLTO bots on Windows. More
details are in the associated Chromium bug (crbug.com/786127).
We can prevent this from happening by keeping a handle to the file open while
the mapping is active. So this patch changes the mapped_file_region class to
duplicate the file handle when mapping the file and close it upon unmapping it.
One gotcha is that the file handle that we keep open must not have been
created with FILE_FLAG_DELETE_ON_CLOSE, as otherwise the operating system
will prevent other processes from opening the file. We can achieve this
by avoiding the use of FILE_FLAG_DELETE_ON_CLOSE altogether. Instead,
we use SetFileInformationByHandle with FileDispositionInfo to manage the
delete-on-close bit. This lets us remove the hack that we used to use to
clear the delete-on-close bit on a file opened with FILE_FLAG_DELETE_ON_CLOSE.
A downside of using SetFileInformationByHandle/FileDispositionInfo as
opposed to FILE_FLAG_DELETE_ON_CLOSE is that it prevents us from using
CreateFile to open the file while the flag is set, even within the same
process. This doesn't seem to matter for almost every client of TempFile,
except for LockFileManager, which calls sys::fs::create_link to create a
hard link from the lock file, and in the process of doing so tries to open
the file. To prevent this change from breaking LockFileManager I changed it
to stop using TempFile by effectively reverting r318550.
Differential Revision: https://reviews.llvm.org/D48051
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334630 91177308-0d34-0410-b5e6-96231b3b80d8
Even if we support no-canonical-prefix on
clang-cl(https://reviews.llvm.org/D47480), argv0 becomes absolute path
in clang-cl and that embeds absolute path in /showIncludes.
This patch removes such full path normalization from InitLLVM on
windows, and that removes absolute path from clang-cl output
(obj/stdout/stderr) when debug flag is disabled.
Patch by Takuto Ikuta!
Differential Revision https://reviews.llvm.org/D47578
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334602 91177308-0d34-0410-b5e6-96231b3b80d8
This simplifies some code which had StringRefs to begin with, and
makes other code more complicated which had const char* to begin
with.
In the end, I think this makes for a more idiomatic and platform
agnostic API. Not all platforms launch process with null terminated
c-string arrays for the environment pointer and argv, but the api
was designed that way because it allowed easy pass-through for
posix-based platforms. There's a little additional overhead now
since on posix based platforms we'll be takign StringRefs which
were constructed from null terminated strings and then copying
them to null terminate them again, but from a readability and
usability standpoint of the API user, I think this API signature
is strictly better.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334518 91177308-0d34-0410-b5e6-96231b3b80d8
I took some liberties and quoted fewer characters than before,
based on an article from MSDN which says that only certain characters
cause an arg to require quoting. This seems to be incorrect, though,
and worse it seems to be a difference in Windows version. The bot
that fails is Windows 7, and I can't reproduce the failure on Win
10. But it's definitely related to quoting and special characters,
because both tests that fail have a * in the argument, which is one
of the special characters that would cause an argument to be quoted
before but not any longer after the new patch.
Since I don't have Win 7, all I can do is just guess that I need to
restore the old quoting rules. So this patch does that in hopes that
it fixes the problem on Windows 7.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334375 91177308-0d34-0410-b5e6-96231b3b80d8
This reverts commit 65243b6d19143cb7a03f68df0169dcb63e8b4632.
Seems like it's not a flake. It might have something to do with
the '*' character being in a command line.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334356 91177308-0d34-0410-b5e6-96231b3b80d8
There were a few linux compilation failures, but other than that
I think this was just a flake that caused the tests to fail. I'm
going to resubmit and see if the failures go away, if not I'll
revert again.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334355 91177308-0d34-0410-b5e6-96231b3b80d8
This reverts commit 10d2e88e87150a35dc367ba30716189d2af26774.
This is causing some test failures for some reason, reverting
while I investigate.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334354 91177308-0d34-0410-b5e6-96231b3b80d8
This function was internal to Program.inc, but I've needed this
on several occasions when I've had to use CreateProcess without
llvm's sys::Execute functions. In doing so, I noticed that the
function was written using unsafe C-string access and was pretty
hard to understand / make sense of, so I've also re-written the
functions to use more modern LLVM constructs.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334353 91177308-0d34-0410-b5e6-96231b3b80d8
NFC here, this just raises some platform specific ifdef hackery
out of a class and creates proper platform-independent typedefs
for the relevant things. This allows these typedefs to be
reused in other places without having to reinvent this preprocessor
logic.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334294 91177308-0d34-0410-b5e6-96231b3b80d8
O_CLOEXEC is the right default, but occasionally you don't
want this. This is especially true for tools like debuggers
where you might need to spawn the child process with specific
files already open, but it's occasionally useful in other
scenarios as well, like when you want to do some IPC between
parent and child.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334293 91177308-0d34-0410-b5e6-96231b3b80d8
This one allows much more flexibility than the standard
openFileForRead / openFileForWrite functions. Since there is now
just one "real" function that does the work, all other implementations
simply delegate to this one.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334246 91177308-0d34-0410-b5e6-96231b3b80d8
This breaks the OpenFlags enumeration into two separate
enumerations: OpenFlags and CreationDisposition. The first
controls the behavior of the API depending on whether or not
the target file already exists, and is not a flags-based
enum. The second controls more flags-like values.
This yields a more easy to understand API, while also allowing
flags to be passed to the openForRead api, where most of the
values didn't make sense before. This also makes the apis more
testable as it becomes easy to enumerate all the configurations
which make sense, so I've added many new tests to exercise all
the different values.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@334221 91177308-0d34-0410-b5e6-96231b3b80d8
Windows' CRT has a limit of 512 open file descriptors, and fds which are
generated by converting a HANDLE via _get_osfhandle count towards this
limit as well.
Regardless, often you find yourself marshalling back and forth between
native HANDLE objects and fds anyway. If we know from the getgo that
we're going to need to work directly with the handle, we can cut out the
marshalling layer while also not contributing to filling up the CRT's
very limited handle table.
On Unix these functions just delegate directly to the existing set of
functions since an fd *is* the native file type. It would be nice, very
long term, if we could convert most uses of fds to file_t.
Differential Revision: https://reviews.llvm.org/D47688
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@333945 91177308-0d34-0410-b5e6-96231b3b80d8
The idea behind WindowsSupport.h is that it's in the source directory so
that windows.h'isms don't leak out into the larger LLVM project. To that
end, any symbol that references a symbol from windows.h must be in this
private header, and not in a public header.
However, we had some useful utility functions in WindowsSupport.h which
have no dependency on the Windows API, but still only make sense on
Windows. Those functions should be usable outside of Support since there
is no risk of causing a windows.h leak. Although this introduces some
preprocessor logic in some header files, It's not too egregious and it's
better than the alternative of duplicating a ton of code.
Differential Revision: https://reviews.llvm.org/D47662
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@333798 91177308-0d34-0410-b5e6-96231b3b80d8
