This revision is a part of a series of patches extending AddressSanitizer C++ container overflow detection capabilities by adding annotations, similar to those existing in std::vector, to std::string and `std::deque` collections. These changes allow ASan to detect cases when the instrumented program accesses memory which is internally allocated by the collection but is still not in-use (accesses before or after the stored elements for `std::deque`, or between the size and capacity bounds for `std::string`).
The motivation for the research and those changes was a bug, found by Trail of Bits, in a real code where an out-of-bounds read could happen as two strings were compared via a std::equals function that took `iter1_begin`, `iter1_end`, `iter2_begin` iterators (with a custom comparison function). When object `iter1` was longer than `iter2`, read out-of-bounds on `iter2` could happen. Container sanitization would detect it.
This revision introduces annotations for `std::deque`. Each chunk of the container can now be annotated using the `__sanitizer_annotate_double_ended_contiguous_container` function, which was added in the rG1c5ad6d2c01294a0decde43a88e9c27d7437d157. Any attempt to access poisoned memory will trigger an ASan error. Although false negatives are rare, they are possible due to limitations in the ASan API, where a few (usually up to 7) bytes before the container may remain unpoisoned. There are no false positives in the same way as with `std::vector` annotations.
This patch only supports objects (deques) that use the standard allocator. However, it can be easily extended to support all allocators, as suggested in the D146815 revision.
Furthermore, the patch includes the addition of the `is_double_ended_contiguous_container_asan_correct` function to libcxx/test/support/asan_testing.h. This function can be used to verify whether a `std::deque` object has been correctly annotated.
Finally, the patch extends the unit tests to verify ASan annotations (added LIBCPP_ASSERTs).
If a program is compiled without ASan, all helper functions will be no-ops. In binaries with ASan, there is a negligible performance impact since the code from the change is only executed when the deque container changes in size and it’s proportional to the change. It is important to note that regardless of whether or not these changes are in use, every access to the container's memory is instrumented.
Reviewed By: #libc, philnik
Spies: vitalybuka, hans, mikhail.ramalho, Enna1, #sanitizers, philnik, libcxx-commits
Differential Revision: https://reviews.llvm.org/D132092
We plan to add concepts for checking that iterators actually provide what they claim to. This is to avoid people thinking that these type traits actually check the iterator requirements in more detail.
Reviewed By: ldionne, #libc
Spies: Mordante, libcxx-commits, wenlei
Differential Revision: https://reviews.llvm.org/D150801
- add the `from_range_t` constructors and the related deduction guides;
- add the `insert_range`/`assign_range`/etc. member functions.
(Note: this patch is split from https://reviews.llvm.org/D142335)
Differential Revision: https://reviews.llvm.org/D149827
This revision is a part of a series of patches extending AddressSanitizer C++ container overflow detection capabilities by adding annotations, similar to those existing in std::vector, to std::string and `std::deque` collections. These changes allow ASan to detect cases when the instrumented program accesses memory which is internally allocated by the collection but is still not in-use (accesses before or after the stored elements for `std::deque`, or between the size and capacity bounds for `std::string`).
The motivation for the research and those changes was a bug, found by Trail of Bits, in a real code where an out-of-bounds read could happen as two strings were compared via a std::equals function that took `iter1_begin`, `iter1_end`, `iter2_begin` iterators (with a custom comparison function). When object `iter1` was longer than `iter2`, read out-of-bounds on `iter2` could happen. Container sanitization would detect it.
This revision introduces annotations for `std::deque`. Each chunk of the container can now be annotated using the `__sanitizer_annotate_double_ended_contiguous_container` function, which was added in the rG1c5ad6d2c01294a0decde43a88e9c27d7437d157. Any attempt to access poisoned memory will trigger an ASan error. Although false negatives are rare, they are possible due to limitations in the ASan API, where a few (usually up to 7) bytes before the container may remain unpoisoned. There are no false positives in the same way as with `std::vector` annotations.
This patch only supports objects (deques) that use the standard allocator. However, it can be easily extended to support all allocators, as suggested in the D146815 revision.
Furthermore, the patch includes the addition of the `is_double_ended_contiguous_container_asan_correct` function to libcxx/test/support/asan_testing.h. This function can be used to verify whether a `std::deque` object has been correctly annotated.
Finally, the patch extends the unit tests to verify ASan annotations (added LIBCPP_ASSERTs).
If a program is compiled without ASan, all helper functions will be no-ops. In binaries with ASan, there is a negligible performance impact since the code from the change is only executed when the deque container changes in size and it’s proportional to the change. It is important to note that regardless of whether or not these changes are in use, every access to the container's memory is instrumented.
Reviewed By: #libc, philnik
Spies: mikhail.ramalho, Enna1, #sanitizers, philnik, libcxx-commits
Differential Revision: https://reviews.llvm.org/D132092
We changed the `abort` calls when trying to throw exceptions in `-fno-exceptions` mode to `__verbose_abort` calls, which removes the dependency in most files.
Reviewed By: ldionne, #libc
Spies: dim, emaste, mikhail.ramalho, smeenai, libcxx-commits
Differential Revision: https://reviews.llvm.org/D146076
These changes make it possible to use __synth_three_way in modules. The
change from a lambda to a function is a Clang issue.
The change is list was needed since the compiler couldn't deduce the
comparison template argument.
Adds a few missing includes too.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D146545
This patch also updates the moved code to the new style (i.e. formatted, replaced marcos and typedefs)
Reviewed By: ldionne, #libc
Spies: arichardson, libcxx-commits
Differential Revision: https://reviews.llvm.org/D145095
Other macros that disable parts of the library are named `_LIBCPP_HAS_NO_WHATEVER`.
Reviewed By: ldionne, Mordante, #libc
Spies: libcxx-commits, smeenai
Differential Revision: https://reviews.llvm.org/D143163
This change is almost fully mechanical. The only interesting change is in `generate_feature_test_macro_components.py` to generate `_LIBCPP_STD_VER >=` instead. To avoid churn in the git-blame this commit should be added to the `.git-blame-ignore-revs` once committed.
Reviewed By: ldionne, var-const, #libc
Spies: jloser, libcxx-commits, arichardson, arphaman, wenlei
Differential Revision: https://reviews.llvm.org/D143962
This has multiple benefits:
- The optimizations are also performed for the `ranges::` versions of the algorithms
- Code duplication is reduced
- it is simpler to add this optimization for other segmented iterators,
like `ranges::join_view::iterator`
- Algorithm code is removed from `<deque>`
Reviewed By: ldionne, huixie90, #libc
Spies: mstorsjo, sstefan1, EricWF, libcxx-commits, mgorny
Differential Revision: https://reviews.llvm.org/D132505
This patch is the rebase and squash of three earlier patches.
It supersedes all three of them.
- D47111: experimental monotonic_buffer_resource.
- D47358: experimental pool resources.
- D47360: Copy std::experimental::pmr to std::pmr.
The significant difference between this patch and the-sum-of-those-three
is that this patch does not add `std::experimental::pmr::monotonic_buffer_resource`
and so on. This patch simply adds the C++17 standard facilities, and
leaves the `std::experimental` namespace entirely alone.
Differential Revision: https://reviews.llvm.org/D89057
This defines a new policy for removal of transitive includes.
The goal of the policy it to make it relatively easy to remove
headers when needed, but avoid breaking developers using and
vendors shipping libc++.
The method used is to guard transitive includes based on the
C++ language version. For the upcoming C++23 we can remove
headers when we want, but for other language versions we try
to keep it to a minimum.
In this code the transitive include of `<chrono>` is removed
since D128577 introduces a header cycle between `<format>`
and `<chrono>`. This cycle is indirectly required by the
Standard. Our cycle dependency tool basically is a grep based
tool, so it needs some hints to ignore cycles. With the input
of our transitive include tests we can create a better tool.
However that's out of the scope of this patch.
Note the flag `_LIBCPP_REMOVE_TRANSITIVE_INCLUDES` remains
unchanged. So users can still opt-out of transitives includes
entirely.
Reviewed By: #libc, ldionne, philnik
Differential Revision: https://reviews.llvm.org/D132284
This patch simplifies the implementation of `deque` by removing the `__deque_base` class which results in a lot less indirections and removes the need for `__base::`.
Reviewed By: ldionne, #libc
Spies: AdvenamTacet, libcxx-commits
Differential Revision: https://reviews.llvm.org/D132081
This commit re-adds transitive includes that had been removed by
4cd04d1687, c36870c8e7, a83f4b9cda, 1458458b55, 2e2f3158c6,
and 489637e66d. This should cover almost all the includes that had
been removed since LLVM 14 and that would contribute to breaking user
code when releasing LLVM 15.
It is possible to disable the inclusion of these headers by defining
_LIBCPP_REMOVE_TRANSITIVE_INCLUDES. The intent is that vendors will
enable that macro and start fixing downstream issues immediately. We
can then remove the macro (and the transitive includes) by default in
a future release. That way, we will break users only once by removing
transitive includes in bulk instead of doing it bit by bit a every
release, which is more disruptive for users.
Note 1: The set of headers to re-add was found by re-generating the
transitive include test on a checkout of release/14.x, which
provided the list of all transitive includes we used to provide.
Note 2: Several includes of <vector>, <optional>, <array> and <unordered_map>
have been added in this commit. These transitive inclusions were
added when we implemented boyer_moore_searcher in <functional>.
Note 3: This is a best effort patch to try and resolve downstream breakage
caused since branching LLVM 14. I wasn't able to perfectly mirror
transitive includes in LLVM 14 for a few headers, so I added a
release note explaining it. To summarize, adding boyer_moore_searcher
created a bunch of circular dependencies, so we have to break
backwards compatibility in a few cases.
Differential Revision: https://reviews.llvm.org/D128661
Instead of writing every character directly into the container by using
a `back_insert_iterator` the data is buffered in an `array`. This buffer
is then inserted to the container by calling its `insert` member function.
Since there's no guarantee every container's `insert` behaves properly
containers need to opt-in to this behaviour. The appropriate standard
containers opt-in to this behaviour.
This change improves the performance of the format functions that use a
`back_insert_iterator`.
Depends on D110495
Reviewed By: ldionne, vitaut, #libc
Differential Revision: https://reviews.llvm.org/D110497
This patch changes the requirement for getting the declaration of the
assertion handler from including <__assert> to including any public
C++ header of the library. Note that C compatibility headers are
excluded because we don't implement all the C headers ourselves --
some of them are taken straight from the C library, like assert.h.
It also adds a generated test to check it. Furthermore, this new
generated test is designed in a way that will make it possible to
replace almost all the existing test-generation scripts with this
system in upcoming patches.
Differential Revision: https://reviews.llvm.org/D122506
In C++20 the type trait `type_identity` was introduced. For the same purpose there is `__identity` for pre-C++20 code. The name is confusing, because since C++20 there is also `identity`, which isn't a type trait.
Reviewed By: ldionne, Mordante, #libc
Spies: EricWF, libcxx-commits
Differential Revision: https://reviews.llvm.org/D122017
Notably the following ctors remain non-explicit because they
are used as implicit conversions in too many places:
* __debug_less(_Compare&)
* __map_iterator(_TreeIterator)
* __map_const_iterator(_TreeIterator)
* __hash_map_iterator(_HashIterator)
* __hash_map_const_iterator(_HashIterator)
Differential Revision: https://reviews.llvm.org/D119894
This is the first step towards disentangling the debug mode and assertions
in libc++. This patch doesn't make any functional change: it simply moves
_LIBCPP_ASSERT-related stuff to its own file so as to make it clear that
libc++ assertions and the debug mode are different things. Future patches
will make it possible to enable assertions without enabling the debug
mode.
Differential Revision: https://reviews.llvm.org/D119769
Disable the constructors taking `(size_type, const value_type&,
allocator_type)` if `allocator_type` is not a valid allocator.
Otherwise, these constructors are considered when resolving e.g.
`(int*, int*, NotAnAllocator())`, leading to a hard error during
instantiation. A hard error makes the Standard's requirement to not
consider deduction guides of the form `(Iterator, Iterator,
BadAllocator)` during overload resolution essentially non-functional.
The previous approach was to SFINAE away `allocator_traits`. This patch
SFINAEs away the specific constructors instead, for consistency with
`basic_string` -- see [LWG3076](wg21.link/lwg3076) which describes
a very similar problem for strings (note, however, that unlike LWG3076,
no valid constructor call is affected by the bad instantiation).
Differential Revision: https://reviews.llvm.org/D114311
We've stopped doing it in libc++ for a while now because these names
would end up rotting as we move things around and copy/paste stuff.
This cleans up all the existing files so as to stop the spreading
as people copy-paste headers around.
Deduction guides for containers should not participate in overload
resolution when called with certain incorrect types (e.g. when called
with a template argument in place of an `InputIterator` that doesn't
qualify as an input iterator). Similarly, class template argument
deduction should not select `unique_ptr` constructors that take a
a pointer.
The tests try out every possible incorrect parameter (but never more
than one incorrect parameter in the same invocation).
Also add deduction guides to the synopsis for associative and unordered
containers (this was accidentally omitted from [D112510](https://reviews.llvm.org/D112510)).
Differential Revision: https://reviews.llvm.org/D112904
Replace `&__rhs` with `_VSTD::addressof(__rhs)` to guard against ADL hijacking
of `operator&` in `operator=`. Thanks to @CaseyCarter for bringing it to our
attention.
Similar issues with hijacking `operator&` still exist, they will be
addressed separately.
Reviewed By: #libc, Quuxplusone, ldionne
Differential Revision: https://reviews.llvm.org/D110852
Detected by evil-izing the widely used `MoveOnly` testing type.
I had to patch some tests that were themselves using its comma operator,
but I think that's a worthwhile cost in order to catch more places
in our headers that needed comma-proofing.
The trick here is that even `++ptr, SomeClass()` can find a comma operator
by ADL, if `ptr` is of type `Evil*`. (A comma between two operands
of non-class-or-enum type is always treated as the built-in
comma, without ADL. But if either operand is class-or-enum, then
ADL happens for _both_ operands' types.)
Differential Revision: https://reviews.llvm.org/D109414
I just ran into a compiler error involving __bind_back and some overloads
that were being disabled with _EnableIf. I noticed that the error message
was quite bad and did not mention the reason for the overload being
excluded. Specifically, the error looked like this:
candidate template ignored: substitution failure [with _Args =
<ContiguousView>]: no member named '_EnableIfImpl' in 'std::_MetaBase<false>'
Instead, when using enable_if or enable_if_t, the compiler is clever and
can produce better diagnostics, like so:
candidate template ignored: requirement 'is_invocable_v<
std::__bind_back_op<1, std::integer_sequence<unsigned long, 0>>,
std::ranges::views::__transform::__fn &, std::tuple<PlusOne> &,
ContiguousView>' was not satisfied [with _Args = <ContiguousView>]
Basically, it tries to do a poor man's implementation of concepts, which
is already a lot better than simply complaining about substitution failure.
Hence, this commit uses enable_if_t instead of _EnableIf whenever
possible. That is both more straightforward than using the internal
helper, and also leads to better error messages in those cases.
I understand the motivation for _EnableIf's implementation was to improve
compile-time performance, however I believe striving to improve error
messages is even more important for our QOI, hence this patch. Furthermore,
it is unclear that _EnableIf actually improved compile-time performance
in any noticeable way (see discussion in the review for details).
Differential Revision: https://reviews.llvm.org/D108216
__split_buffer_common was entirely unused, and __deque_base_common
was unused except for two calls to __throw_out_of_range(), which have
been inlined.
The usual intent of the __xxx_base_common base classes is to localize
where the exception-throwing code is instantiated, however that wasn't
the case here because we never explicitly instantiated those base classes
in the shared library, unlike what we do for basic_string and vector.
Differential Revision: https://reviews.llvm.org/D108384
All supported compilers have supported deduction guides in C++17 for a
while, so this isn't necessary anymore.
Differential Revision: https://reviews.llvm.org/D108213
Moves:
* `std::move`, `std::forward`, `std::declval`, and `std::swap` into
`__utility/${FUNCTION_NAME}`.
* `std::swap_ranges` and `std::iter_swap` into
`__algorithm/${FUNCTION_NAME}`
Differential Revision: https://reviews.llvm.org/D103734
P1518 does the following in C++23 but we'll just do it in C++17 as well:
- Stop requiring `Alloc` to be an allocator on some container-adaptor deduction guides
- Stop deducing from `Allocator` on some sequence container constructors
- Stop deducing from `Allocator` on some other container constructors (libc++ already did this)
The affected constructors are the "allocator-extended" versions of
constructors where the non-allocator arguments are already sufficient
to deduce the allocator type. For example,
std::pmr::vector<int> v1;
std::vector v2(v1, std::pmr::new_delete_resource());
std::stack s2(v1, std::pmr::new_delete_resource());
Differential Revision: https://reviews.llvm.org/D97742
