The way BufMut uses MaybeUninit can lead to unsoundness. This replaces
MaybeUnit with a type owned by bytes so we can ensure the usage patterns
are sound.
Refs: #328
Use case:
```
let bytes: Bytes = ...
let subbytes = bytes.slice(a..b); // where a == b
let slice = &subbytes[..];
let slice_bytes = bytes.slice_ref(slice);
```
Last line should not panic, because `slice` object is derived from
the original `Bytes` object.
Before this commit it panics, because `Bytes::slice` returns a fresh
`Bytes` object when `begin == end`.
This separates the `SharedVtable` into 3:
- `PromotableEvenVtable`: The original `SharedVtable`, which will
promote the `Vec` to `Shared` on the first clone, and is selected when
the `Vec`'s pointer has the LSB unset.
- `PromotableOddVtable`: Similar to the `PromotableEvenVtable`, but
selected when the `Vec`'s pointer has the LSB set. This vtable differs
in the masking used when reconstructing the `Vec`.
- `SharedVtable`: This no longer checks if its current kind is `VEC` or
`ARC`, and is only created by the "promotable" vtables.
This also adds a test using an "odd" global allocator that purposefully
bumps all pointers with alignment of 1.
Closes#343
This brings `BytesMut` in line with `Vec<u8>` behavior.
This also fixes an existing bug in BytesMut::bytes_mut that exposes
invalid slices. The bug was recently introduced and was only on master
and never released to `crates.io`.
In order to fix a test, `BufMutExt::chain_mut` is provided. Withou this,
it is not possible to chain two `&mut [u8]`.
Closes#170
- The return type of `BufMut::bytes_mut` is now
`&mut [MaybeUninit<u8>]`.
- The argument type of `BufMut::bytes_vectored_mut` is now
`&mut [bytes::buf::IoSliceMut]`.
- `bytes::buf::IoSliceMut` is a `repr(transparent)` wrapper around an
`std::io::IoSliceMut`, but does not expose the inner bytes with a safe
API, since they might be uninitialized.
- `BufMut::bytesMut` and `BufMut::bytes_vectored_mut` are no longer
`unsafe fn`, since the types encapsulate the unsafety instead.
Bytes is a useful tool for managing multiple slices into the same region
of memory, and the other things it used to have been removed to reduce
complexity. The exact strategy for managing the multiple references is
no longer hard-coded, but instead backing by a customizable vtable.
- Removed ability to mutate the underlying memory from the `Bytes` type.
- Removed the "inline" (SBO) mechanism in `Bytes`. The reduces a large
amount of complexity, and improves performance when accessing the
slice of bytes, since a branch is no longer needed to check if the
data is inline.
- Removed `Bytes` knowledge of `BytesMut` (`BytesMut` may grow that
knowledge back at a future point.)
As consequence Buf::collect is removed as well, which is replaced with `Buf::into_bytes`. The advantage of `Buf::into_bytes` is that it can be optimized in cases where converting a `T: Buf` into a `Bytes` instance is efficient.
There's no reason the user should be forced to wrap it in BufReader in
case the trait is needed, because the Reader has all the bits for
supporting it naturally.
With this if foo is a mutable slice, it is possible to do
foo.into_buf().put_u32_le(42);
Before this patch into_buf would create a Cursor<&'a [u8]> and it
would not be possible to write into it.
With this if foo is a mutable slice, it is possible to do
foo.into_buf().put_u32_le(42);
Before this patch into_buf would create a Cursor<&'a [u8]> and it
would not be possible to write into it.
* Recycle space when reserving from Vec-backed Bytes
BytesMut::reserve, when called on a BytesMut instance which is backed by
a non-shared Vec<u8>, would previously just delegate to Vec::reserve
regardless of the current location in the buffer. If the Bytes is
actually the trailing component of a larger Vec, then the unused space
won't be recycled. In applications which continually move the pointer
forward to consume data as it comes in, this can cause the underlying
buffer to get extremely large.
This commit checks whether there's extra space at the start of the
backing Vec in this case, and reuses the unused space if possible
instead of allocating.
* Avoid excessive copying when reusing Vec space
Only reuse space in a Vec-backed Bytes when doing so would gain back
more than half of the current capacity. This avoids excessive copy
operations when a large buffer is almost (but not completely) full.
* Recycle space when reserving from Vec-backed Bytes
BytesMut::reserve, when called on a BytesMut instance which is backed by
a non-shared Vec<u8>, would previously just delegate to Vec::reserve
regardless of the current location in the buffer. If the Bytes is
actually the trailing component of a larger Vec, then the unused space
won't be recycled. In applications which continually move the pointer
forward to consume data as it comes in, this can cause the underlying
buffer to get extremely large.
This commit checks whether there's extra space at the start of the
backing Vec in this case, and reuses the unused space if possible
instead of allocating.
* Avoid excessive copying when reusing Vec space
Only reuse space in a Vec-backed Bytes when doing so would gain back
more than half of the current capacity. This avoids excessive copy
operations when a large buffer is almost (but not completely) full.
* make Buf and BufMut usable as trait objects
- All the `get_*` and `put_*` methods that take `T: ByteOrder` have
a `where Self: Sized` bound added, so that they are only usable from
sized types. It was impossible to make `Buf` or `BufMut` into trait
objects before, so this change doesn't break anyone.
- Add `get_n_be`/`get_n_le`/`put_n_be`/`put_n_le` methods that can be
used on trait objects.
- Deprecate the export of `ByteOrder` and methods generic on it.
* remove deprecated ByteOrder methods
Removes the `_be` suffix from all methods, implying that the default
people should use is network endian.
- All the `get_*` and `put_*` methods that take `T: ByteOrder` have
a `where Self: Sized` bound added, so that they are only usable from
sized types. It was impossible to make `Buf` or `BufMut` into trait
objects before, so this change doesn't break anyone.
- Add `get_n_be`/`get_n_le`/`put_n_be`/`put_n_le` methods that can be
used on trait objects.
- Deprecate the export of `ByteOrder` and methods generic on it.
Fixes#163
* Compact Bytes original capacity representation
In order to avoid unnecessary allocations, a `Bytes` structure remembers
the capacity with which it was first created. When a reserve operation
is issued, this original capacity value is used to as a baseline for
reallocating new storage.
Previously, this original capacity value was stored in its raw form. In
other words, the original capacity `usize` was stored as is. In order to
reclaim some `Bytes` internal storage space for additional features,
this original capacity value is compressed from requiring 16 bits to 3.
To do this, instead of storing the exact original capacity. The original
capacity is rounded down to the nearest power of two. If the original
capacity is less than 1024, then it is rounded down to zero. This
roughly means that the original capacity is now stored as a table:
0 => 0
1 => 1k
2 => 2k
3 => 4k
4 => 8k
5 => 16k
6 => 32k
7 => 64k
For the purposes that the original capacity feature was introduced, this
is sufficient granularity.
* Provide `advance` on Bytes and BytesMut
This is the `advance` function that would be part of a `Buf`
implementation. However, `Bytes` and `BytesMut` cannot impl `Buf` until
the next breaking release.
The implementation uses the additional storage made available by the
previous commit to store the number of bytes that the view was advanced.
The `ptr` pointer will point to the start of the window, avoiding any
pointer arithmetic when dereferencing the `Bytes` handle.
Carl Lerche