openharmony/third_party_littlefs
1
0
Fork 0
mirror of https://gitee.com/openharmony/third_party_littlefs synced 2024-11-23 14:59:50 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
0aba71d0d6
third_party_littlefs/lfs.c

4927 lines
149 KiB
C
Raw Normal View History

Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
/*
* The little filesystem
*
Fixed license header in lfs.c Found by pabigot
2019-07-19 00:43:49 +00:00
* Copyright (c) 2017, Arm Limited. All rights reserved.
* SPDX-License-Identifier: BSD-3-Clause
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
*/
#include "lfs.h"
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
#include "lfs_util.h"
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
#define LFS_BLOCK_NULL ((lfs_block_t)-1)
#define LFS_BLOCK_INLINE ((lfs_block_t)-2)
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// Caching block device operations ///
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
static inline void lfs_cache_drop(lfs_t *lfs, lfs_cache_t *rcache) {
// do not zero, cheaper if cache is readonly or only going to be
// written with identical data (during relocates)
(void)lfs;
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
rcache->block = LFS_BLOCK_NULL;
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
}
static inline void lfs_cache_zero(lfs_t *lfs, lfs_cache_t *pcache) {
// zero to avoid information leak
Fixed buffer overflow due to mistaking prog_size for cache_size found by ajaybhargav
2019-04-12 13:41:42 +00:00
memset(pcache->buffer, 0xff, lfs->cfg->cache_size);
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
pcache->block = LFS_BLOCK_NULL;
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
}
static int lfs_bd_read(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache, lfs_size_t hint,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_block_t block, lfs_off_t off,
void *buffer, lfs_size_t size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
uint8_t *data = buffer;
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
if (block >= lfs->cfg->block_count ||
off+size > lfs->cfg->block_size) {
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
return LFS_ERR_CORRUPT;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
lfs_size_t diff = size;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (pcache && block == pcache->block &&
off < pcache->off + pcache->size) {
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
if (off >= pcache->off) {
// is already in pcache?
diff = lfs_min(diff, pcache->size - (off-pcache->off));
memcpy(data, &pcache->buffer[off-pcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
// pcache takes priority
diff = lfs_min(diff, pcache->off-off);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (block == rcache->block &&
off < rcache->off + rcache->size) {
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
if (off >= rcache->off) {
// is already in rcache?
diff = lfs_min(diff, rcache->size - (off-rcache->off));
memcpy(data, &rcache->buffer[off-rcache->off], diff);
data += diff;
off += diff;
size -= diff;
continue;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
// rcache takes priority
diff = lfs_min(diff, rcache->off-off);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Revert "Don't bypass cache in `lfs_cache_prog()` and `lfs_cache_read()`" This reverts commit fdd239fe21a3721dc7fb55a9cf816544af7e9647. Bypassing cache turned out to be a mistake which causes more problems than it solves. Device driver should deal with alignment if this is required - trying to do that in a file system is not a viable solution anyway.
2019-08-09 21:02:33 +00:00
if (size >= hint && off % lfs->cfg->read_size == 0 &&
size >= lfs->cfg->read_size) {
// bypass cache?
diff = lfs_aligndown(diff, lfs->cfg->read_size);
int err = lfs->cfg->read(lfs->cfg, block, off, data, diff);
if (err) {
return err;
}
data += diff;
off += diff;
size -= diff;
continue;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
// load to cache, first condition can no longer fail
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
rcache->block = block;
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
rcache->off = lfs_aligndown(off, lfs->cfg->read_size);
Fixed read cache amount based on hint and offset Found by apmorton
2019-05-21 22:21:52 +00:00
rcache->size = lfs_min(
lfs_min(
lfs_alignup(off+hint, lfs->cfg->read_size),
lfs->cfg->block_size)
- rcache->off,
lfs->cfg->cache_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
int err = lfs->cfg->read(lfs->cfg, rcache->block,
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
rcache->off, rcache->buffer, rcache->size);
Added asserts on positive return values from block device functions This has been a large source of porting errors, partially due to my fault in not having enough porting documentation, which is also planned. In the short term, asserts should at least help catch these types of errors instead of just letting the filesystem collapse after recieving an odd error code.
2019-07-16 20:55:29 +00:00
LFS_ASSERT(err <= 0);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
enum {
LFS_CMP_EQ = 0,
LFS_CMP_LT = 1,
LFS_CMP_GT = 2,
};
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
static int lfs_bd_cmp(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache, lfs_size_t hint,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_block_t block, lfs_off_t off,
const void *buffer, lfs_size_t size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
const uint8_t *data = buffer;
for (lfs_off_t i = 0; i < size; i++) {
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
uint8_t dat;
int err = lfs_bd_read(lfs,
pcache, rcache, hint-i,
block, off+i, &dat, 1);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (err) {
return err;
}
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
if (dat != data[i]) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return (dat < data[i]) ? LFS_CMP_LT : LFS_CMP_GT;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return LFS_CMP_EQ;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
static int lfs_bd_flush(lfs_t *lfs,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_cache_t *pcache, lfs_cache_t *rcache, bool validate) {
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
if (pcache->block != LFS_BLOCK_NULL && pcache->block != LFS_BLOCK_INLINE) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(pcache->block < lfs->cfg->block_count);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_size_t diff = lfs_alignup(pcache->size, lfs->cfg->prog_size);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs->cfg->prog(lfs->cfg, pcache->block,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
pcache->off, pcache->buffer, diff);
Added asserts on positive return values from block device functions This has been a large source of porting errors, partially due to my fault in not having enough porting documentation, which is also planned. In the short term, asserts should at least help catch these types of errors instead of just letting the filesystem collapse after recieving an odd error code.
2019-07-16 20:55:29 +00:00
LFS_ASSERT(err <= 0);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (validate) {
// check data on disk
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, rcache);
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int res = lfs_bd_cmp(lfs,
NULL, rcache, diff,
pcache->block, pcache->off, pcache->buffer, diff);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (res < 0) {
return res;
}
if (res != LFS_CMP_EQ) {
return LFS_ERR_CORRUPT;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
}
Fixed information leaks through reused caches As a shortcut, littlefs never bother to zero any of the buffers is used. It didn't need to because it would always write out the entirety of the data it needed. Unfortunately, this, combined with the extra padding used to align buffers to the nearest prog size, would lead to uninitialized data getting written out to disk. This means unrelated file data could be written to different parts of storage, or worse, information leaked from the malloc calls could be written out to disk unnecessarily. found by rojer
2018-07-06 16:14:30 +00:00
lfs_cache_zero(lfs, pcache);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
return 0;
}
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
static int lfs_bd_sync(lfs_t *lfs,
lfs_cache_t *pcache, lfs_cache_t *rcache, bool validate) {
lfs_cache_drop(lfs, rcache);
int err = lfs_bd_flush(lfs, pcache, rcache, validate);
if (err) {
return err;
}
Added asserts on positive return values from block device functions This has been a large source of porting errors, partially due to my fault in not having enough porting documentation, which is also planned. In the short term, asserts should at least help catch these types of errors instead of just letting the filesystem collapse after recieving an odd error code.
2019-07-16 20:55:29 +00:00
err = lfs->cfg->sync(lfs->cfg);
LFS_ASSERT(err <= 0);
return err;
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
}
static int lfs_bd_prog(lfs_t *lfs,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_cache_t *pcache, lfs_cache_t *rcache, bool validate,
lfs_block_t block, lfs_off_t off,
const void *buffer, lfs_size_t size) {
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
const uint8_t *data = buffer;
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
LFS_ASSERT(block == LFS_BLOCK_INLINE || block < lfs->cfg->block_count);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
LFS_ASSERT(off + size <= lfs->cfg->block_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (block == pcache->block &&
off >= pcache->off &&
off < pcache->off + lfs->cfg->cache_size) {
// already fits in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->cfg->cache_size - (off-pcache->off));
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
memcpy(&pcache->buffer[off-pcache->off], data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Fixed issues with reading and caching inline files Kind of a two-fold issue. One, the programming to the middle of inline files was causing the cache to get updated to a half programmed state. While fine, as all programs do occur in order in a block, this is less efficient when writing to inline files as it would cause the inline file to need to be reread even if it fits in the cache. Two, the rereading of the inline file was broken and passed the file's tag all the way to where a user would expect an error. This was easy to fix but adds to the reasons we should have test coverage information. Found by ebinans
2019-05-31 05:58:48 +00:00
pcache->size = lfs_max(pcache->size, off - pcache->off);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (pcache->size == lfs->cfg->cache_size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// eagerly flush out pcache if we fill up
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int err = lfs_bd_flush(lfs, pcache, rcache, validate);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// pcache must have been flushed, either by programming and
// entire block or manually flushing the pcache
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
LFS_ASSERT(pcache->block == LFS_BLOCK_NULL);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// prepare pcache, first condition can no longer fail
pcache->block = block;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
pcache->off = lfs_aligndown(off, lfs->cfg->prog_size);
pcache->size = 0;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_erase(lfs_t *lfs, lfs_block_t block) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added asserts on positive return values from block device functions This has been a large source of porting errors, partially due to my fault in not having enough porting documentation, which is also planned. In the short term, asserts should at least help catch these types of errors instead of just letting the filesystem collapse after recieving an odd error code.
2019-07-16 20:55:29 +00:00
int err = lfs->cfg->erase(lfs->cfg, block);
LFS_ASSERT(err <= 0);
return err;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
/// Small type-level utilities ///
// operations on block pairs
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline void lfs_pair_swap(lfs_block_t pair[2]) {
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
lfs_block_t t = pair[0];
pair[0] = pair[1];
pair[1] = t;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline bool lfs_pair_isnull(const lfs_block_t pair[2]) {
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
return pair[0] == LFS_BLOCK_NULL || pair[1] == LFS_BLOCK_NULL;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline int lfs_pair_cmp(
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
return !(paira[0] == pairb[0] || paira[1] == pairb[1] ||
paira[0] == pairb[1] || paira[1] == pairb[0]);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline bool lfs_pair_sync(
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
return (paira[0] == pairb[0] && paira[1] == pairb[1]) ||
(paira[0] == pairb[1] && paira[1] == pairb[0]);
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline void lfs_pair_fromle32(lfs_block_t pair[2]) {
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
pair[0] = lfs_fromle32(pair[0]);
pair[1] = lfs_fromle32(pair[1]);
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline void lfs_pair_tole32(lfs_block_t pair[2]) {
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
pair[0] = lfs_tole32(pair[0]);
pair[1] = lfs_tole32(pair[1]);
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// operations on 32-bit entry tags
typedef uint32_t lfs_tag_t;
typedef int32_t lfs_stag_t;
Added root entry and expanding superblocks Expanding superblocks has been on my wishlist for a while. The basic idea is that instead of maintaining a fixed offset blocks {0, 1} to the the root directory (1 pointer), we maintain a dynamically sized linked-list of superblocks that point to the actual root. If the number of writes to the root exceeds some value, we increase the size of the superblock linked-list. This can leverage existing metadata-pair operations. The revision count for metadata-pairs provides some knowledge on how much wear we've put on the superblock, and the threaded linked-list can also be reused for this purpose. This means superblock expansion is both optional and cheap to implement. Expanding superblocks helps both extremely small and extremely large filesystem (extreme being relative of course). On the small end, we can actually collapse the superblock into the root directory and drop the hard requirement of 4-blocks for the superblock. On the large end, our superblock will now last longer than the rest of the filesystem. Each time we expand, the number of cycles until the superblock dies is increased by a power. Before we were stuck with this layout: level cycles limit layout 1 E^2 390 MiB s0 -> root Now we expand every time a fixed offset is exceeded: level cycles limit layout 0 E 4 KiB s0+root 1 E^2 390 MiB s0 -> root 2 E^3 37 TiB s0 -> s1 -> root 3 E^4 3.6 EiB s0 -> s1 -> s2 -> root ... Where the cycles are the number of cycles before death, and the limit is the worst-case size a filesystem where early superblock death becomes a concern (all writes to root using this formula: E^|s| = E*B, E = erase cycles = 100000, B = block count, assuming 4096 byte blocks). Note we can also store copies of the superblock entry on the expanded superblocks. This may help filesystem recover tools in the future.
2018-08-06 18:30:51 +00:00
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
#define LFS_MKTAG(type, id, size) \
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
(((lfs_tag_t)(type) << 20) | ((lfs_tag_t)(id) << 10) | (lfs_tag_t)(size))
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
#define LFS_MKTAG_IF(cond, type, id, size) \
((cond) ? LFS_MKTAG(type, id, size) : LFS_MKTAG(LFS_FROM_NOOP, 0, 0))
#define LFS_MKTAG_IF_ELSE(cond, type1, id1, size1, type2, id2, size2) \
Fixed typo in LFS_MKTAG_IF_ELSE
2020-02-12 17:31:34 +00:00
((cond) ? LFS_MKTAG(type1, id1, size1) : LFS_MKTAG(type2, id2, size2))
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline bool lfs_tag_isvalid(lfs_tag_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return !(tag & 0x80000000);
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static inline bool lfs_tag_isdelete(lfs_tag_t tag) {
return ((int32_t)(tag << 22) >> 22) == -1;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static inline uint16_t lfs_tag_type1(lfs_tag_t tag) {
return (tag & 0x70000000) >> 20;
}
static inline uint16_t lfs_tag_type3(lfs_tag_t tag) {
return (tag & 0x7ff00000) >> 20;
Added support for deleting attributes littlefs has a mechanism for deleting file entries, but it doesn't have a mechanism for deleting individual tags. This _is_ sufficient for a filesystem, but limits our flexibility. Deleting attributes would be useful in the custom attribute API and for future improvements (hint the child pointers in B-trees). However, deleteing attributes is tricky. We can't just omit the attribute, since we can only add new tags. Additionally, we need a way to track what attributes have been deleted during compaction, which currently relies on writing out attributes to disk. The solution here is pretty nifty. First we have to come up with a way to represent a "deleted" attribute. Rather than adding an additional bit to the already squished tag structure, we use a -1 length field, specifically 0xfff. Now we can commit a delete attribute, and this deleted tag acts as a place holder during compacts. However our delete tag will never leave our metadata log. We need some way to discard our delete tag if we know it's the only representation of that tag on the metadata log. Ah! We know it's the only tag if it's in the first commit on the metadata log. So we add an additional bit to the CRC entry to indicate if we're on the first commit, and use that to decide if we need to keep delete tags around. Now we have working tag deletion. Interestingly enough, tag deletion is actually indirectly more efficient than entry deletion, since compacting entries requires multiple passes, whereas tag deletion gets cleaned up lazily. However we can't adopt the same strategy in entry deletion because of the compact ordering of entries. Tag deletion works because tag types are unique and static. Managing entry deletion in this manner would require static id allocation, which would cause problems when creating files, running out of space, and disallow arbitrary insertions of files.
2018-09-09 22:48:11 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static inline uint8_t lfs_tag_chunk(lfs_tag_t tag) {
return (tag & 0x0ff00000) >> 20;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static inline int8_t lfs_tag_splice(lfs_tag_t tag) {
return (int8_t)lfs_tag_chunk(tag);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline uint16_t lfs_tag_id(lfs_tag_t tag) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return (tag & 0x000ffc00) >> 10;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline lfs_size_t lfs_tag_size(lfs_tag_t tag) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return tag & 0x000003ff;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline lfs_size_t lfs_tag_dsize(lfs_tag_t tag) {
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
return sizeof(tag) + lfs_tag_size(tag + lfs_tag_isdelete(tag));
Added support for deleting attributes littlefs has a mechanism for deleting file entries, but it doesn't have a mechanism for deleting individual tags. This _is_ sufficient for a filesystem, but limits our flexibility. Deleting attributes would be useful in the custom attribute API and for future improvements (hint the child pointers in B-trees). However, deleteing attributes is tricky. We can't just omit the attribute, since we can only add new tags. Additionally, we need a way to track what attributes have been deleted during compaction, which currently relies on writing out attributes to disk. The solution here is pretty nifty. First we have to come up with a way to represent a "deleted" attribute. Rather than adding an additional bit to the already squished tag structure, we use a -1 length field, specifically 0xfff. Now we can commit a delete attribute, and this deleted tag acts as a place holder during compacts. However our delete tag will never leave our metadata log. We need some way to discard our delete tag if we know it's the only representation of that tag on the metadata log. Ah! We know it's the only tag if it's in the first commit on the metadata log. So we add an additional bit to the CRC entry to indicate if we're on the first commit, and use that to decide if we need to keep delete tags around. Now we have working tag deletion. Interestingly enough, tag deletion is actually indirectly more efficient than entry deletion, since compacting entries requires multiple passes, whereas tag deletion gets cleaned up lazily. However we can't adopt the same strategy in entry deletion because of the compact ordering of entries. Tag deletion works because tag types are unique and static. Managing entry deletion in this manner would require static id allocation, which would cause problems when creating files, running out of space, and disallow arbitrary insertions of files.
2018-09-09 22:48:11 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// operations on attributes in attribute lists
struct lfs_mattr {
lfs_tag_t tag;
const void *buffer;
};
struct lfs_diskoff {
lfs_block_t block;
lfs_off_t off;
};
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
#define LFS_MKATTRS(...) \
(struct lfs_mattr[]){__VA_ARGS__}, \
sizeof((struct lfs_mattr[]){__VA_ARGS__}) / sizeof(struct lfs_mattr)
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// operations on global state
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
static inline void lfs_gstate_xor(lfs_gstate_t *a, const lfs_gstate_t *b) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
for (int i = 0; i < 3; i++) {
((uint32_t*)a)[i] ^= ((const uint32_t*)b)[i];
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
static inline bool lfs_gstate_iszero(const lfs_gstate_t *a) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
for (int i = 0; i < 3; i++) {
if (((uint32_t*)a)[i] != 0) {
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
return false;
}
}
return true;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
static inline bool lfs_gstate_hasorphans(const lfs_gstate_t *a) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
return lfs_tag_size(a->tag);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
static inline uint8_t lfs_gstate_getorphans(const lfs_gstate_t *a) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
return lfs_tag_size(a->tag);
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
static inline bool lfs_gstate_hasmove(const lfs_gstate_t *a) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
return lfs_tag_type1(a->tag);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
static inline bool lfs_gstate_hasmovehere(const lfs_gstate_t *a,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
const lfs_block_t *pair) {
return lfs_tag_type1(a->tag) && lfs_pair_cmp(a->pair, pair) == 0;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
static inline void lfs_gstate_fromle32(lfs_gstate_t *a) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
a->tag = lfs_fromle32(a->tag);
a->pair[0] = lfs_fromle32(a->pair[0]);
a->pair[1] = lfs_fromle32(a->pair[1]);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
static inline void lfs_gstate_tole32(lfs_gstate_t *a) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
a->tag = lfs_tole32(a->tag);
a->pair[0] = lfs_tole32(a->pair[0]);
a->pair[1] = lfs_tole32(a->pair[1]);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// other endianness operations
static void lfs_ctz_fromle32(struct lfs_ctz *ctz) {
ctz->head = lfs_fromle32(ctz->head);
ctz->size = lfs_fromle32(ctz->size);
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static void lfs_ctz_tole32(struct lfs_ctz *ctz) {
ctz->head = lfs_tole32(ctz->head);
ctz->size = lfs_tole32(ctz->size);
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline void lfs_superblock_fromle32(lfs_superblock_t *superblock) {
superblock->version = lfs_fromle32(superblock->version);
superblock->block_size = lfs_fromle32(superblock->block_size);
superblock->block_count = lfs_fromle32(superblock->block_count);
Changed attr_max to be specific to custom attributes While technically, both system and user attributes share the same disk limitations, that's not what attr_max represents when considered from the user's perspective. To the user, attr_max applies only to custom attributes. This means attr_max should not impact other configurable limitations, such as inline files, and the ordering should be reconsidered with what the user finds most important.
2018-10-02 23:28:37 +00:00
superblock->name_max = lfs_fromle32(superblock->name_max);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
superblock->file_max = lfs_fromle32(superblock->file_max);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
superblock->attr_max = lfs_fromle32(superblock->attr_max);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline void lfs_superblock_tole32(lfs_superblock_t *superblock) {
superblock->version = lfs_tole32(superblock->version);
superblock->block_size = lfs_tole32(superblock->block_size);
superblock->block_count = lfs_tole32(superblock->block_count);
Changed attr_max to be specific to custom attributes While technically, both system and user attributes share the same disk limitations, that's not what attr_max represents when considered from the user's perspective. To the user, attr_max applies only to custom attributes. This means attr_max should not impact other configurable limitations, such as inline files, and the ordering should be reconsidered with what the user finds most important.
2018-10-02 23:28:37 +00:00
superblock->name_max = lfs_tole32(superblock->name_max);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
superblock->file_max = lfs_tole32(superblock->file_max);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
superblock->attr_max = lfs_tole32(superblock->attr_max);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
Added support for entry insertion Currently unused, the insertion of new file entries in arbitrary locations in a metadata-pair is very easy to add into the existing metadata logging. The only tricky things: 1. Name tags must strictly precede any tags related to a file. We can pull this off during a compact, but must make two passes. One for the name tag, one for the file. Though a benefit of this is that now our scans during moves can exit early upon finding the name tag. 1. We need to handle name tags appearing out of order. This makes name tags symmetric to deletes, although it doesn't seem like we can leverage this fact very well. Note this also means we need to make the superblock tag a type of name tag.
2018-09-09 14:01:06 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
/// Internal operations predeclared here ///
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir,
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
const struct lfs_mattr *attrs, int attrcount);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_compact(lfs_t *lfs,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_mdir_t *dir, const struct lfs_mattr *attrs, int attrcount,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_mdir_t *source, uint16_t begin, uint16_t end);
Fixed issue where inline files were not cleaned up Due to the logging nature of metadata pairs, switching from inline files (type3 = 0x201) to CTZ skip-lists (type3 = 0x202) does not explicitly erase inline files, but instead leaves them up to compaction to omit. To save code size, this is handled by the same logic that deduplicates tags. Unfortunately, this wasn't working. Due to a relatively late change in v2 the struct's type field was changed to no longer be a part of determining a tag's "uniqueness". A part of this should have been the modification of directory traversal filtering to respect type-dependent uniqueness, but I missed this. The fix is to add in correct type-dependent filtering. Also there was some clean up necessary around removing delete tags during compaction and outlining files. Note that while this appears to conflict with the possibility of combining inline + ctz files, we still have the device-side-only LFS_TYPE_FROM tag that can be repurposed for 256 additional inline "chunks". Found by Johnxjj
2019-07-09 22:51:15 +00:00
static int lfs_file_outline(lfs_t *lfs, lfs_file_t *file);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file);
static void lfs_fs_preporphans(lfs_t *lfs, int8_t orphans);
static void lfs_fs_prepmove(lfs_t *lfs,
uint16_t id, const lfs_block_t pair[2]);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static int lfs_fs_pred(lfs_t *lfs, const lfs_block_t dir[2],
lfs_mdir_t *pdir);
static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t dir[2],
lfs_mdir_t *parent);
static int lfs_fs_relocate(lfs_t *lfs,
const lfs_block_t oldpair[2], lfs_block_t newpair[2]);
Fixed lfs_fs_size doubling metadata-pairs This was caused by the previous fix for allocations during lfs_fs_deorphan in this branch. To catch half-orphans during block allocations we needed to duplicate all metadata-pairs reported to lfs_fs_traverse. Unfortunately this causes lfs_fs_size to report 2x the number of metadata-pairs, which would undoubtably confuse users. The fix here is inelegantly simple, just do a different traversale for allocations and size measurements. It reuses the same code but touches slightly different sets of blocks. Unfortunately, this causes the public lfs_fs_traverse and lfs_fs_size functions to split in how they report blocks. This is technically allowed, since lfs_fs_traverse may report blocks multiple times due to CoW behavior, however it's undesirable and I'm sure there will be some confusion. But I don't have a better solution, so from this point lfs_fs_traverse will be reporting 2x metadata-blocks and shouldn't be used for finding the number of available blocks on the filesystem.
2020-02-09 15:05:37 +00:00
int lfs_fs_traverseraw(lfs_t *lfs,
int (*cb)(void *data, lfs_block_t block), void *data,
bool includeorphans);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static int lfs_fs_forceconsistency(lfs_t *lfs);
static int lfs_deinit(lfs_t *lfs);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
#ifdef LFS_MIGRATE
static int lfs1_traverse(lfs_t *lfs,
int (*cb)(void*, lfs_block_t), void *data);
#endif
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
/// Block allocator ///
static int lfs_alloc_lookahead(void *p, lfs_block_t block) {
lfs_t *lfs = (lfs_t*)p;
lfs_block_t off = ((block - lfs->free.off)
+ lfs->cfg->block_count) % lfs->cfg->block_count;
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (off < lfs->free.size) {
lfs->free.buffer[off / 32] |= 1U << (off % 32);
}
return 0;
}
Removed unnecessary randomization of offsets in lfs_alloc_reset On first read, randomizing the allocators offset may seem appropriate for lfs_alloc_reset. However, it ends up using the filesystem-fed pseudorandom seed in situations it wasn't designed for. As noted by gtaska, the combination of using xors for feeding the seed and multiple traverses of the same CRCs can cause the seed to flip to zeros with concerning frequency. Removed the randomization from lfs_alloc_reset, leaving it in only lfs_mount. Found by gtaska
2020-11-20 06:18:13 +00:00
// indicate allocated blocks have been committed into the filesystem, this
// is to prevent blocks from being garbage collected in the middle of a
// commit operation
(chore) updates from PR, we decided not to move forward with changing v1 code since it can be risky. Let's improve the future! Also renamed and moved around a the lookahead free / reset function
2020-03-26 12:47:12 +00:00
static void lfs_alloc_ack(lfs_t *lfs) {
lfs->free.ack = lfs->cfg->block_count;
}
Removed unnecessary randomization of offsets in lfs_alloc_reset On first read, randomizing the allocators offset may seem appropriate for lfs_alloc_reset. However, it ends up using the filesystem-fed pseudorandom seed in situations it wasn't designed for. As noted by gtaska, the combination of using xors for feeding the seed and multiple traverses of the same CRCs can cause the seed to flip to zeros with concerning frequency. Removed the randomization from lfs_alloc_reset, leaving it in only lfs_mount. Found by gtaska
2020-11-20 06:18:13 +00:00
// drop the lookahead buffer, this is done during mounting and failed
// traversals in order to avoid invalid lookahead state
static void lfs_alloc_drop(lfs_t *lfs) {
(chore) updates from PR, we decided not to move forward with changing v1 code since it can be risky. Let's improve the future! Also renamed and moved around a the lookahead free / reset function
2020-03-26 12:47:12 +00:00
lfs->free.size = 0;
lfs->free.i = 0;
lfs_alloc_ack(lfs);
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) {
while (true) {
while (lfs->free.i != lfs->free.size) {
lfs_block_t off = lfs->free.i;
lfs->free.i += 1;
lfs->free.ack -= 1;
if (!(lfs->free.buffer[off / 32] & (1U << (off % 32)))) {
// found a free block
*block = (lfs->free.off + off) % lfs->cfg->block_count;
// eagerly find next off so an alloc ack can
// discredit old lookahead blocks
while (lfs->free.i != lfs->free.size &&
(lfs->free.buffer[lfs->free.i / 32]
& (1U << (lfs->free.i % 32)))) {
lfs->free.i += 1;
lfs->free.ack -= 1;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
return 0;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Added support for entry insertion Currently unused, the insertion of new file entries in arbitrary locations in a metadata-pair is very easy to add into the existing metadata logging. The only tricky things: 1. Name tags must strictly precede any tags related to a file. We can pull this off during a compact, but must make two passes. One for the name tag, one for the file. Though a benefit of this is that now our scans during moves can exit early upon finding the name tag. 1. We need to handle name tags appearing out of order. This makes name tags symmetric to deletes, although it doesn't seem like we can leverage this fact very well. Note this also means we need to make the superblock tag a type of name tag.
2018-09-09 14:01:06 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// check if we have looked at all blocks since last ack
if (lfs->free.ack == 0) {
Tweaked debug output - Changed "No more free space" to be an error as suggested by davidefer - Tweaked output to be more parsable (no space between lfs and warn)
2019-07-16 20:40:26 +00:00
LFS_ERROR("No more free space %"PRIu32,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->free.i + lfs->free.off);
return LFS_ERR_NOSPC;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->free.off = (lfs->free.off + lfs->free.size)
% lfs->cfg->block_count;
Changed lookahead configuration unit to bytes instead of bits The fact that the lookahead buffer uses bits instead of bytes is an internal detail. Poking this through to the user API has caused a decent amount of confusion. Most buffers are provided as bytes and the inconsistency here can be surprising. The use of bytes instead of bits also makes us forward compatible in the case that we want to change the lookahead internal representation (hint segment list). Additionally, we change the configuration name to lookahead_size. This matches other configurations, such as cache_size and read_size, while also notifying the user that something important changed at compile time (by breaking).
2018-10-03 01:18:30 +00:00
lfs->free.size = lfs_min(8*lfs->cfg->lookahead_size, lfs->free.ack);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->free.i = 0;
// find mask of free blocks from tree
Changed lookahead configuration unit to bytes instead of bits The fact that the lookahead buffer uses bits instead of bytes is an internal detail. Poking this through to the user API has caused a decent amount of confusion. Most buffers are provided as bytes and the inconsistency here can be surprising. The use of bytes instead of bits also makes us forward compatible in the case that we want to change the lookahead internal representation (hint segment list). Additionally, we change the configuration name to lookahead_size. This matches other configurations, such as cache_size and read_size, while also notifying the user that something important changed at compile time (by breaking).
2018-10-03 01:18:30 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead_size);
Fixed lfs_fs_size doubling metadata-pairs This was caused by the previous fix for allocations during lfs_fs_deorphan in this branch. To catch half-orphans during block allocations we needed to duplicate all metadata-pairs reported to lfs_fs_traverse. Unfortunately this causes lfs_fs_size to report 2x the number of metadata-pairs, which would undoubtably confuse users. The fix here is inelegantly simple, just do a different traversale for allocations and size measurements. It reuses the same code but touches slightly different sets of blocks. Unfortunately, this causes the public lfs_fs_traverse and lfs_fs_size functions to split in how they report blocks. This is technically allowed, since lfs_fs_traverse may report blocks multiple times due to CoW behavior, however it's undesirable and I'm sure there will be some confusion. But I don't have a better solution, so from this point lfs_fs_traverse will be reporting 2x metadata-blocks and shouldn't be used for finding the number of available blocks on the filesystem.
2020-02-09 15:05:37 +00:00
int err = lfs_fs_traverseraw(lfs, lfs_alloc_lookahead, lfs, true);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
Removed unnecessary randomization of offsets in lfs_alloc_reset On first read, randomizing the allocators offset may seem appropriate for lfs_alloc_reset. However, it ends up using the filesystem-fed pseudorandom seed in situations it wasn't designed for. As noted by gtaska, the combination of using xors for feeding the seed and multiple traverses of the same CRCs can cause the seed to flip to zeros with concerning frequency. Removed the randomization from lfs_alloc_reset, leaving it in only lfs_mount. Found by gtaska
2020-11-20 06:18:13 +00:00
lfs_alloc_drop(lfs);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
return err;
}
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
/// Metadata pair and directory operations ///
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
static lfs_stag_t lfs_dir_getslice(lfs_t *lfs, const lfs_mdir_t *dir,
lfs_tag_t gmask, lfs_tag_t gtag,
lfs_off_t goff, void *gbuffer, lfs_size_t gsize) {
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_off_t off = dir->off;
lfs_tag_t ntag = dir->etag;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t gdiff = 0;
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (lfs_gstate_hasmovehere(&lfs->gdisk, dir->pair) &&
lfs_tag_id(gmask) != 0 &&
lfs_tag_id(lfs->gdisk.tag) <= lfs_tag_id(gtag)) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// synthetic moves
gdiff -= LFS_MKTAG(0, 1, 0);
}
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// iterate over dir block backwards (for faster lookups)
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
while (off >= sizeof(lfs_tag_t) + lfs_tag_dsize(ntag)) {
off -= lfs_tag_dsize(ntag);
lfs_tag_t tag = ntag;
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(ntag),
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->pair[0], off, &ntag, sizeof(ntag));
if (err) {
return err;
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
ntag = (lfs_frombe32(ntag) ^ tag) & 0x7fffffff;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_id(gmask) != 0 &&
lfs_tag_type1(tag) == LFS_TYPE_SPLICE &&
lfs_tag_id(tag) <= lfs_tag_id(gtag - gdiff)) {
if (tag == (LFS_MKTAG(LFS_TYPE_CREATE, 0, 0) |
(LFS_MKTAG(0, 0x3ff, 0) & (gtag - gdiff)))) {
// found where we were created
return LFS_ERR_NOENT;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// move around splices
gdiff += LFS_MKTAG(0, lfs_tag_splice(tag), 0);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if ((gmask & tag) == (gmask & (gtag - gdiff))) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (lfs_tag_isdelete(tag)) {
return LFS_ERR_NOENT;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
lfs_size_t diff = lfs_min(lfs_tag_size(tag), gsize);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, diff,
dir->pair[0], off+sizeof(tag)+goff, gbuffer, diff);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (err) {
return err;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
memset((uint8_t*)gbuffer + diff, 0, gsize - diff);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return tag + gdiff;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return LFS_ERR_NOENT;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
static lfs_stag_t lfs_dir_get(lfs_t *lfs, const lfs_mdir_t *dir,
lfs_tag_t gmask, lfs_tag_t gtag, void *buffer) {
return lfs_dir_getslice(lfs, dir,
gmask, gtag,
0, buffer, lfs_tag_size(gtag));
}
static int lfs_dir_getread(lfs_t *lfs, const lfs_mdir_t *dir,
const lfs_cache_t *pcache, lfs_cache_t *rcache, lfs_size_t hint,
lfs_tag_t gmask, lfs_tag_t gtag,
lfs_off_t off, void *buffer, lfs_size_t size) {
uint8_t *data = buffer;
if (off+size > lfs->cfg->block_size) {
return LFS_ERR_CORRUPT;
}
while (size > 0) {
lfs_size_t diff = size;
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
if (pcache && pcache->block == LFS_BLOCK_INLINE &&
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
off < pcache->off + pcache->size) {
if (off >= pcache->off) {
// is already in pcache?
diff = lfs_min(diff, pcache->size - (off-pcache->off));
memcpy(data, &pcache->buffer[off-pcache->off], diff);
data += diff;
off += diff;
size -= diff;
continue;
}
// pcache takes priority
diff = lfs_min(diff, pcache->off-off);
}
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
if (rcache->block == LFS_BLOCK_INLINE &&
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
off < rcache->off + rcache->size) {
if (off >= rcache->off) {
// is already in rcache?
diff = lfs_min(diff, rcache->size - (off-rcache->off));
memcpy(data, &rcache->buffer[off-rcache->off], diff);
data += diff;
off += diff;
size -= diff;
continue;
}
// rcache takes priority
diff = lfs_min(diff, rcache->off-off);
}
// load to cache, first condition can no longer fail
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
rcache->block = LFS_BLOCK_INLINE;
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
rcache->off = lfs_aligndown(off, lfs->cfg->read_size);
rcache->size = lfs_min(lfs_alignup(off+hint, lfs->cfg->read_size),
lfs->cfg->cache_size);
int err = lfs_dir_getslice(lfs, dir, gmask, gtag,
rcache->off, rcache->buffer, rcache->size);
Fixed issues with reading and caching inline files Kind of a two-fold issue. One, the programming to the middle of inline files was causing the cache to get updated to a half programmed state. While fine, as all programs do occur in order in a block, this is less efficient when writing to inline files as it would cause the inline file to need to be reread even if it fits in the cache. Two, the rereading of the inline file was broken and passed the file's tag all the way to where a user would expect an error. This was easy to fix but adds to the reasons we should have test coverage information. Found by ebinans
2019-05-31 05:58:48 +00:00
if (err < 0) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
return err;
}
}
return 0;
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_traverse_filter(void *p,
lfs_tag_t tag, const void *buffer) {
lfs_tag_t *filtertag = p;
(void)buffer;
Fixed issue where inline files were not cleaned up Due to the logging nature of metadata pairs, switching from inline files (type3 = 0x201) to CTZ skip-lists (type3 = 0x202) does not explicitly erase inline files, but instead leaves them up to compaction to omit. To save code size, this is handled by the same logic that deduplicates tags. Unfortunately, this wasn't working. Due to a relatively late change in v2 the struct's type field was changed to no longer be a part of determining a tag's "uniqueness". A part of this should have been the modification of directory traversal filtering to respect type-dependent uniqueness, but I missed this. The fix is to add in correct type-dependent filtering. Also there was some clean up necessary around removing delete tags during compaction and outlining files. Note that while this appears to conflict with the possibility of combining inline + ctz files, we still have the device-side-only LFS_TYPE_FROM tag that can be repurposed for 256 additional inline "chunks". Found by Johnxjj
2019-07-09 22:51:15 +00:00
// which mask depends on unique bit in tag structure
uint32_t mask = (tag & LFS_MKTAG(0x100, 0, 0))
? LFS_MKTAG(0x7ff, 0x3ff, 0)
: LFS_MKTAG(0x700, 0x3ff, 0);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// check for redundancy
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if ((mask & tag) == (mask & *filtertag) ||
Fixed issue where inline files were not cleaned up Due to the logging nature of metadata pairs, switching from inline files (type3 = 0x201) to CTZ skip-lists (type3 = 0x202) does not explicitly erase inline files, but instead leaves them up to compaction to omit. To save code size, this is handled by the same logic that deduplicates tags. Unfortunately, this wasn't working. Due to a relatively late change in v2 the struct's type field was changed to no longer be a part of determining a tag's "uniqueness". A part of this should have been the modification of directory traversal filtering to respect type-dependent uniqueness, but I missed this. The fix is to add in correct type-dependent filtering. Also there was some clean up necessary around removing delete tags during compaction and outlining files. Note that while this appears to conflict with the possibility of combining inline + ctz files, we still have the device-side-only LFS_TYPE_FROM tag that can be repurposed for 256 additional inline "chunks". Found by Johnxjj
2019-07-09 22:51:15 +00:00
lfs_tag_isdelete(*filtertag) ||
(LFS_MKTAG(0x7ff, 0x3ff, 0) & tag) == (
LFS_MKTAG(LFS_TYPE_DELETE, 0, 0) |
(LFS_MKTAG(0, 0x3ff, 0) & *filtertag))) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
return true;
}
// check if we need to adjust for created/deleted tags
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type1(tag) == LFS_TYPE_SPLICE &&
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_tag_id(tag) <= lfs_tag_id(*filtertag)) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
*filtertag += LFS_MKTAG(0, lfs_tag_splice(tag), 0);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
return false;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static int lfs_dir_traverse(lfs_t *lfs,
const lfs_mdir_t *dir, lfs_off_t off, lfs_tag_t ptag,
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
const struct lfs_mattr *attrs, int attrcount,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_tag_t tmask, lfs_tag_t ttag,
uint16_t begin, uint16_t end, int16_t diff,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
int (*cb)(void *data, lfs_tag_t tag, const void *buffer), void *data) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// iterate over directory and attrs
while (true) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_tag_t tag;
const void *buffer;
struct lfs_diskoff disk;
if (off+lfs_tag_dsize(ptag) < dir->off) {
off += lfs_tag_dsize(ptag);
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(tag),
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
dir->pair[0], off, &tag, sizeof(tag));
if (err) {
return err;
}
tag = (lfs_frombe32(tag) ^ ptag) | 0x80000000;
disk.block = dir->pair[0];
disk.off = off+sizeof(lfs_tag_t);
buffer = &disk;
ptag = tag;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
} else if (attrcount > 0) {
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
tag = attrs[0].tag;
buffer = attrs[0].buffer;
attrs += 1;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
attrcount -= 1;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
} else {
return 0;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
}
lfs_tag_t mask = LFS_MKTAG(0x7ff, 0, 0);
if ((mask & tmask & tag) != (mask & tmask & ttag)) {
continue;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
// do we need to filter? inlining the filtering logic here allows
// for some minor optimizations
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_id(tmask) != 0) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// scan for duplicates and update tag based on creates/deletes
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int filter = lfs_dir_traverse(lfs,
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
dir, off, ptag, attrs, attrcount,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
0, 0, 0, 0, 0,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_dir_traverse_filter, &tag);
if (filter < 0) {
return filter;
}
if (filter) {
continue;
}
// in filter range?
if (!(lfs_tag_id(tag) >= begin && lfs_tag_id(tag) < end)) {
continue;
}
}
// handle special cases for mcu-side operations
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
if (lfs_tag_type3(tag) == LFS_FROM_NOOP) {
// do nothing
} else if (lfs_tag_type3(tag) == LFS_FROM_MOVE) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
uint16_t fromid = lfs_tag_size(tag);
uint16_t toid = lfs_tag_id(tag);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_traverse(lfs,
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
buffer, 0, 0xffffffff, NULL, 0,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x600, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_STRUCT, 0, 0),
fromid, fromid+1, toid-fromid+diff,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
cb, data);
if (err) {
return err;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(tag) == LFS_FROM_USERATTRS) {
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
for (unsigned i = 0; i < lfs_tag_size(tag); i++) {
const struct lfs_attr *a = buffer;
int err = cb(data, LFS_MKTAG(LFS_TYPE_USERATTR + a[i].type,
lfs_tag_id(tag) + diff, a[i].size), a[i].buffer);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
return err;
}
}
} else {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = cb(data, tag + LFS_MKTAG(0, diff, 0), buffer);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
return err;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs,
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
lfs_mdir_t *dir, const lfs_block_t pair[2],
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_tag_t fmask, lfs_tag_t ftag, uint16_t *id,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
int (*cb)(void *data, lfs_tag_t tag, const void *buffer), void *data) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// we can find tag very efficiently during a fetch, since we're already
// scanning the entire directory
lfs_stag_t besttag = -1;
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
// if either block address is invalid we return LFS_ERR_CORRUPT here,
// otherwise later writes to the pair could fail
if (pair[0] >= lfs->cfg->block_count || pair[1] >= lfs->cfg->block_count) {
return LFS_ERR_CORRUPT;
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// find the block with the most recent revision
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
uint32_t revs[2] = {0, 0};
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
int r = 0;
for (int i = 0; i < 2; i++) {
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(revs[i]),
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
pair[i], 0, &revs[i], sizeof(revs[i]));
revs[i] = lfs_fromle32(revs[i]);
if (err && err != LFS_ERR_CORRUPT) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return err;
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
if (err != LFS_ERR_CORRUPT &&
lfs_scmp(revs[i], revs[(i+1)%2]) > 0) {
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
r = i;
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->pair[0] = pair[(r+0)%2];
dir->pair[1] = pair[(r+1)%2];
dir->rev = revs[(r+0)%2];
dir->off = 0; // nonzero = found some commits
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// now scan tags to fetch the actual dir and find possible match
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (int i = 0; i < 2; i++) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_off_t off = 0;
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
lfs_tag_t ptag = 0xffffffff;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
uint16_t tempcount = 0;
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
lfs_block_t temptail[2] = {LFS_BLOCK_NULL, LFS_BLOCK_NULL};
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
bool tempsplit = false;
lfs_stag_t tempbesttag = besttag;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->rev = lfs_tole32(dir->rev);
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
uint32_t crc = lfs_crc(0xffffffff, &dir->rev, sizeof(dir->rev));
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->rev = lfs_fromle32(dir->rev);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
while (true) {
// extract next tag
lfs_tag_t tag;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
off += lfs_tag_dsize(ptag);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, lfs->cfg->block_size,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->pair[0], off, &tag, sizeof(tag));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err == LFS_ERR_CORRUPT) {
// can't continue?
dir->erased = false;
break;
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return err;
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
crc = lfs_crc(crc, &tag, sizeof(tag));
Tweaked tag endianness to catch power-loss after <1 word is written There was an interesting subtlety with the existing layout of tags that could become a problem in the future. Basically, littlefs avoids writing to any region of storage it is not absolutely sure has been erased beforehand. This is a part of limiting the number of assumptions about storage. It's possible a storage technology can't support writes without erases in a way that is undetectable at write time (Maybe changing a bit without an erase decreases the longevity of the information stored on the bit). But the existing layout had a very tiny corner case where this wasn't true. Consider the location of the valid bit in the tag struct: [1|--- 31 ---] ^--- valid bit The responsibility of this bit is to indicate if an attempt has been made to write the following commit. If it is not set (the specific value is dependent on a previous read and identified by the preceeding commit), the assumption is that it is safe to write to the next region because it has been erased previously. If it is set, we check if the next commit is valid, if it isn't (because of CRC failure, likely due to power-loss), we discard the commit. But because an attempt has been made to write to that storage, we must then do a compaction to move to the other block in the metadata-pair. This plan looks good on paper, but what does it look like on storage? The problem is that words in littlefs are in little-endian. So on storage the tag actually looks like this: [- 8 -|- 8 -|- 8 -|1|- 7 -] ^-- valid bit This means that we don't actually set the valid bit before writing the tag! We write the lower bytes first. If we lose power, we may have written 3 bytes without this fact being detectable. We could restructure the tag structure to store the valid bit lower, however because none of the fields are 7 bits, this would make the extraction more costly, and we then lose the ability to check this valid bit with a sign comparison. The simple solution is to just store the tag in big-endian. A small benefit is that this will actually have a negative code cost on big-endian machines. This mixture of endiannesses is frustrating, however it is a pragmatic solution with only a 20-byte code size cost.
2018-10-22 21:42:30 +00:00
tag = lfs_frombe32(tag) ^ ptag;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// next commit not yet programmed or we're not in valid range
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
if (!lfs_tag_isvalid(tag)) {
Added better handling of large program sizes (> 1024) The issue here is how commits handle padding to the nearest program size. This is done by exploiting the size field of the LFS_TYPE_CRC tag that completes the commit. Unfortunately, during developement, the size field shrank in size to make room for more type information, limiting the size field to 1024. Normally this isn't a problem, as very rarely do program sizes exceed 1024 bytes. However, using a simulated block device, user earlephilhower found that exceeding 1024 caused littlefs to crash. To make this corner case behave in a more user friendly manner, I've modified this situtation to treat >1024 program sizes as small commits that don't match the prog size. As a part of this, littlefs also needed to understand that non-matching commits indicate an "unerased" dir block, which would be needed for portability (something which notably lacks testing). This raises the question of if the tag size field size needs to be reconsidered, but to change that at this point would need a new major version. found by earlephilhower
2019-04-09 21:06:43 +00:00
dir->erased = (lfs_tag_type1(ptag) == LFS_TYPE_CRC &&
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
dir->off % lfs->cfg->prog_size == 0);
break;
} else if (off + lfs_tag_dsize(tag) > lfs->cfg->block_size) {
dir->erased = false;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
break;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
ptag = tag;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type1(tag) == LFS_TYPE_CRC) {
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// check the crc attr
uint32_t dcrc;
err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, lfs->cfg->block_size,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->pair[0], off+sizeof(tag), &dcrc, sizeof(dcrc));
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
dir->erased = false;
break;
}
return err;
}
dcrc = lfs_fromle32(dcrc);
if (crc != dcrc) {
dir->erased = false;
break;
}
// reset the next bit if we need to
build: Fix warnings about shift count width difference for 16 bit compiler Build warnings exist on a gcc based 16 bit compiler. Cast relevant types to fix. littlefs/lfs.c: In function 'lfs_gstate_xororphans': littlefs/lfs.c:355:5: warning: left shift count >= width of type littlefs/lfs.c: In function 'lfs_dir_fetchmatch': littlefs/lfs.c:849:17: warning: left shift count >= width of type littlefs/lfs.c: In function 'lfs_dir_commitcrc': littlefs/lfs.c:1278:9: warning: left shift count >= width of type
2019-09-09 03:53:50 +00:00
ptag ^= (lfs_tag_t)(lfs_tag_chunk(tag) & 1U) << 31;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// toss our crc into the filesystem seed for
Switched to CRC as seed collection function instead of xor As noted by gtaska, we are sitting on a better hash-combining function than xor: CRC. Previous issues with xor were solvable, but relying on xor for this isn't really worth the risk when we already have a CRC function readily available. To quote a study found by gtaska: https://michiel.buddingh.eu/distribution-of-hash-values > CRC32 seems to score really well, but its graph is skewed by the results > of Dataset 5 (binary numbers), which may or may not be too synthetic to > be considered a fair benchmark. But even if you substract the results > from that test, it does not fare significantly worse than other, > cryptographic hash functions.
2020-11-20 06:38:41 +00:00
// pseudorandom numbers, note we use another crc here
// as a collection function because it is sufficiently
// random and convenient
lfs->seed = lfs_crc(lfs->seed, &crc, sizeof(crc));
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// update with what's found so far
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
besttag = tempbesttag;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
dir->off = off + lfs_tag_dsize(tag);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->etag = ptag;
dir->count = tempcount;
dir->tail[0] = temptail[0];
dir->tail[1] = temptail[1];
dir->split = tempsplit;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// reset crc
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
crc = 0xffffffff;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
continue;
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// crc the entry first, hopefully leaving it in the cache
for (lfs_off_t j = sizeof(tag); j < lfs_tag_dsize(tag); j++) {
uint8_t dat;
err = lfs_bd_read(lfs,
NULL, &lfs->rcache, lfs->cfg->block_size,
dir->pair[0], off+j, &dat, 1);
if (err) {
if (err == LFS_ERR_CORRUPT) {
dir->erased = false;
break;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return err;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
crc = lfs_crc(crc, &dat, 1);
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// directory modification tags?
if (lfs_tag_type1(tag) == LFS_TYPE_NAME) {
// increase count of files if necessary
if (lfs_tag_id(tag) >= tempcount) {
tempcount = lfs_tag_id(tag) + 1;
}
} else if (lfs_tag_type1(tag) == LFS_TYPE_SPLICE) {
tempcount += lfs_tag_splice(tag);
if (tag == (LFS_MKTAG(LFS_TYPE_DELETE, 0, 0) |
(LFS_MKTAG(0, 0x3ff, 0) & tempbesttag))) {
tempbesttag |= 0x80000000;
} else if (tempbesttag != -1 &&
lfs_tag_id(tag) <= lfs_tag_id(tempbesttag)) {
tempbesttag += LFS_MKTAG(0, lfs_tag_splice(tag), 0);
}
} else if (lfs_tag_type1(tag) == LFS_TYPE_TAIL) {
tempsplit = (lfs_tag_chunk(tag) & 1);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, lfs->cfg->block_size,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
dir->pair[0], off+sizeof(tag), &temptail, 8);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
dir->erased = false;
break;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_pair_fromle32(temptail);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// found a match for our fetcher?
if ((fmask & tag) == (fmask & ftag)) {
int res = cb(data, tag, &(struct lfs_diskoff){
dir->pair[0], off+sizeof(tag)});
if (res < 0) {
if (res == LFS_ERR_CORRUPT) {
dir->erased = false;
break;
}
return res;
}
if (res == LFS_CMP_EQ) {
// found a match
tempbesttag = tag;
Fixed lfs_dir_fetchmatch not understanding overwritten tags Sometimes small, single line code change hides behind it a complicated story. This is one of those times. If you look at this diff, you may note that this is a case of lfs_dir_fetchmatch not correctly handling a tag that invalidates a callback used to search for some condition, in this case a search for a parent, which is invalidated by a later dir tag overwritting the previous dir pair. But how can this happen? Dir-pair-tags are only overwritten during relocations (when a block goes bad or exceeds the block_cycles config option for dynamic wear-leveling). Other dir operations create new directory entries. And the only lfs_dir_fetchmatch condition that relies on overwrites (as opposed to proper deletes) is when we need to find a directory's parent, an operation that only occurs during a _different_ relocation. And a false _positive_, can only happen if we don't have a parent. Which is really unlikely when we search for directory parents! This bug and minimal test case was found by Matthew Renzelmann. In a unfortunate series of events, first a file creation causes a directory split to occur. This creates a new, orphaned metadata-pair containing our new file. However, the revision count on this metadata-pair indicates the pair is due for relocation as a part of wear-leveling. Normally, this is fine, even though this metadata-pair has no parent, the lfs_dir_find should return ENOENT and continue without error. However, here we get hit by our fetchmatch bug. A previous, unrelated relocation overwrites a pair which just happens to contain the block allocated for a new metadata-pair. When we search for a parent, lfs_dir_fetchmatch incorrectly finds this old, outdated metadata pair and incorrectly tells our orphan it's found its parent. As you can imagine the orphan's dissapointment must be immense. So an unfortunately timed dir split triggers a relocation which incorrectly finds a previously written parent that has been outdated by another relocation. As a solution we can outdate our found tag if it is overwritten by an exact match during lfs_dir_fetchmatch. As a part of this I started adding a new set of tests: tests/test_relocations, for aggressive relocations tests. This is already by appended to by another PR. I suspect relocations is relatively under-tested and is becoming more important due to recent improvements in wear-leveling.
2019-11-26 07:21:42 +00:00
} else if ((LFS_MKTAG(0x7ff, 0x3ff, 0) & tag) ==
(LFS_MKTAG(0x7ff, 0x3ff, 0) & tempbesttag)) {
// found an identical tag, but contents didn't match
// this must mean that our besttag has been overwritten
tempbesttag = -1;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (res == LFS_CMP_GT &&
lfs_tag_id(tag) <= lfs_tag_id(tempbesttag)) {
// found a greater match, keep track to keep things sorted
tempbesttag = tag | 0x80000000;
}
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
// consider what we have good enough
if (dir->off > 0) {
// synthetic move
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (lfs_gstate_hasmovehere(&lfs->gdisk, dir->pair)) {
if (lfs_tag_id(lfs->gdisk.tag) == lfs_tag_id(besttag)) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
besttag |= 0x80000000;
} else if (besttag != -1 &&
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs_tag_id(lfs->gdisk.tag) < lfs_tag_id(besttag)) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
besttag -= LFS_MKTAG(0, 1, 0);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// found tag? or found best id?
if (id) {
*id = lfs_min(lfs_tag_id(besttag), dir->count);
}
if (lfs_tag_isvalid(besttag)) {
return besttag;
} else if (lfs_tag_id(besttag) < dir->count) {
return LFS_ERR_NOENT;
} else {
return 0;
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// failed, try the other block?
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_pair_swap(dir->pair);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->rev = revs[(r+1)%2];
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_ERROR("Corrupted dir pair at {0x%"PRIx32", 0x%"PRIx32"}",
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
dir->pair[0], dir->pair[1]);
return LFS_ERR_CORRUPT;
}
static int lfs_dir_fetch(lfs_t *lfs,
lfs_mdir_t *dir, const lfs_block_t pair[2]) {
prepare upstream bugfix of lfs -> call lfs_dir_fetchmatch with ftag=-1 in order to set the invalid bit and never let the function match a dir
2019-05-17 11:48:37 +00:00
// note, mask=-1, tag=-1 can never match a tag since this
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// pattern has the invalid bit set
Fixed implicit change-of-sign warning in lfs_dir_fetch Warning on MDK v5.27.1 Found by geniusgogo
2019-11-26 22:42:49 +00:00
return (int)lfs_dir_fetchmatch(lfs, dir, pair,
(lfs_tag_t)-1, (lfs_tag_t)-1, NULL, NULL, NULL);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static int lfs_dir_getgstate(lfs_t *lfs, const lfs_mdir_t *dir,
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs_gstate_t *gstate) {
lfs_gstate_t temp;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_stag_t res = lfs_dir_get(lfs, dir, LFS_MKTAG(0x7ff, 0, 0),
LFS_MKTAG(LFS_TYPE_MOVESTATE, 0, sizeof(temp)), &temp);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (res < 0 && res != LFS_ERR_NOENT) {
return res;
}
if (res != LFS_ERR_NOENT) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// xor together to find resulting gstate
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs_gstate_fromle32(&temp);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_gstate_xor(gstate, &temp);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
return 0;
}
static int lfs_dir_getinfo(lfs_t *lfs, lfs_mdir_t *dir,
uint16_t id, struct lfs_info *info) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (id == 0x3ff) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// special case for root
strcpy(info->name, "/");
info->type = LFS_TYPE_DIR;
return 0;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t tag = lfs_dir_get(lfs, dir, LFS_MKTAG(0x780, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_NAME, id, lfs->name_max+1), info->name);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (tag < 0) {
lfs_dir_*: Cast error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 05:24:17 +00:00
return (int)tag;
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
info->type = lfs_tag_type3(tag);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
struct lfs_ctz ctz;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
tag = lfs_dir_get(lfs, dir, LFS_MKTAG(0x700, 0x3ff, 0),
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, id, sizeof(ctz)), &ctz);
if (tag < 0) {
lfs_dir_*: Cast error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 05:24:17 +00:00
return (int)tag;
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
lfs_ctz_fromle32(&ctz);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_CTZSTRUCT) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
info->size = ctz.size;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(tag) == LFS_TYPE_INLINESTRUCT) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
info->size = lfs_tag_size(tag);
}
return 0;
}
struct lfs_dir_find_match {
lfs_t *lfs;
const void *name;
lfs_size_t size;
};
static int lfs_dir_find_match(void *data,
lfs_tag_t tag, const void *buffer) {
struct lfs_dir_find_match *name = data;
lfs_t *lfs = name->lfs;
const struct lfs_diskoff *disk = buffer;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// compare with disk
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_size_t diff = lfs_min(name->size, lfs_tag_size(tag));
int res = lfs_bd_cmp(lfs,
NULL, &lfs->rcache, diff,
disk->block, disk->off, name->name, diff);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (res != LFS_CMP_EQ) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return res;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// only equal if our size is still the same
if (name->size != lfs_tag_size(tag)) {
return (name->size < lfs_tag_size(tag)) ? LFS_CMP_LT : LFS_CMP_GT;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// found a match!
return LFS_CMP_EQ;
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
Adjust lfs_dir_find return code to ensure 32 bit value. lfs_dir_find returns either a negative return code or a tag. For 32 bit machines with int as 32 bits this co-incides, but for smaller bit processors, we need to ensure a 32 bit value is returned so change the return type to lfs_stag_t.
2019-09-29 22:28:03 +00:00
static lfs_stag_t lfs_dir_find(lfs_t *lfs, lfs_mdir_t *dir,
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
const char **path, uint16_t *id) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// we reduce path to a single name if we can find it
const char *name = *path;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (id) {
*id = 0x3ff;
}
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// default to root dir
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t tag = LFS_MKTAG(LFS_TYPE_DIR, 0x3ff, 0);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
dir->tail[0] = lfs->root[0];
dir->tail[1] = lfs->root[1];
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
while (true) {
nextname:
// skip slashes
name += strspn(name, "/");
lfs_size_t namelen = strcspn(name, "/");
// skip '.' and root '..'
if ((namelen == 1 && memcmp(name, ".", 1) == 0) ||
(namelen == 2 && memcmp(name, "..", 2) == 0)) {
name += namelen;
goto nextname;
}
// skip if matched by '..' in name
const char *suffix = name + namelen;
lfs_size_t sufflen;
int depth = 1;
while (true) {
suffix += strspn(suffix, "/");
sufflen = strcspn(suffix, "/");
if (sufflen == 0) {
break;
}
if (sufflen == 2 && memcmp(suffix, "..", 2) == 0) {
depth -= 1;
if (depth == 0) {
name = suffix + sufflen;
goto nextname;
}
} else {
depth += 1;
}
suffix += sufflen;
}
// found path
if (name[0] == '\0') {
return tag;
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
// update what we've found so far
*path = name;
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// only continue if we hit a directory
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) != LFS_TYPE_DIR) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
return LFS_ERR_NOTDIR;
}
// grab the entry data
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_id(tag) != 0x3ff) {
lfs_stag_t res = lfs_dir_get(lfs, dir, LFS_MKTAG(0x700, 0x3ff, 0),
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tag_id(tag), 8), dir->tail);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (res < 0) {
return res;
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_pair_fromle32(dir->tail);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
// find entry matching name
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
while (true) {
tag = lfs_dir_fetchmatch(lfs, dir, dir->tail,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x780, 0, 0),
LFS_MKTAG(LFS_TYPE_NAME, 0, namelen),
// are we last name?
(strchr(name, '/') == NULL) ? id : NULL,
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_dir_find_match, &(struct lfs_dir_find_match){
lfs, name, namelen});
if (tag < 0) {
return tag;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (tag) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
break;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (!dir->split) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return LFS_ERR_NOENT;
}
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
// to next name
name += namelen;
}
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// commit logic
struct lfs_commit {
lfs_block_t block;
lfs_off_t off;
lfs_tag_t ptag;
uint32_t crc;
lfs_off_t begin;
lfs_off_t end;
};
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_commitprog(lfs_t *lfs, struct lfs_commit *commit,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
const void *buffer, lfs_size_t size) {
int err = lfs_bd_prog(lfs,
&lfs->pcache, &lfs->rcache, false,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
commit->block, commit->off ,
(const uint8_t*)buffer, size);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
return err;
}
commit->crc = lfs_crc(commit->crc, buffer, size);
commit->off += size;
return 0;
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_commitattr(lfs_t *lfs, struct lfs_commit *commit,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_tag_t tag, const void *buffer) {
// check if we fit
lfs_size_t dsize = lfs_tag_dsize(tag);
if (commit->off + dsize > commit->end) {
return LFS_ERR_NOSPC;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// write out tag
Tweaked tag endianness to catch power-loss after <1 word is written There was an interesting subtlety with the existing layout of tags that could become a problem in the future. Basically, littlefs avoids writing to any region of storage it is not absolutely sure has been erased beforehand. This is a part of limiting the number of assumptions about storage. It's possible a storage technology can't support writes without erases in a way that is undetectable at write time (Maybe changing a bit without an erase decreases the longevity of the information stored on the bit). But the existing layout had a very tiny corner case where this wasn't true. Consider the location of the valid bit in the tag struct: [1|--- 31 ---] ^--- valid bit The responsibility of this bit is to indicate if an attempt has been made to write the following commit. If it is not set (the specific value is dependent on a previous read and identified by the preceeding commit), the assumption is that it is safe to write to the next region because it has been erased previously. If it is set, we check if the next commit is valid, if it isn't (because of CRC failure, likely due to power-loss), we discard the commit. But because an attempt has been made to write to that storage, we must then do a compaction to move to the other block in the metadata-pair. This plan looks good on paper, but what does it look like on storage? The problem is that words in littlefs are in little-endian. So on storage the tag actually looks like this: [- 8 -|- 8 -|- 8 -|1|- 7 -] ^-- valid bit This means that we don't actually set the valid bit before writing the tag! We write the lower bytes first. If we lose power, we may have written 3 bytes without this fact being detectable. We could restructure the tag structure to store the valid bit lower, however because none of the fields are 7 bits, this would make the extraction more costly, and we then lose the ability to check this valid bit with a sign comparison. The simple solution is to just store the tag in big-endian. A small benefit is that this will actually have a negative code cost on big-endian machines. This mixture of endiannesses is frustrating, however it is a pragmatic solution with only a 20-byte code size cost.
2018-10-22 21:42:30 +00:00
lfs_tag_t ntag = lfs_tobe32((tag & 0x7fffffff) ^ commit->ptag);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
int err = lfs_dir_commitprog(lfs, commit, &ntag, sizeof(ntag));
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
return err;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (!(tag & 0x80000000)) {
// from memory
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitprog(lfs, commit, buffer, dsize-sizeof(tag));
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
return err;
}
} else {
// from disk
const struct lfs_diskoff *disk = buffer;
for (lfs_off_t i = 0; i < dsize-sizeof(tag); i++) {
// rely on caching to make this efficient
uint8_t dat;
err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, dsize-sizeof(tag)-i,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
disk->block, disk->off+i, &dat, 1);
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
if (err) {
return err;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitprog(lfs, commit, &dat, 1);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
return err;
Added support for entry insertion Currently unused, the insertion of new file entries in arbitrary locations in a metadata-pair is very easy to add into the existing metadata logging. The only tricky things: 1. Name tags must strictly precede any tags related to a file. We can pull this off during a compact, but must make two passes. One for the name tag, one for the file. Though a benefit of this is that now our scans during moves can exit early upon finding the name tag. 1. We need to handle name tags appearing out of order. This makes name tags symmetric to deletes, although it doesn't seem like we can leverage this fact very well. Note this also means we need to make the superblock tag a type of name tag.
2018-09-09 14:01:06 +00:00
}
}
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
commit->ptag = tag & 0x7fffffff;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
return 0;
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// align to program units
Fixed single unchecked bit during commit verification This bug was exposed by the bad-block tests due to changes to block allocation, but could have been hit before these changes. In flash, when blocks fail, they don't fail in a predictable manner. To account for this, the bad-block tests check a number of failure behaviors. The interesting one here is "LFS_TESTBD_BADBLOCK_ERASENOOP", in which bad blocks can not be erased or programmed, and are stuck with the data written at the time the blocks go bad. This is actually a pretty realistic failure behavior, since flash needs a large voltage to force the electrons of the floating gates. Though realistically, such a failure would like corrupt the data a bit, not leave the underlying data perfectly intact. LFS_TESTBD_BADBLOCK_ERASENOOP is rather interesting to test for because it means bad blocks can end up with perfectly valid CRCs after a failed write, confusing littlefs. --- In this case, we had the perfect series of operations such that a test was repeatedly writing the same sequence of metadata commits to the same block, which eventually goes bad, leaving the block stuck with metadata that occurs later in the sequence. What this means is that after the first commit, the metadata block contained both the first and second commits, even though the loop in the test hadn't reached that point yet. expected actual .----------. .----------. | commit 1 | | commit 1 | | crc 1 | | crc 1 | | | | commit 2 <-- (from previous iteration) | | | crc 2 | '----------' '----------' To protect against this, littlefs normally compares the written CRC against the expected CRC, but because this was the exact same data that it was going to write, this CRCs end up the same. Ah! But doesn't littlefs also encode the state of the next page to keep track of if the next page has been erased or not? Wouldn't that change between iterations? It does! In a single bit in the CRC-tag. But thanks to some incorrect logic attempting to avoid an extra condition in the loop for writing out padding commits, the CRC that littlefs checked against was the CRC immediately before we include the "is-next-page-erased" bit. Changing the verification check to use the same CRC as what is used to verify commits on fetch solves this problem.
2020-11-22 21:07:16 +00:00
const lfs_off_t end = lfs_alignup(commit->off + 2*sizeof(uint32_t),
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
lfs->cfg->prog_size);
Fixed single unchecked bit during commit verification This bug was exposed by the bad-block tests due to changes to block allocation, but could have been hit before these changes. In flash, when blocks fail, they don't fail in a predictable manner. To account for this, the bad-block tests check a number of failure behaviors. The interesting one here is "LFS_TESTBD_BADBLOCK_ERASENOOP", in which bad blocks can not be erased or programmed, and are stuck with the data written at the time the blocks go bad. This is actually a pretty realistic failure behavior, since flash needs a large voltage to force the electrons of the floating gates. Though realistically, such a failure would like corrupt the data a bit, not leave the underlying data perfectly intact. LFS_TESTBD_BADBLOCK_ERASENOOP is rather interesting to test for because it means bad blocks can end up with perfectly valid CRCs after a failed write, confusing littlefs. --- In this case, we had the perfect series of operations such that a test was repeatedly writing the same sequence of metadata commits to the same block, which eventually goes bad, leaving the block stuck with metadata that occurs later in the sequence. What this means is that after the first commit, the metadata block contained both the first and second commits, even though the loop in the test hadn't reached that point yet. expected actual .----------. .----------. | commit 1 | | commit 1 | | crc 1 | | crc 1 | | | | commit 2 <-- (from previous iteration) | | | crc 2 | '----------' '----------' To protect against this, littlefs normally compares the written CRC against the expected CRC, but because this was the exact same data that it was going to write, this CRCs end up the same. Ah! But doesn't littlefs also encode the state of the next page to keep track of if the next page has been erased or not? Wouldn't that change between iterations? It does! In a single bit in the CRC-tag. But thanks to some incorrect logic attempting to avoid an extra condition in the loop for writing out padding commits, the CRC that littlefs checked against was the CRC immediately before we include the "is-next-page-erased" bit. Changing the verification check to use the same CRC as what is used to verify commits on fetch solves this problem.
2020-11-22 21:07:16 +00:00
lfs_off_t off1 = 0;
uint32_t crc1 = 0;
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
// create crc tags to fill up remainder of commit, note that
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
// padding is not crced, which lets fetches skip padding but
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
// makes committing a bit more complicated
while (commit->off < end) {
lfs_off_t off = commit->off + sizeof(lfs_tag_t);
lfs_off_t noff = lfs_min(end - off, 0x3fe) + off;
if (noff < end) {
noff = lfs_min(noff, end - 2*sizeof(uint32_t));
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
// read erased state from next program unit
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
lfs_tag_t tag = 0xffffffff;
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
int err = lfs_bd_read(lfs,
NULL, &lfs->rcache, sizeof(tag),
commit->block, noff, &tag, sizeof(tag));
if (err && err != LFS_ERR_CORRUPT) {
return err;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
// build crc tag
bool reset = ~lfs_frombe32(tag) >> 31;
tag = LFS_MKTAG(LFS_TYPE_CRC + reset, 0x3ff, noff - off);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
// write out crc
uint32_t footer[2];
footer[0] = lfs_tobe32(tag ^ commit->ptag);
commit->crc = lfs_crc(commit->crc, &footer[0], sizeof(footer[0]));
footer[1] = lfs_tole32(commit->crc);
err = lfs_bd_prog(lfs,
&lfs->pcache, &lfs->rcache, false,
commit->block, commit->off, &footer, sizeof(footer));
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
if (err) {
return err;
}
Fixed single unchecked bit during commit verification This bug was exposed by the bad-block tests due to changes to block allocation, but could have been hit before these changes. In flash, when blocks fail, they don't fail in a predictable manner. To account for this, the bad-block tests check a number of failure behaviors. The interesting one here is "LFS_TESTBD_BADBLOCK_ERASENOOP", in which bad blocks can not be erased or programmed, and are stuck with the data written at the time the blocks go bad. This is actually a pretty realistic failure behavior, since flash needs a large voltage to force the electrons of the floating gates. Though realistically, such a failure would like corrupt the data a bit, not leave the underlying data perfectly intact. LFS_TESTBD_BADBLOCK_ERASENOOP is rather interesting to test for because it means bad blocks can end up with perfectly valid CRCs after a failed write, confusing littlefs. --- In this case, we had the perfect series of operations such that a test was repeatedly writing the same sequence of metadata commits to the same block, which eventually goes bad, leaving the block stuck with metadata that occurs later in the sequence. What this means is that after the first commit, the metadata block contained both the first and second commits, even though the loop in the test hadn't reached that point yet. expected actual .----------. .----------. | commit 1 | | commit 1 | | crc 1 | | crc 1 | | | | commit 2 <-- (from previous iteration) | | | crc 2 | '----------' '----------' To protect against this, littlefs normally compares the written CRC against the expected CRC, but because this was the exact same data that it was going to write, this CRCs end up the same. Ah! But doesn't littlefs also encode the state of the next page to keep track of if the next page has been erased or not? Wouldn't that change between iterations? It does! In a single bit in the CRC-tag. But thanks to some incorrect logic attempting to avoid an extra condition in the loop for writing out padding commits, the CRC that littlefs checked against was the CRC immediately before we include the "is-next-page-erased" bit. Changing the verification check to use the same CRC as what is used to verify commits on fetch solves this problem.
2020-11-22 21:07:16 +00:00
// keep track of non-padding checksum to verify
if (off1 == 0) {
off1 = commit->off + sizeof(uint32_t);
crc1 = commit->crc;
}
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
commit->off += sizeof(tag)+lfs_tag_size(tag);
build: Fix warnings about shift count width difference for 16 bit compiler Build warnings exist on a gcc based 16 bit compiler. Cast relevant types to fix. littlefs/lfs.c: In function 'lfs_gstate_xororphans': littlefs/lfs.c:355:5: warning: left shift count >= width of type littlefs/lfs.c: In function 'lfs_dir_fetchmatch': littlefs/lfs.c:849:17: warning: left shift count >= width of type littlefs/lfs.c: In function 'lfs_dir_commitcrc': littlefs/lfs.c:1278:9: warning: left shift count >= width of type
2019-09-09 03:53:50 +00:00
commit->ptag = tag ^ ((lfs_tag_t)reset << 31);
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
commit->crc = 0xffffffff; // reset crc for next "commit"
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
}
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
// flush buffers
int err = lfs_bd_sync(lfs, &lfs->pcache, &lfs->rcache, false);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
return err;
}
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
// successful commit, check checksums to make sure
lfs_off_t off = commit->begin;
Fixed single unchecked bit during commit verification This bug was exposed by the bad-block tests due to changes to block allocation, but could have been hit before these changes. In flash, when blocks fail, they don't fail in a predictable manner. To account for this, the bad-block tests check a number of failure behaviors. The interesting one here is "LFS_TESTBD_BADBLOCK_ERASENOOP", in which bad blocks can not be erased or programmed, and are stuck with the data written at the time the blocks go bad. This is actually a pretty realistic failure behavior, since flash needs a large voltage to force the electrons of the floating gates. Though realistically, such a failure would like corrupt the data a bit, not leave the underlying data perfectly intact. LFS_TESTBD_BADBLOCK_ERASENOOP is rather interesting to test for because it means bad blocks can end up with perfectly valid CRCs after a failed write, confusing littlefs. --- In this case, we had the perfect series of operations such that a test was repeatedly writing the same sequence of metadata commits to the same block, which eventually goes bad, leaving the block stuck with metadata that occurs later in the sequence. What this means is that after the first commit, the metadata block contained both the first and second commits, even though the loop in the test hadn't reached that point yet. expected actual .----------. .----------. | commit 1 | | commit 1 | | crc 1 | | crc 1 | | | | commit 2 <-- (from previous iteration) | | | crc 2 | '----------' '----------' To protect against this, littlefs normally compares the written CRC against the expected CRC, but because this was the exact same data that it was going to write, this CRCs end up the same. Ah! But doesn't littlefs also encode the state of the next page to keep track of if the next page has been erased or not? Wouldn't that change between iterations? It does! In a single bit in the CRC-tag. But thanks to some incorrect logic attempting to avoid an extra condition in the loop for writing out padding commits, the CRC that littlefs checked against was the CRC immediately before we include the "is-next-page-erased" bit. Changing the verification check to use the same CRC as what is used to verify commits on fetch solves this problem.
2020-11-22 21:07:16 +00:00
lfs_off_t noff = off1;
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
while (off < end) {
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
uint32_t crc = 0xffffffff;
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
for (lfs_off_t i = off; i < noff+sizeof(uint32_t); i++) {
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
// check against written crc, may catch blocks that
// become readonly and match our commit size exactly
if (i == off1 && crc != crc1) {
return LFS_ERR_CORRUPT;
}
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
// leave it up to caching to make this efficient
uint8_t dat;
err = lfs_bd_read(lfs,
NULL, &lfs->rcache, noff+sizeof(uint32_t)-i,
commit->block, i, &dat, 1);
if (err) {
return err;
}
crc = lfs_crc(crc, &dat, 1);
}
// detected write error?
if (crc != 0) {
return LFS_ERR_CORRUPT;
}
// skip padding
off = lfs_min(end - noff, 0x3fe) + noff;
if (off < end) {
off = lfs_min(off, end - 2*sizeof(uint32_t));
}
noff = off + sizeof(uint32_t);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
return 0;
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
static int lfs_dir_alloc(lfs_t *lfs, lfs_mdir_t *dir) {
Added root entry and expanding superblocks Expanding superblocks has been on my wishlist for a while. The basic idea is that instead of maintaining a fixed offset blocks {0, 1} to the the root directory (1 pointer), we maintain a dynamically sized linked-list of superblocks that point to the actual root. If the number of writes to the root exceeds some value, we increase the size of the superblock linked-list. This can leverage existing metadata-pair operations. The revision count for metadata-pairs provides some knowledge on how much wear we've put on the superblock, and the threaded linked-list can also be reused for this purpose. This means superblock expansion is both optional and cheap to implement. Expanding superblocks helps both extremely small and extremely large filesystem (extreme being relative of course). On the small end, we can actually collapse the superblock into the root directory and drop the hard requirement of 4-blocks for the superblock. On the large end, our superblock will now last longer than the rest of the filesystem. Each time we expand, the number of cycles until the superblock dies is increased by a power. Before we were stuck with this layout: level cycles limit layout 1 E^2 390 MiB s0 -> root Now we expand every time a fixed offset is exceeded: level cycles limit layout 0 E 4 KiB s0+root 1 E^2 390 MiB s0 -> root 2 E^3 37 TiB s0 -> s1 -> root 3 E^4 3.6 EiB s0 -> s1 -> s2 -> root ... Where the cycles are the number of cycles before death, and the limit is the worst-case size a filesystem where early superblock death becomes a concern (all writes to root using this formula: E^|s| = E*B, E = erase cycles = 100000, B = block count, assuming 4096 byte blocks). Note we can also store copies of the superblock entry on the expanded superblocks. This may help filesystem recover tools in the future.
2018-08-06 18:30:51 +00:00
// allocate pair of dir blocks (backwards, so we write block 1 first)
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
for (int i = 0; i < 2; i++) {
int err = lfs_alloc(lfs, &dir->pair[(i+1)%2]);
if (err) {
return err;
}
}
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
// zero for reproducability in case initial block is unreadable
dir->rev = 0;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// rather than clobbering one of the blocks we just pretend
// the revision may be valid
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(dir->rev),
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
dir->pair[0], 0, &dir->rev, sizeof(dir->rev));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->rev = lfs_fromle32(dir->rev);
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
if (err && err != LFS_ERR_CORRUPT) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return err;
}
Fixed corner case in block_cycles eviction logic The problem was when we allocate a dir-pair, it's possible for the revision count to immediately overflow and the dir-pair be evicted and returned to the unused blocks without being written even once. In the case that block_cycles = 1, this made it impossible to ever create a dir-pair, even in lfs_format. I've also added a bit of logic to lfs_dir_alloc that will prevent any immediate evictions because of the revision count. found by TheLoneWolfling
2019-01-30 03:53:56 +00:00
// make sure we don't immediately evict
dir->rev += dir->rev & 1;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// set defaults
dir->off = sizeof(dir->rev);
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
dir->etag = 0xffffffff;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->count = 0;
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
dir->tail[0] = LFS_BLOCK_NULL;
dir->tail[1] = LFS_BLOCK_NULL;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->erased = false;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
dir->split = false;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// don't write out yet, let caller take care of that
return 0;
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static int lfs_dir_drop(lfs_t *lfs, lfs_mdir_t *dir, lfs_mdir_t *tail) {
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
// steal state
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
int err = lfs_dir_getgstate(lfs, tail, &lfs->gdelta);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
return err;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// steal tail
lfs_pair_tole32(tail->tail);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, dir, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_TAIL + tail->split, 0x3ff, 8), tail->tail}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(tail->tail);
if (err) {
return err;
}
return 0;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_split(lfs_t *lfs,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_mdir_t *dir, const struct lfs_mattr *attrs, int attrcount,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_mdir_t *source, uint16_t split, uint16_t end) {
// create tail directory
Fixed lfs_dir_fetchmatch not understanding overwritten tags Sometimes small, single line code change hides behind it a complicated story. This is one of those times. If you look at this diff, you may note that this is a case of lfs_dir_fetchmatch not correctly handling a tag that invalidates a callback used to search for some condition, in this case a search for a parent, which is invalidated by a later dir tag overwritting the previous dir pair. But how can this happen? Dir-pair-tags are only overwritten during relocations (when a block goes bad or exceeds the block_cycles config option for dynamic wear-leveling). Other dir operations create new directory entries. And the only lfs_dir_fetchmatch condition that relies on overwrites (as opposed to proper deletes) is when we need to find a directory's parent, an operation that only occurs during a _different_ relocation. And a false _positive_, can only happen if we don't have a parent. Which is really unlikely when we search for directory parents! This bug and minimal test case was found by Matthew Renzelmann. In a unfortunate series of events, first a file creation causes a directory split to occur. This creates a new, orphaned metadata-pair containing our new file. However, the revision count on this metadata-pair indicates the pair is due for relocation as a part of wear-leveling. Normally, this is fine, even though this metadata-pair has no parent, the lfs_dir_find should return ENOENT and continue without error. However, here we get hit by our fetchmatch bug. A previous, unrelated relocation overwrites a pair which just happens to contain the block allocated for a new metadata-pair. When we search for a parent, lfs_dir_fetchmatch incorrectly finds this old, outdated metadata pair and incorrectly tells our orphan it's found its parent. As you can imagine the orphan's dissapointment must be immense. So an unfortunately timed dir split triggers a relocation which incorrectly finds a previously written parent that has been outdated by another relocation. As a solution we can outdate our found tag if it is overwritten by an exact match during lfs_dir_fetchmatch. As a part of this I started adding a new set of tests: tests/test_relocations, for aggressive relocations tests. This is already by appended to by another PR. I suspect relocations is relatively under-tested and is becoming more important due to recent improvements in wear-leveling.
2019-11-26 07:21:42 +00:00
lfs_alloc_ack(lfs);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_mdir_t tail;
int err = lfs_dir_alloc(lfs, &tail);
if (err) {
return err;
}
tail.split = dir->split;
tail.tail[0] = dir->tail[0];
tail.tail[1] = dir->tail[1];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
err = lfs_dir_compact(lfs, &tail, attrs, attrcount, source, split, end);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
return err;
}
dir->tail[0] = tail.pair[0];
dir->tail[1] = tail.pair[1];
dir->split = true;
// update root if needed
if (lfs_pair_cmp(dir->pair, lfs->root) == 0 && split == 0) {
lfs->root[0] = tail.pair[0];
lfs->root[1] = tail.pair[1];
}
return 0;
}
static int lfs_dir_commit_size(void *p, lfs_tag_t tag, const void *buffer) {
lfs_size_t *size = p;
(void)buffer;
*size += lfs_tag_dsize(tag);
return 0;
}
struct lfs_dir_commit_commit {
lfs_t *lfs;
struct lfs_commit *commit;
};
static int lfs_dir_commit_commit(void *p, lfs_tag_t tag, const void *buffer) {
struct lfs_dir_commit_commit *commit = p;
return lfs_dir_commitattr(commit->lfs, commit->commit, tag, buffer);
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_dir_compact(lfs_t *lfs,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_mdir_t *dir, const struct lfs_mattr *attrs, int attrcount,
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *source, uint16_t begin, uint16_t end) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// save some state in case block is bad
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
const lfs_block_t oldpair[2] = {dir->pair[0], dir->pair[1]};
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
bool relocated = false;
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
bool tired = false;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Fixed issue with long names causing unbounded recursion This was caused by any commit containing entries large enough to _always_ force a compaction. This would cause littlefs to think that it would need to split infinitely because there was no base case. The fix here is pretty simple: treat any commit with only a single entry as unsplittable. This forces littlefs to first try overcompacting (fitting more in a block than what has optimal runtime), and then failing that return LFS_ERR_NOSPC for higher layers to handle. found by TheLoneWolfling
2019-01-31 20:54:47 +00:00
// should we split?
while (end - begin > 1) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// find size
lfs_size_t size = 0;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_traverse(lfs,
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
source, 0, 0xffffffff, attrs, attrcount,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x400, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_NAME, 0, 0),
begin, end, -begin,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_dir_commit_size, &size);
if (err) {
return err;
}
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// space is complicated, we need room for tail, crc, gstate,
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
// cleanup delete, and we cap at half a block to give room
Fixed issue with long names causing unbounded recursion This was caused by any commit containing entries large enough to _always_ force a compaction. This would cause littlefs to think that it would need to split infinitely because there was no base case. The fix here is pretty simple: treat any commit with only a single entry as unsplittable. This forces littlefs to first try overcompacting (fitting more in a block than what has optimal runtime), and then failing that return LFS_ERR_NOSPC for higher layers to handle. found by TheLoneWolfling
2019-01-31 20:54:47 +00:00
// for metadata updates.
Artificially limited number of file ids per metadata block This is an expirement to determine which field in the tag structure is the most critical: tag id or tag size. This came from looking at NAND storage and discussions around behaviour of large prog_sizes. Initial exploration indicates that prog_sizes around 2KiB are not _that_ uncommon, and the 1KiB limitation is surprising. It's possible to increase the lfs_tag size to 12-bits (4096), but at the cost of only 8-bit ids (256). [---- 32 ----] a [1|-3-|-- 8 --|-- 10 --|-- 10 --] b [1|-3-|-- 8 --|-- 8 --|-- 12 --] This requires more investigation, but in order to allow us to change the tag sizes with minimal impact I've artificially limited the number of file ids to 0xfe (255) different file ids per metadata pair. If 12-bit lengths turn out to be a bad idea, we can remove the artificial limit without backwards incompatible changes. To avoid breaking users already on v2-alpha, this change will refuse _creating_ file ids > 255, but should read file ids > 255 without issues.
2019-04-09 23:56:53 +00:00
if (end - begin < 0xff &&
size <= lfs_min(lfs->cfg->block_size - 36,
lfs_alignup(lfs->cfg->block_size/2,
lfs->cfg->prog_size))) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
break;
}
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// can't fit, need to split, we should really be finding the
// largest size that fits with a small binary search, but right now
// it's not worth the code size
uint16_t split = (end - begin) / 2;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
err = lfs_dir_split(lfs, dir, attrs, attrcount,
source, begin+split, end);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
// if we fail to split, we may be able to overcompact, unless
// we're too big for even the full block, in which case our
// only option is to error
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (err == LFS_ERR_NOSPC && size <= lfs->cfg->block_size - 36) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
break;
}
return err;
}
end = begin + split;
}
// increment revision count
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
dir->rev += 1;
// If our revision count == n * block_cycles, we should force a relocation,
// this is how littlefs wear-levels at the metadata-pair level. Note that we
// actually use (block_cycles+1)|1, this is to avoid two corner cases:
// 1. block_cycles = 1, which would prevent relocations from terminating
// 2. block_cycles = 2n, which, due to aliasing, would only ever relocate
// one metadata block in the pair, effectively making this useless
Changed block_cycles disable from 0 to -1 As it is now, block_cycles = 0 disables wear leveling. This was a mistake as 0 is the "default" value for several other config options. It's even worse when migrating from v1 as it's easy to miss the addition of block_cycles and end up with a filesystem that is not actually wear-leveling. Clearly, block_cycles = 0 should do anything but disable wear-leveling. Here, I've changed block_cycles = 0 to assert. Forcing users to set a value for block_cycles (500 is suggested). block_cycles can be set to -1 to explicitly disable wear leveling if desired.
2019-07-17 22:05:20 +00:00
if (lfs->cfg->block_cycles > 0 &&
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
(dir->rev % ((lfs->cfg->block_cycles+1)|1) == 0)) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (lfs_pair_cmp(dir->pair, (const lfs_block_t[2]){0, 1}) == 0) {
// oh no! we're writing too much to the superblock,
// should we expand?
lfs_ssize_t res = lfs_fs_size(lfs);
if (res < 0) {
return res;
}
// do we have extra space? littlefs can't reclaim this space
// by itself, so expand cautiously
if ((lfs_size_t)res < lfs->cfg->block_count/2) {
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
LFS_DEBUG("Expanding superblock at rev %"PRIu32, dir->rev);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_split(lfs, dir, attrs, attrcount,
source, begin, end);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err && err != LFS_ERR_NOSPC) {
return err;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// welp, we tried, if we ran out of space there's not much
// we can do, we'll error later if we've become frozen
if (!err) {
end = begin;
}
}
Fix build with -Wundef Part of https://github.com/mongoose-os-libs/vfs-fs-lfs/issues/2
2019-08-07 21:58:13 +00:00
#ifdef LFS_MIGRATE
Stop wear-leveling during migration Stop proactively relocate blocks during migrations, this can cause a number of failure states such: clobbering the v1 superblock if we relocate root, and invalidating directory pointers if we relocate the head of a directory. On top of this, relocations increase the overall complexity of lfs_migration, which is already a delicate operation.
2019-11-26 19:59:45 +00:00
} else if (lfs->lfs1) {
// do not proactively relocate blocks during migrations, this
// can cause a number of failure states such: clobbering the
// v1 superblock if we relocate root, and invalidating directory
// pointers if we relocate the head of a directory. On top of
// this, relocations increase the overall complexity of
// lfs_migration, which is already a delicate operation.
Fixed issue where lfs_migrate would relocate root and corrupt superblock Found during testing, the issue was with lfs_migrate in combination with wear leveling. Normally, we can expect lfs_migrate to be able to respect the user-configured block_cycles. It already has allocation information on which blocks are used by both v1 and v2, so it should be safe to relocate blocks as needed. However, this fell apart when root was relocated. If lfs_migrate found a root that needed migration, it would happily relocate the root. This would normally be fine, except relocating the root has a side-effect of needed to update the superblock. Which, during migration, is in a delicate state of containing both v1's and v2's superblocks in the same metadata pair. If the superblock ends up needing to compact, this would clobber the v1 superblock and corrupt the filesystem during migration. The best fix I could come up with is to specifically dissallow migrating the root directory during migration. Fortunately this is behind the LFS_MIGRATE macro, so the code cost for this check is not normally paid.
2019-07-29 06:34:23 +00:00
#endif
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
} else {
// we're writing too much, time to relocate
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
tired = true;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
goto relocate;
}
}
// begin loop to commit compaction to blocks until a compact sticks
while (true) {
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// setup commit state
struct lfs_commit commit = {
.block = dir->pair[1],
.off = 0,
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
.ptag = 0xffffffff,
.crc = 0xffffffff,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
.begin = 0,
.end = lfs->cfg->block_size - 8,
};
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
// erase block to write to
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
int err = lfs_bd_erase(lfs, dir->pair[1]);
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
// write out header
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
dir->rev = lfs_tole32(dir->rev);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitprog(lfs, &commit,
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
&dir->rev, sizeof(dir->rev));
dir->rev = lfs_fromle32(dir->rev);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// traverse the directory, this time writing out all unique tags
err = lfs_dir_traverse(lfs,
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
source, 0, 0xffffffff, attrs, attrcount,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x400, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_NAME, 0, 0),
begin, end, -begin,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_dir_commit_commit, &(struct lfs_dir_commit_commit){
lfs, &commit});
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
return err;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// commit tail, which may be new after last size check
if (!lfs_pair_isnull(dir->tail)) {
lfs_pair_tole32(dir->tail);
err = lfs_dir_commitattr(lfs, &commit,
LFS_MKTAG(LFS_TYPE_TAIL + dir->split, 0x3ff, 8),
dir->tail);
lfs_pair_fromle32(dir->tail);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// bring over gstate?
lfs_gstate_t delta = {0};
if (!relocated) {
lfs_gstate_xor(&delta, &lfs->gdisk);
lfs_gstate_xor(&delta, &lfs->gstate);
}
lfs_gstate_xor(&delta, &lfs->gdelta);
delta.tag &= ~LFS_MKTAG(0, 0, 0x3ff);
err = lfs_dir_getgstate(lfs, dir, &delta);
if (err) {
return err;
}
if (!lfs_gstate_iszero(&delta)) {
lfs_gstate_tole32(&delta);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitattr(lfs, &commit,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
LFS_MKTAG(LFS_TYPE_MOVESTATE, 0x3ff,
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
sizeof(delta)), &delta);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
}
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// complete commit with crc
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitcrc(lfs, &commit);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
// successful compaction, swap dir pair to indicate most recent
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
LFS_ASSERT(commit.off % lfs->cfg->prog_size == 0);
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
lfs_pair_swap(dir->pair);
dir->count = end - begin;
dir->off = commit.off;
dir->etag = commit.ptag;
Fixed broken wear-leveling when block_cycles = 2n-1 This was an interesting issue found during a GitHub discussion with rmollway and thrasher8390. Blocks in the metadata-pair are relocated every "block_cycles", or, more mathy, when rev % block_cycles == 0 as long as rev += 1 every block write. But there's a problem, rev isn't += 1 every block write. There are two blocks in a metadata-pair, so looking at it from each blocks perspective, rev += 2 every block write. This leads to a sort of aliasing issue, where, if block_cycles is divisible by 2, one block in the metadata-pair is always relocated, and the other block is _never_ relocated. Causing a complete failure of block-level wear-leveling. Fortunately, because of a previous workaround to avoid block_cycles = 1 (since this will cause the relocation algorithm to never terminate), the actual math is rev % (block_cycles+1) == 0. This means the bug only shows its head in the much less likely case where block_cycles is a multiple of 2 plus 1, or, in more mathy terms, block_cycles = 2n+1 for some n. To workaround this we can bitwise or our block_cycles with 1 to force it to never be a multiple of 2n. (Maybe we should do this during initialization? But then block_cycles would need to be mutable.) --- There's a few unrelated changes mixed into this commit that shouldn't be there since I added this as part of a branch of bug fixes I'm putting together rather hastily, so unfortunately this is not easily cherry-pickable.
2020-02-09 14:52:20 +00:00
// update gstate
lfs->gdelta = (lfs_gstate_t){0};
if (!relocated) {
lfs->gdisk = lfs->gstate;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
break;
relocate:
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// commit was corrupted, drop caches and prepare to relocate block
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
relocated = true;
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, &lfs->pcache);
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
if (!tired) {
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Bad block at 0x%"PRIx32, dir->pair[1]);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// can't relocate superblock, filesystem is now frozen
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
if (lfs_pair_cmp(dir->pair, (const lfs_block_t[2]){0, 1}) == 0) {
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_WARN("Superblock 0x%"PRIx32" has become unwritable",
dir->pair[1]);
Changed unwritable superblock to ENOSPC for consistency While ECORRUPT is not a wrong error code, it doesn't match other instances of hitting a corrupt block during write. During writes, if blocks are detected as corrupt their data is evicted and moved to a new clean block. This means that at the end of a disk's lifetime, exhaustion errors will be reported as ENOSPC when littlefs can't find any new block to store the data. This has the benefit of matching behaviour when a new file is written and no more blocks can be found, due to either a small disk or corrupted blocks on disk. To littlefs it's like the disk shrinks in size over time.
2018-08-13 19:20:40 +00:00
return LFS_ERR_NOSPC;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
// relocate half of pair
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
int err = lfs_alloc(lfs, &dir->pair[1]);
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
if (err && (err != LFS_ERR_NOSPC || !tired)) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
return err;
}
Restructured block devices again for better test exploitation Also finished migrating tests with test_relocations and test_exhaustion. The issue I was running into when migrating these tests was a lack of flexibility with what you could do with the block devices. It was possible to hack in some hooks for things like bad blocks and power loss, but it wasn't clean or easily extendable. The solution here was to just put all of these test extensions into a third block device, testbd, that uses the other two example block devices internally. testbd has several useful features for testing. Note this makes it a pretty terrible block device _example_ since these hooks look more complicated than a block device needs to be. - testbd can simulate different erase values, supporting 1s, 0s, other byte patterns, or no erases at all (which can cause surprising bugs). This actually depends on the simulated erase values in ramdb and filebd. I did try to move this out of rambd/filebd, but it's not possible to simulate erases in testbd without buffering entire blocks and creating an excessive amount of extra write operations. - testbd also helps simulate power-loss by containing a "power cycles" counter that is decremented every write operation until it calls exit. This is notably faster than the previous gdb approach, which is valuable since the reentrant tests tend to take a while to resolve. - testbd also tracks wear, which can be manually set and read. This is very useful for testing things like bad block handling, wear leveling, or even changing the effective size of the block device at runtime.
2020-01-16 12:30:40 +00:00
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
tired = false;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
continue;
}
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
if (relocated) {
// update references if we relocated
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Relocating {0x%"PRIx32", 0x%"PRIx32"} "
"-> {0x%"PRIx32", 0x%"PRIx32"}",
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]);
int err = lfs_fs_relocate(lfs, oldpair, dir->pair);
if (err) {
return err;
}
}
Removed file outlining on ENOSPC in lfs_file_sync This was initially added as protection against the case where a file grew to no longer fit in a metadata-pair. While in most cases this should be caught by the math in lfs_file_write, it doesn't handle a problem that can happen if the files metadata is large enough that even small inline files can't fit. This can happen if you combine a small block size with large file names and many custom attributes. But trying to outline on ENOSPC creates creates a lot of problems. If we are actually low on space, this is one of the worst things we can do. Inline files take up less space than CTZ skip-lists, but inline files are rendered useless if we outline inline files as soon as we run low on space. On top of this, the outlining logic tries multiple mdir commits if it gets ENOSPC, which can hide errors if ENOSPC is returned for other reasons. In a perfect world, we would be using a different error code for no-room-in-metadata-pair, and no-blocks-on-disk. For now I've removed the outlining logic and we will need to figure out how to handle this situation more robustly.
2020-01-30 04:05:58 +00:00
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
return 0;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir,
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
const struct lfs_mattr *attrs, int attrcount) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
// check for any inline files that aren't RAM backed and
// forcefully evict them, needed for filesystem consistency
for (lfs_file_t *f = (lfs_file_t*)lfs->mlist; f; f = f->next) {
if (dir != &f->m && lfs_pair_cmp(f->m.pair, dir->pair) == 0 &&
f->type == LFS_TYPE_REG && (f->flags & LFS_F_INLINE) &&
f->ctz.size > lfs->cfg->cache_size) {
Fixed issue where inline files were not cleaned up Due to the logging nature of metadata pairs, switching from inline files (type3 = 0x201) to CTZ skip-lists (type3 = 0x202) does not explicitly erase inline files, but instead leaves them up to compaction to omit. To save code size, this is handled by the same logic that deduplicates tags. Unfortunately, this wasn't working. Due to a relatively late change in v2 the struct's type field was changed to no longer be a part of determining a tag's "uniqueness". A part of this should have been the modification of directory traversal filtering to respect type-dependent uniqueness, but I missed this. The fix is to add in correct type-dependent filtering. Also there was some clean up necessary around removing delete tags during compaction and outlining files. Note that while this appears to conflict with the possibility of combining inline + ctz files, we still have the device-side-only LFS_TYPE_FROM tag that can be repurposed for 256 additional inline "chunks". Found by Johnxjj
2019-07-09 22:51:15 +00:00
int err = lfs_file_outline(lfs, f);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
if (err) {
return err;
}
err = lfs_file_flush(lfs, f);
if (err) {
return err;
}
}
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// calculate changes to the directory
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
lfs_mdir_t olddir = *dir;
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
bool hasdelete = false;
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
for (int i = 0; i < attrcount; i++) {
if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_CREATE) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
dir->count += 1;
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
} else if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_DELETE) {
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
LFS_ASSERT(dir->count > 0);
dir->count -= 1;
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
hasdelete = true;
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
} else if (lfs_tag_type1(attrs[i].tag) == LFS_TYPE_TAIL) {
dir->tail[0] = ((lfs_block_t*)attrs[i].buffer)[0];
dir->tail[1] = ((lfs_block_t*)attrs[i].buffer)[1];
dir->split = (lfs_tag_chunk(attrs[i].tag) & 1);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(dir->tail);
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
}
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
// should we actually drop the directory block?
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
if (hasdelete && dir->count == 0) {
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
lfs_mdir_t pdir;
int err = lfs_fs_pred(lfs, dir->pair, &pdir);
if (err && err != LFS_ERR_NOENT) {
Fixed broken wear-leveling when block_cycles = 2n-1 This was an interesting issue found during a GitHub discussion with rmollway and thrasher8390. Blocks in the metadata-pair are relocated every "block_cycles", or, more mathy, when rev % block_cycles == 0 as long as rev += 1 every block write. But there's a problem, rev isn't += 1 every block write. There are two blocks in a metadata-pair, so looking at it from each blocks perspective, rev += 2 every block write. This leads to a sort of aliasing issue, where, if block_cycles is divisible by 2, one block in the metadata-pair is always relocated, and the other block is _never_ relocated. Causing a complete failure of block-level wear-leveling. Fortunately, because of a previous workaround to avoid block_cycles = 1 (since this will cause the relocation algorithm to never terminate), the actual math is rev % (block_cycles+1) == 0. This means the bug only shows its head in the much less likely case where block_cycles is a multiple of 2 plus 1, or, in more mathy terms, block_cycles = 2n+1 for some n. To workaround this we can bitwise or our block_cycles with 1 to force it to never be a multiple of 2n. (Maybe we should do this during initialization? But then block_cycles would need to be mutable.) --- There's a few unrelated changes mixed into this commit that shouldn't be there since I added this as part of a branch of bug fixes I'm putting together rather hastily, so unfortunately this is not easily cherry-pickable.
2020-02-09 14:52:20 +00:00
*dir = olddir;
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
return err;
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
}
Added support for entry insertion Currently unused, the insertion of new file entries in arbitrary locations in a metadata-pair is very easy to add into the existing metadata logging. The only tricky things: 1. Name tags must strictly precede any tags related to a file. We can pull this off during a compact, but must make two passes. One for the name tag, one for the file. Though a benefit of this is that now our scans during moves can exit early upon finding the name tag. 1. We need to handle name tags appearing out of order. This makes name tags symmetric to deletes, although it doesn't seem like we can leverage this fact very well. Note this also means we need to make the superblock tag a type of name tag.
2018-09-09 14:01:06 +00:00
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
if (err != LFS_ERR_NOENT && pdir.split) {
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
err = lfs_dir_drop(lfs, &pdir, dir);
if (err) {
*dir = olddir;
return err;
}
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
}
Artificially limited number of file ids per metadata block This is an expirement to determine which field in the tag structure is the most critical: tag id or tag size. This came from looking at NAND storage and discussions around behaviour of large prog_sizes. Initial exploration indicates that prog_sizes around 2KiB are not _that_ uncommon, and the 1KiB limitation is surprising. It's possible to increase the lfs_tag size to 12-bits (4096), but at the cost of only 8-bit ids (256). [---- 32 ----] a [1|-3-|-- 8 --|-- 10 --|-- 10 --] b [1|-3-|-- 8 --|-- 8 --|-- 12 --] This requires more investigation, but in order to allow us to change the tag sizes with minimal impact I've artificially limited the number of file ids to 0xfe (255) different file ids per metadata pair. If 12-bit lengths turn out to be a bad idea, we can remove the artificial limit without backwards incompatible changes. To avoid breaking users already on v2-alpha, this change will refuse _creating_ file ids > 255, but should read file ids > 255 without issues.
2019-04-09 23:56:53 +00:00
if (dir->erased || dir->count >= 0xff) {
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// try to commit
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
struct lfs_commit commit = {
.block = dir->pair[0],
.off = dir->off,
.ptag = dir->etag,
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
.crc = 0xffffffff,
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
.begin = dir->off,
.end = lfs->cfg->block_size - 8,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
};
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// traverse attrs that need to be written out
lfs_pair_tole32(dir->tail);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_traverse(lfs,
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
dir, dir->off, dir->etag, attrs, attrcount,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
0, 0, 0, 0, 0,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_dir_commit_commit, &(struct lfs_dir_commit_commit){
lfs, &commit});
lfs_pair_fromle32(dir->tail);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
}
Fixed broken wear-leveling when block_cycles = 2n-1 This was an interesting issue found during a GitHub discussion with rmollway and thrasher8390. Blocks in the metadata-pair are relocated every "block_cycles", or, more mathy, when rev % block_cycles == 0 as long as rev += 1 every block write. But there's a problem, rev isn't += 1 every block write. There are two blocks in a metadata-pair, so looking at it from each blocks perspective, rev += 2 every block write. This leads to a sort of aliasing issue, where, if block_cycles is divisible by 2, one block in the metadata-pair is always relocated, and the other block is _never_ relocated. Causing a complete failure of block-level wear-leveling. Fortunately, because of a previous workaround to avoid block_cycles = 1 (since this will cause the relocation algorithm to never terminate), the actual math is rev % (block_cycles+1) == 0. This means the bug only shows its head in the much less likely case where block_cycles is a multiple of 2 plus 1, or, in more mathy terms, block_cycles = 2n+1 for some n. To workaround this we can bitwise or our block_cycles with 1 to force it to never be a multiple of 2n. (Maybe we should do this during initialization? But then block_cycles would need to be mutable.) --- There's a few unrelated changes mixed into this commit that shouldn't be there since I added this as part of a branch of bug fixes I'm putting together rather hastily, so unfortunately this is not easily cherry-pickable.
2020-02-09 14:52:20 +00:00
*dir = olddir;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
return err;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
// commit any global diffs if we have any
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs_gstate_t delta = {0};
lfs_gstate_xor(&delta, &lfs->gstate);
lfs_gstate_xor(&delta, &lfs->gdisk);
lfs_gstate_xor(&delta, &lfs->gdelta);
delta.tag &= ~LFS_MKTAG(0, 0, 0x3ff);
if (!lfs_gstate_iszero(&delta)) {
err = lfs_dir_getgstate(lfs, dir, &delta);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
Fixed broken wear-leveling when block_cycles = 2n-1 This was an interesting issue found during a GitHub discussion with rmollway and thrasher8390. Blocks in the metadata-pair are relocated every "block_cycles", or, more mathy, when rev % block_cycles == 0 as long as rev += 1 every block write. But there's a problem, rev isn't += 1 every block write. There are two blocks in a metadata-pair, so looking at it from each blocks perspective, rev += 2 every block write. This leads to a sort of aliasing issue, where, if block_cycles is divisible by 2, one block in the metadata-pair is always relocated, and the other block is _never_ relocated. Causing a complete failure of block-level wear-leveling. Fortunately, because of a previous workaround to avoid block_cycles = 1 (since this will cause the relocation algorithm to never terminate), the actual math is rev % (block_cycles+1) == 0. This means the bug only shows its head in the much less likely case where block_cycles is a multiple of 2 plus 1, or, in more mathy terms, block_cycles = 2n+1 for some n. To workaround this we can bitwise or our block_cycles with 1 to force it to never be a multiple of 2n. (Maybe we should do this during initialization? But then block_cycles would need to be mutable.) --- There's a few unrelated changes mixed into this commit that shouldn't be there since I added this as part of a branch of bug fixes I'm putting together rather hastily, so unfortunately this is not easily cherry-pickable.
2020-02-09 14:52:20 +00:00
*dir = olddir;
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
return err;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs_gstate_tole32(&delta);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
err = lfs_dir_commitattr(lfs, &commit,
LFS_MKTAG(LFS_TYPE_MOVESTATE, 0x3ff,
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
sizeof(delta)), &delta);
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
Fixed broken wear-leveling when block_cycles = 2n-1 This was an interesting issue found during a GitHub discussion with rmollway and thrasher8390. Blocks in the metadata-pair are relocated every "block_cycles", or, more mathy, when rev % block_cycles == 0 as long as rev += 1 every block write. But there's a problem, rev isn't += 1 every block write. There are two blocks in a metadata-pair, so looking at it from each blocks perspective, rev += 2 every block write. This leads to a sort of aliasing issue, where, if block_cycles is divisible by 2, one block in the metadata-pair is always relocated, and the other block is _never_ relocated. Causing a complete failure of block-level wear-leveling. Fortunately, because of a previous workaround to avoid block_cycles = 1 (since this will cause the relocation algorithm to never terminate), the actual math is rev % (block_cycles+1) == 0. This means the bug only shows its head in the much less likely case where block_cycles is a multiple of 2 plus 1, or, in more mathy terms, block_cycles = 2n+1 for some n. To workaround this we can bitwise or our block_cycles with 1 to force it to never be a multiple of 2n. (Maybe we should do this during initialization? But then block_cycles would need to be mutable.) --- There's a few unrelated changes mixed into this commit that shouldn't be there since I added this as part of a branch of bug fixes I'm putting together rather hastily, so unfortunately this is not easily cherry-pickable.
2020-02-09 14:52:20 +00:00
*dir = olddir;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
}
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
// finalize commit with the crc
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitcrc(lfs, &commit);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
Fixed broken wear-leveling when block_cycles = 2n-1 This was an interesting issue found during a GitHub discussion with rmollway and thrasher8390. Blocks in the metadata-pair are relocated every "block_cycles", or, more mathy, when rev % block_cycles == 0 as long as rev += 1 every block write. But there's a problem, rev isn't += 1 every block write. There are two blocks in a metadata-pair, so looking at it from each blocks perspective, rev += 2 every block write. This leads to a sort of aliasing issue, where, if block_cycles is divisible by 2, one block in the metadata-pair is always relocated, and the other block is _never_ relocated. Causing a complete failure of block-level wear-leveling. Fortunately, because of a previous workaround to avoid block_cycles = 1 (since this will cause the relocation algorithm to never terminate), the actual math is rev % (block_cycles+1) == 0. This means the bug only shows its head in the much less likely case where block_cycles is a multiple of 2 plus 1, or, in more mathy terms, block_cycles = 2n+1 for some n. To workaround this we can bitwise or our block_cycles with 1 to force it to never be a multiple of 2n. (Maybe we should do this during initialization? But then block_cycles would need to be mutable.) --- There's a few unrelated changes mixed into this commit that shouldn't be there since I added this as part of a branch of bug fixes I'm putting together rather hastily, so unfortunately this is not easily cherry-pickable.
2020-02-09 14:52:20 +00:00
*dir = olddir;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
return err;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// successful commit, update dir
Added a better solution for large prog sizes A current limitation of the lfs tag is the 10-bit (1024) length field. This field is used to indicate padding for commits and effectively limits the size of commits to 1KiB. Because commits must be prog size aligned, this is a problem on devices with prog size > 1024. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ] This can be increased to 12-bit (4096), but for NAND devices this is still to small to completely solve the issue. The previous workaround was to just create unaligned commits. This can occur naturally if littlefs is used on portable media as the prog size does not have to be consistent on different drivers. If littlefs sees an unaligned commit, it treats the dir as unerased and must compact the dir if it creates any new commits. Unfortunately this isn't great. It effectively means that every small commit forced an erase on devices with prog size > 1024. This is pretty terrible. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit |------------------- wasted ---------------------] A different solution, implemented here, is to use multiple crc tags to pad the commit until the remaining space fits in the padding. This effectively looks like multiple empty commits and has a small runtime cost to parse these tags, but otherwise does no harm. [---- 6KiB erase block ----] [-- 2KiB prog size --|-- 2KiB prog size --|-- 2KiB prog size --] [ 1KiB commit | noop | 1KiB commit | noop | 1KiB commit | noop ] It was a bit tricky to implement, but now we can effectively support unlimited prog sizes since there's no limit to the number of commits in a block. found by kazink and joicetm
2019-07-24 19:24:29 +00:00
LFS_ASSERT(commit.off % lfs->cfg->prog_size == 0);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->off = commit.off;
dir->etag = commit.ptag;
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
// and update gstate
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->gdisk = lfs->gstate;
lfs->gdelta = (lfs_gstate_t){0};
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
} else {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
compact:
// fall back to compaction
lfs_cache_drop(lfs, &lfs->pcache);
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_compact(lfs, dir, attrs, attrcount,
dir, 0, dir->count);
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
if (err) {
Fixed broken wear-leveling when block_cycles = 2n-1 This was an interesting issue found during a GitHub discussion with rmollway and thrasher8390. Blocks in the metadata-pair are relocated every "block_cycles", or, more mathy, when rev % block_cycles == 0 as long as rev += 1 every block write. But there's a problem, rev isn't += 1 every block write. There are two blocks in a metadata-pair, so looking at it from each blocks perspective, rev += 2 every block write. This leads to a sort of aliasing issue, where, if block_cycles is divisible by 2, one block in the metadata-pair is always relocated, and the other block is _never_ relocated. Causing a complete failure of block-level wear-leveling. Fortunately, because of a previous workaround to avoid block_cycles = 1 (since this will cause the relocation algorithm to never terminate), the actual math is rev % (block_cycles+1) == 0. This means the bug only shows its head in the much less likely case where block_cycles is a multiple of 2 plus 1, or, in more mathy terms, block_cycles = 2n+1 for some n. To workaround this we can bitwise or our block_cycles with 1 to force it to never be a multiple of 2n. (Maybe we should do this during initialization? But then block_cycles would need to be mutable.) --- There's a few unrelated changes mixed into this commit that shouldn't be there since I added this as part of a branch of bug fixes I'm putting together rather hastily, so unfortunately this is not easily cherry-pickable.
2020-02-09 14:52:20 +00:00
*dir = olddir;
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
return err;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
// this complicated bit of logic is for fixing up any active
// metadata-pairs that we may have affected
//
// note we have to make two passes since the mdir passed to
// lfs_dir_commit could also be in this list, and even then
// we need to copy the pair so they don't get clobbered if we refetch
// our mdir.
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) {
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
if (&d->m != dir && lfs_pair_cmp(d->m.pair, olddir.pair) == 0) {
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
d->m = *dir;
for (int i = 0; i < attrcount; i++) {
if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_DELETE &&
d->id == lfs_tag_id(attrs[i].tag)) {
d->m.pair[0] = LFS_BLOCK_NULL;
d->m.pair[1] = LFS_BLOCK_NULL;
} else if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_DELETE &&
d->id > lfs_tag_id(attrs[i].tag)) {
d->id -= 1;
if (d->type == LFS_TYPE_DIR) {
((lfs_dir_t*)d)->pos -= 1;
}
} else if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_CREATE &&
d->id >= lfs_tag_id(attrs[i].tag)) {
d->id += 1;
if (d->type == LFS_TYPE_DIR) {
((lfs_dir_t*)d)->pos += 1;
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
}
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) {
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
if (lfs_pair_cmp(d->m.pair, olddir.pair) == 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
while (d->id >= d->m.count && d->m.split) {
// we split and id is on tail now
d->id -= d->m.count;
int err = lfs_dir_fetch(lfs, &d->m, d->m.tail);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
return err;
}
}
}
}
return 0;
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level directory operations ///
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
int lfs_mkdir(lfs_t *lfs, const char *path) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir(%p, \"%s\")", (void*)lfs, path);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", err);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
return err;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
struct lfs_mlist cwd;
cwd.next = lfs->mlist;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
uint16_t id;
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_dir_find(lfs, &cwd.m, &path, &id);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (!(err == LFS_ERR_NOENT && id != 0x3ff)) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", (err < 0) ? err : LFS_ERR_EXIST);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return (err < 0) ? err : LFS_ERR_EXIST;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
// check that name fits
lfs_size_t nlen = strlen(path);
Changed name of upper-limits from blah_size to blah_max This standardizes the naming between the LFS_BLAH_MAX macros and the blah_max configuration in the lfs_config structure.
2018-08-05 01:10:08 +00:00
if (nlen > lfs->name_max) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", LFS_ERR_NAMETOOLONG);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return LFS_ERR_NAMETOOLONG;
}
// build up new directory
lfs_alloc_ack(lfs);
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = lfs_dir_alloc(lfs, &dir);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", err);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
// find end of list
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs_mdir_t pred = cwd.m;
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
while (pred.split) {
err = lfs_dir_fetch(lfs, &pred, pred.tail);
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", err);
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
return err;
}
}
// setup dir
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_tole32(pred.tail);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &dir, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), pred.tail}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(pred.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", err);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
// current block end of list?
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (cwd.m.split) {
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
// update tails, this creates a desync
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, +1);
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// it's possible our predecessor has to be relocated, and if
// our parent is our predecessor's predecessor, this could have
// caused our parent to go out of date, fortunately we can hook
// ourselves into littlefs to catch this
cwd.type = 0;
cwd.id = 0;
lfs->mlist = &cwd;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_tole32(dir.pair);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &pred, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), dir.pair}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(dir.pair);
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
if (err) {
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->mlist = cwd.next;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", err);
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
return err;
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->mlist = cwd.next;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -1);
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
}
// now insert into our parent block
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_tole32(dir.pair);
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_dir_commit(lfs, &cwd.m, LFS_MKATTRS(
Enabled -Wextra This only required adding NULLs where commit statements were not fully initialized. Unfortunately we still need -Wno-missing-field-initializers because of a bug in GCC that persists on Travis. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60784 Found by apmorton
2019-02-12 06:11:01 +00:00
{LFS_MKTAG(LFS_TYPE_CREATE, id, 0), NULL},
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
{LFS_MKTAG(LFS_TYPE_DIR, id, nlen), path},
{LFS_MKTAG(LFS_TYPE_DIRSTRUCT, id, 8), dir.pair},
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
{LFS_MKTAG_IF(!cwd.m.split,
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
LFS_TYPE_SOFTTAIL, 0x3ff, 8), dir.pair}));
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_fromle32(dir.pair);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", err);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mkdir -> %d", 0);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_open(lfs_t *lfs, lfs_dir_t *dir, const char *path) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_open(%p, %p, \"%s\")", (void*)lfs, (void*)dir, path);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &dir->m, &path, NULL);
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_dir_open -> %"PRId32, tag);
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
return tag;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) != LFS_TYPE_DIR) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_open -> %d", LFS_ERR_NOTDIR);
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
return LFS_ERR_NOTDIR;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_block_t pair[2];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_id(tag) == 0x3ff) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// handle root dir separately
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
pair[0] = lfs->root[0];
pair[1] = lfs->root[1];
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else {
// get dir pair from parent
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &dir->m, LFS_MKTAG(0x700, 0x3ff, 0),
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tag_id(tag), 8), pair);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (res < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_dir_open -> %"PRId32, res);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return res;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_fromle32(pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// fetch first pair
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_open -> %d", err);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// setup entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->head[0] = dir->m.pair[0];
dir->head[1] = dir->m.pair[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->id = 0;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->pos = 0;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// add to list of mdirs
dir->type = LFS_TYPE_DIR;
dir->next = (lfs_dir_t*)lfs->mlist;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->mlist = (struct lfs_mlist*)dir;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_open -> %d", 0);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_close(%p, %p)", (void*)lfs, (void*)dir);
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// remove from list of mdirs
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (struct lfs_mlist **p = &lfs->mlist; *p; p = &(*p)->next) {
if (*p == (struct lfs_mlist*)dir) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
*p = (*p)->next;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
break;
}
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_close -> %d", 0);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_read(lfs_t *lfs, lfs_dir_t *dir, struct lfs_info *info) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_read(%p, %p, %p)",
(void*)lfs, (void*)dir, (void*)info);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
memset(info, 0, sizeof(*info));
// special offset for '.' and '..'
if (dir->pos == 0) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, ".");
dir->pos += 1;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_read -> %d", true);
return true;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
} else if (dir->pos == 1) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, "..");
dir->pos += 1;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_read -> %d", true);
return true;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
while (true) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (dir->id == dir->m.count) {
if (!dir->m.split) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_read -> %d", false);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return false;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->m.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_read -> %d", err);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
dir->id = 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_getinfo(lfs, &dir->m, dir->id, info);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err && err != LFS_ERR_NOENT) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_read -> %d", err);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
dir->id += 1;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err != LFS_ERR_NOENT) {
break;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
dir->pos += 1;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_read -> %d", true);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return true;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_seek(lfs_t *lfs, lfs_dir_t *dir, lfs_off_t off) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_seek(%p, %p, %"PRIu32")",
(void*)lfs, (void*)dir, off);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// simply walk from head dir
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_rewind(lfs, dir);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_seek -> %d", err);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
return err;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// first two for ./..
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->pos = lfs_min(2, off);
off -= dir->pos;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Fixed issue with superblock breaking lfs_dir_seek The superblock entry takes up id 0 in the root directory (not all entries are files, though currently the superblock is the only exception). Normally, reading a directory correctly skips the superblock and only reports non-superblock files. However, this doesn't work perfectly for lfs_dir_seek, which tries to be clever to not touch the disk. Fortunately, we can fix this by adding an offset for the superblock. This will only work while the superblock is the only non-file entry, otherwise we would need to touch the disk to properly seek in a directory (though we already touch the disk a bit to get dir-tails during seeks). Found by jhartika
2019-11-27 17:27:30 +00:00
// skip superblock entry
dir->id = (off > 0 && lfs_pair_cmp(dir->head, lfs->root) == 0);
while (off > 0) {
int diff = lfs_min(dir->m.count - dir->id, off);
dir->id += diff;
dir->pos += diff;
off -= diff;
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (dir->id == dir->m.count) {
if (!dir->m.split) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_seek -> %d", LFS_ERR_INVAL);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
return LFS_ERR_INVAL;
}
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = lfs_dir_fetch(lfs, &dir->m, dir->m.tail);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_seek -> %d", err);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
return err;
}
Fixed issue with superblock breaking lfs_dir_seek The superblock entry takes up id 0 in the root directory (not all entries are files, though currently the superblock is the only exception). Normally, reading a directory correctly skips the superblock and only reports non-superblock files. However, this doesn't work perfectly for lfs_dir_seek, which tries to be clever to not touch the disk. Fortunately, we can fix this by adding an offset for the superblock. This will only work while the superblock is the only non-file entry, otherwise we would need to touch the disk to properly seek in a directory (though we already touch the disk a bit to get dir-tails during seeks). Found by jhartika
2019-11-27 17:27:30 +00:00
dir->id = 0;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_seek -> %d", 0);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_soff_t lfs_dir_tell(lfs_t *lfs, lfs_dir_t *dir) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_tell(%p, %p)", (void*)lfs, (void*)dir);
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_tell -> %"PRId32, dir->pos);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return dir->pos;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_rewind(lfs_t *lfs, lfs_dir_t *dir) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_rewind(%p, %p)", (void*)lfs, (void*)dir);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
// reload the head dir
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->head);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_rewind -> %d", err);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return err;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->id = 0;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->pos = 0;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_dir_rewind -> %d", 0);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// File index list operations ///
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static int lfs_ctz_index(lfs_t *lfs, lfs_off_t *off) {
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
lfs_off_t size = *off;
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
lfs_off_t b = lfs->cfg->block_size - 2*4;
lfs_off_t i = size / b;
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
if (i == 0) {
return 0;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
i = (size - 4*(lfs_popc(i-1)+2)) / b;
*off = size - b*i - 4*lfs_popc(i);
return i;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static int lfs_ctz_find(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_block_t head, lfs_size_t size,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t pos, lfs_block_t *block, lfs_off_t *off) {
if (size == 0) {
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
*block = LFS_BLOCK_NULL;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
*off = 0;
return 0;
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_off_t current = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
lfs_off_t target = lfs_ctz_index(lfs, &pos);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
while (current > target) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t skip = lfs_min(
lfs_npw2(current-target+1) - 1,
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
lfs_ctz(current));
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int err = lfs_bd_read(lfs,
pcache, rcache, sizeof(head),
head, 4*skip, &head, sizeof(head));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
current -= 1 << skip;
}
*block = head;
*off = pos;
return 0;
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static int lfs_ctz_extend(lfs_t *lfs,
lfs_cache_t *pcache, lfs_cache_t *rcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
lfs_block_t *block, lfs_off_t *off) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
// go ahead and grab a block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
if (err) {
return err;
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
err = lfs_bd_erase(lfs, nblock);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (size == 0) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 0;
return 0;
}
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
lfs_size_t noff = size - 1;
lfs_off_t index = lfs_ctz_index(lfs, &noff);
noff = noff + 1;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
// just copy out the last block if it is incomplete
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
if (noff != lfs->cfg->block_size) {
for (lfs_off_t i = 0; i < noff; i++) {
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
uint8_t data;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_bd_read(lfs,
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
NULL, rcache, noff-i,
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
head, i, &data, 1);
if (err) {
return err;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_bd_prog(lfs,
pcache, rcache, true,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, i, &data, 1);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
*off = noff;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
return 0;
}
// append block
index += 1;
lfs_size_t skips = lfs_ctz(index) + 1;
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
lfs_block_t nhead = head;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
for (lfs_off_t i = 0; i < skips; i++) {
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
nhead = lfs_tole32(nhead);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_bd_prog(lfs, pcache, rcache, true,
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
nblock, 4*i, &nhead, 4);
nhead = lfs_fromle32(nhead);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (i != skips-1) {
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_bd_read(lfs,
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
NULL, rcache, sizeof(nhead),
nhead, 4*i, &nhead, sizeof(nhead));
nhead = lfs_fromle32(nhead);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 4*skips;
return 0;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
relocate:
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Bad block at 0x%"PRIx32, nblock);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// just clear cache and try a new block
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, pcache);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static int lfs_ctz_traverse(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int (*cb)(void*, lfs_block_t), void *data) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (size == 0) {
return 0;
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_off_t index = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
while (true) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int err = cb(data, head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
if (index == 0) {
return 0;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
lfs_block_t heads[2];
int count = 2 - (index & 1);
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
err = lfs_bd_read(lfs,
pcache, rcache, count*sizeof(head),
head, 0, &heads, count*sizeof(head));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
heads[0] = lfs_fromle32(heads[0]);
heads[1] = lfs_fromle32(heads[1]);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
for (int i = 0; i < count-1; i++) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
err = cb(data, heads[i]);
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
if (err) {
return err;
}
}
head = heads[count-1];
index -= count;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level file operations ///
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
int lfs_file_opencfg(lfs_t *lfs, lfs_file_t *file,
const char *path, int flags,
const struct lfs_file_config *cfg) {
Switched to using hex for blocks and ids in debug output This is a minor quality of life change to help debugging, specifically when debugging test failures. Before, the test framework used hex, while the log output used decimal. This was slightly annoying to convert between. Why not output lengths/offset in hex? I don't have a big reason. I find it easier to reason about lengths in decimal and ids (such as addresses or block numbers) in hex. But this may just be me.
2019-07-27 01:09:24 +00:00
LFS_TRACE("lfs_file_opencfg(%p, %p, \"%s\", %x, %p {"
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
".buffer=%p, .attrs=%p, .attr_count=%"PRIu32"})",
(void*)lfs, (void*)file, path, flags,
(void*)cfg, cfg->buffer, (void*)cfg->attrs, cfg->attr_count);
Merge remote-tracking branch 'origin/master' into debug-improvements
2019-07-29 02:53:13 +00:00
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
// deorphan if we haven't yet, needed at most once after poweron
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if ((flags & 3) != LFS_O_RDONLY) {
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_opencfg -> %d", err);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
return err;
}
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// setup simple file details
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
int err;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
file->cfg = cfg;
Set LFS_F_OPENED flag at places required by lfs internal logic.
2019-07-21 12:36:40 +00:00
file->flags = flags | LFS_F_OPENED;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
file->pos = 0;
lfs: initialize file offs field The uninitialized value creates confusion when diagnosing anomalies. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:40:10 +00:00
file->off = 0;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
file->cache.buffer = NULL;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
// allocate entry for file if it doesn't exist
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &file->m, &path, &file->id);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (tag < 0 && !(tag == LFS_ERR_NOENT && file->id != 0x3ff)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = tag;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// get id, add to list of mdirs to catch update changes
file->type = LFS_TYPE_REG;
file->next = (lfs_file_t*)lfs->mlist;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->mlist = (struct lfs_mlist*)file;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag == LFS_ERR_NOENT) {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (!(flags & LFS_O_CREAT)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOENT;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
// check that name fits
lfs_size_t nlen = strlen(path);
Changed name of upper-limits from blah_size to blah_max This standardizes the naming between the LFS_BLAH_MAX macros and the blah_max configuration in the lfs_config structure.
2018-08-05 01:10:08 +00:00
if (nlen > lfs->name_max) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NAMETOOLONG;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
// get next slot and create entry to remember name
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &file->m, LFS_MKATTRS(
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
{LFS_MKTAG(LFS_TYPE_CREATE, file->id, 0), NULL},
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
{LFS_MKTAG(LFS_TYPE_REG, file->id, nlen), path},
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
{LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0), NULL}));
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NAMETOOLONG;
goto cleanup;
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
tag = LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, 0);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
} else if (flags & LFS_O_EXCL) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_EXIST;
goto cleanup;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(tag) != LFS_TYPE_REG) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_ISDIR;
goto cleanup;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
} else if (flags & LFS_O_TRUNC) {
// truncate if requested
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
tag = LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0);
file->flags |= LFS_F_DIRTY;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
// try to load what's on disk, if it's inlined we'll fix it later
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
tag = lfs_dir_get(lfs, &file->m, LFS_MKTAG(0x700, 0x3ff, 0),
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, file->id, 8), &file->ctz);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag < 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = tag;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_ctz_fromle32(&file->ctz);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
// fetch attrs
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
for (unsigned i = 0; i < file->cfg->attr_count; i++) {
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if ((file->flags & 3) != LFS_O_WRONLY) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &file->m,
LFS_MKTAG(0x7ff, 0x3ff, 0),
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
LFS_MKTAG(LFS_TYPE_USERATTR + file->cfg->attrs[i].type,
file->id, file->cfg->attrs[i].size),
file->cfg->attrs[i].buffer);
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if (res < 0 && res != LFS_ERR_NOENT) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = res;
goto cleanup;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
}
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if ((file->flags & 3) != LFS_O_RDONLY) {
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
if (file->cfg->attrs[i].size > lfs->attr_max) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOSPC;
goto cleanup;
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
}
file->flags |= LFS_F_DIRTY;
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
// allocate buffer if needed
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if (file->cfg->buffer) {
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
file->cache.buffer = file->cfg->buffer;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOMEM;
goto cleanup;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
}
Fixed information leaks through reused caches As a shortcut, littlefs never bother to zero any of the buffers is used. It didn't need to because it would always write out the entirety of the data it needed. Unfortunately, this, combined with the extra padding used to align buffers to the nearest prog size, would lead to uninitialized data getting written out to disk. This means unrelated file data could be written to different parts of storage, or worse, information leaked from the malloc calls could be written out to disk unnecessarily. found by rojer
2018-07-06 16:14:30 +00:00
// zero to avoid information leak
lfs_cache_zero(lfs, &file->cache);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_INLINESTRUCT) {
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
// load inline files
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
file->ctz.head = LFS_BLOCK_INLINE;
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
file->ctz.size = lfs_tag_size(tag);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->flags |= LFS_F_INLINE;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->cache.block = file->ctz.head;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->cache.off = 0;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.size = lfs->cfg->cache_size;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
// don't always read (may be new/trunc file)
if (file->ctz.size > 0) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &file->m,
LFS_MKTAG(0x700, 0x3ff, 0),
Added better handling of large program sizes (> 1024) The issue here is how commits handle padding to the nearest program size. This is done by exploiting the size field of the LFS_TYPE_CRC tag that completes the commit. Unfortunately, during developement, the size field shrank in size to make room for more type information, limiting the size field to 1024. Normally this isn't a problem, as very rarely do program sizes exceed 1024 bytes. However, using a simulated block device, user earlephilhower found that exceeding 1024 caused littlefs to crash. To make this corner case behave in a more user friendly manner, I've modified this situtation to treat >1024 program sizes as small commits that don't match the prog size. As a part of this, littlefs also needed to understand that non-matching commits indicate an "unerased" dir block, which would be needed for portability (something which notably lacks testing). This raises the question of if the tag size field size needs to be reconsidered, but to change that at this point would need a new major version. found by earlephilhower
2019-04-09 21:06:43 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, file->id,
lfs_min(file->cache.size, 0x3fe)),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->cache.buffer);
if (res < 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = res;
goto cleanup;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_opencfg -> %d", 0);
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
return 0;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
cleanup:
// clean up lingering resources
file->flags |= LFS_F_ERRED;
lfs_file_close(lfs, file);
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_opencfg -> %d", err);
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
return err;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
int lfs_file_open(lfs_t *lfs, lfs_file_t *file,
const char *path, int flags) {
Switched to using hex for blocks and ids in debug output This is a minor quality of life change to help debugging, specifically when debugging test failures. Before, the test framework used hex, while the log output used decimal. This was slightly annoying to convert between. Why not output lengths/offset in hex? I don't have a big reason. I find it easier to reason about lengths in decimal and ids (such as addresses or block numbers) in hex. But this may just be me.
2019-07-27 01:09:24 +00:00
LFS_TRACE("lfs_file_open(%p, %p, \"%s\", %x)",
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
(void*)lfs, (void*)file, path, flags);
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
static const struct lfs_file_config defaults = {0};
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
int err = lfs_file_opencfg(lfs, file, path, flags, &defaults);
LFS_TRACE("lfs_file_open -> %d", err);
return err;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_close(lfs_t *lfs, lfs_file_t *file) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_close(%p, %p)", (void*)lfs, (void*)file);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int err = lfs_file_sync(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// remove from list of mdirs
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (struct lfs_mlist **p = &lfs->mlist; *p; p = &(*p)->next) {
if (*p == (struct lfs_mlist*)file) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
*p = (*p)->next;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
break;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// clean up memory
Fixed minor memory leak - Fixed memory leak - Change lfs_globals_zero to use memset as this made leak checking more effective - Checked for leaks with valgrind
2018-08-11 18:45:13 +00:00
if (!file->cfg->buffer) {
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
lfs_free(file->cache.buffer);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
file->flags &= ~LFS_F_OPENED;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_close -> %d", err);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
return err;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_relocate(lfs_t *lfs, lfs_file_t *file) {
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
while (true) {
// just relocate what exists into new block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
Adopted redundant cache read in lfs_file_relocate Previously had some custom logic that could be reduced
2017-06-25 22:23:40 +00:00
if (err) {
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
err = lfs_bd_erase(lfs, nblock);
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
// either read from dirty cache or disk
for (lfs_off_t i = 0; i < file->off; i++) {
uint8_t data;
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
if (file->flags & LFS_F_INLINE) {
err = lfs_dir_getread(lfs, &file->m,
// note we evict inline files before they can be dirty
NULL, &file->cache, file->off-i,
LFS_MKTAG(0xfff, 0x1ff, 0),
LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0),
i, &data, 1);
if (err) {
return err;
}
} else {
err = lfs_bd_read(lfs,
&file->cache, &lfs->rcache, file->off-i,
file->block, i, &data, 1);
if (err) {
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
err = lfs_bd_prog(lfs,
&lfs->pcache, &lfs->rcache, true,
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
nblock, i, &data, 1);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
}
// copy over new state of file
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
memcpy(file->cache.buffer, lfs->pcache.buffer, lfs->cfg->cache_size);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
file->cache.block = lfs->pcache.block;
file->cache.off = lfs->pcache.off;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.size = lfs->pcache.size;
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_zero(lfs, &lfs->pcache);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
file->block = nblock;
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
file->flags |= LFS_F_WRITING;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
return 0;
relocate:
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Bad block at 0x%"PRIx32, nblock);
Fixed inline files when inline_max == cache_size The initial implementation of inline files was thrown together fairly quicky, however it has worked well so far and there hasn't been much reason to change it. One shortcut was to trick file writes into thinking they are writing to imaginary blocks. This works well and reuses most of the file code paths, as long as we don't flush the imaginary block out to disk. Initially we did this by limiting inline_max to cache_max-1, ensuring that the cache never fills up and gets flushed. This was a rather dirty hack, the better solution, implemented here, is to handle the representation of an "imaginary" block correctly all the way down into the cache layer. So now for files specifically, the value -1 represents a null pointer, and the value -2 represents an "imaginary" block. This may become a problem if the number of blocks approaches the max, however this -2 value is never written to disk and can be changed in the future without breaking compatibility.
2018-08-13 19:08:30 +00:00
// just clear cache and try a new block
lfs_cache_drop(lfs, &lfs->pcache);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Fixed issue where inline files were not cleaned up Due to the logging nature of metadata pairs, switching from inline files (type3 = 0x201) to CTZ skip-lists (type3 = 0x202) does not explicitly erase inline files, but instead leaves them up to compaction to omit. To save code size, this is handled by the same logic that deduplicates tags. Unfortunately, this wasn't working. Due to a relatively late change in v2 the struct's type field was changed to no longer be a part of determining a tag's "uniqueness". A part of this should have been the modification of directory traversal filtering to respect type-dependent uniqueness, but I missed this. The fix is to add in correct type-dependent filtering. Also there was some clean up necessary around removing delete tags during compaction and outlining files. Note that while this appears to conflict with the possibility of combining inline + ctz files, we still have the device-side-only LFS_TYPE_FROM tag that can be repurposed for 256 additional inline "chunks". Found by Johnxjj
2019-07-09 22:51:15 +00:00
static int lfs_file_outline(lfs_t *lfs, lfs_file_t *file) {
file->off = file->pos;
lfs_alloc_ack(lfs);
int err = lfs_file_relocate(lfs, file);
if (err) {
return err;
}
file->flags &= ~LFS_F_INLINE;
return 0;
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file) {
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
if (file->flags & LFS_F_READING) {
if (!(file->flags & LFS_F_INLINE)) {
lfs_cache_drop(lfs, &file->cache);
}
file->flags &= ~LFS_F_READING;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (file->flags & LFS_F_WRITING) {
lfs_off_t pos = file->pos;
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (!(file->flags & LFS_F_INLINE)) {
// copy over anything after current branch
lfs_file_t orig = {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
.ctz.head = file->ctz.head,
.ctz.size = file->ctz.size,
Set LFS_F_OPENED flag at places required by lfs internal logic.
2019-07-21 12:36:40 +00:00
.flags = LFS_O_RDONLY | LFS_F_OPENED,
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
.pos = file->pos,
.cache = lfs->rcache,
};
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, &lfs->rcache);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
while (file->pos < file->ctz.size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// copy over a byte at a time, leave it up to caching
// to make this efficient
uint8_t data;
lfs_ssize_t res = lfs_file_read(lfs, &orig, &data, 1);
if (res < 0) {
return res;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
res = lfs_file_write(lfs, file, &data, 1);
if (res < 0) {
return res;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// keep our reference to the rcache in sync
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
if (lfs->rcache.block != LFS_BLOCK_NULL) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, &orig.cache);
lfs_cache_drop(lfs, &lfs->rcache);
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// write out what we have
while (true) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
int err = lfs_bd_flush(lfs, &file->cache, &lfs->rcache, true);
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
break;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
relocate:
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Bad block at 0x%"PRIx32, file->block);
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
err = lfs_file_relocate(lfs, file);
if (err) {
return err;
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
Fixed accidental truncate after seek on inline files The cause was mistakenly setting file->ctz.size directly instead of file->pos, which file->ctz.size gets overwritten with later in lfs_file_flush. Also added better seek test cases specifically for inline files. This should also catch most of the inline corner cases related to lfs_file_size/lfs_file_tell. Found by ebinans
2019-05-28 18:55:03 +00:00
file->pos = lfs_max(file->pos, file->ctz.size);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// actual file updates
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head = file->block;
file->ctz.size = file->pos;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_WRITING;
file->flags |= LFS_F_DIRTY;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = pos;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_sync(%p, %p)", (void*)lfs, (void*)file);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Fixed minor things to get CI passing again - Added caching to Travis install dirs, because otherwise pip3 install fails randomly - Increased size of littlefs-fuse disk because test script has a larger footprint now - Skip a couple of reentrant tests under byte-level writes because the tests just take too long and cause Travis to bail due to no output for 10m - Fixed various Valgrind errors - Suppressed uninit checks for tests where LFS_BLOCK_ERASE_VALUE == -1. In this case rambd goes uninitialized, which is fine for rambd's purposes. Note I couldn't figure out how to limit this suppression to only the malloc in rambd, this doesn't seem possible with Valgrind. - Fixed memory leaks in exhaustion tests - Fixed off-by-1 string null-terminator issue in paths tests - Fixed lfs_file_sync issue caused by revealed by fixing memory leaks in exhaustion tests. Getting ENOSPC during a file write puts the file in a bad state where littlefs doesn't know how to write it out safely. In this case, lfs_file_sync and lfs_file_close return 0 without writing out state so that device-side resources can still be cleaned up. To recover from ENOSPC, the file needs to be reopened and the writes recreated. Not sure if there is a better way to handle this. - Added some quality-of-life improvements to Valgrind testing - Fit Valgrind messages into truncated output when not in verbose mode - Turned on origin tracking
2020-02-17 18:00:47 +00:00
if (file->flags & LFS_F_ERRED) {
// it's not safe to do anything if our file errored
LFS_TRACE("lfs_file_sync -> %d", 0);
return 0;
}
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
int err = lfs_file_flush(lfs, file);
if (err) {
file->flags |= LFS_F_ERRED;
LFS_TRACE("lfs_file_sync -> %d", err);
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
if ((file->flags & LFS_F_DIRTY) &&
!lfs_pair_isnull(file->m.pair)) {
// update dir entry
uint16_t type;
const void *buffer;
lfs_size_t size;
struct lfs_ctz ctz;
if (file->flags & LFS_F_INLINE) {
// inline the whole file
type = LFS_TYPE_INLINESTRUCT;
buffer = file->cache.buffer;
size = file->ctz.size;
} else {
// update the ctz reference
type = LFS_TYPE_CTZSTRUCT;
// copy ctz so alloc will work during a relocate
ctz = file->ctz;
lfs_ctz_tole32(&ctz);
buffer = &ctz;
size = sizeof(ctz);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
// commit file data and attributes
err = lfs_dir_commit(lfs, &file->m, LFS_MKATTRS(
{LFS_MKTAG(type, file->id, size), buffer},
{LFS_MKTAG(LFS_FROM_USERATTRS, file->id,
file->cfg->attr_count), file->cfg->attrs}));
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
if (err) {
Fixed issue with handling block device errors in lfs_file_sync lfs_file_sync was not correctly setting the LFS_F_ERRED flag. Fortunately this is a relatively easy fix. LFS_F_ERRED prevents further issues from occuring when cleaning up resources with lfs_file_close. found by TheLoneWolfling
2019-04-09 22:41:26 +00:00
file->flags |= LFS_F_ERRED;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_sync -> %d", err);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
return err;
}
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
file->flags &= ~LFS_F_DIRTY;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
}
Cleaned up dependent fixes on branch These should probably have been cleaned up in each commit to allow cherry-picking, but due to time I haven't been able to. - Went with creating an mdir copy in lfs_dir_commit. This handles a number of related cleanup issues in lfs_dir_compact and it does so more robustly. As a plus we can use the copy to update dependencies in the mlist. - Eliminated code left by the ENOSPC file outlining - Cleaned up TODOs and lingering comments - Changed the reentrant many directory create/rename/remove test to use a smaller set of directories because of space issues when READ/PROG_SIZE=512
2020-02-09 16:02:41 +00:00
LFS_TRACE("lfs_file_sync -> %d", 0);
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
void *buffer, lfs_size_t size) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_read(%p, %p, %p, %"PRIu32")",
(void*)lfs, (void*)file, buffer, size);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Changed rdonly/wronly mistakes to assert Previously these returned LFS_ERR_BADF. But attempting to modify a file opened read-only, or reading a write-only flie, is a user error and should not occur in normal use. Changing this to an assert allows the logic to be omitted if the user disables asserts to reduce the code footprint (not suggested unless the user really really knows what they're doing).
2019-07-24 19:59:48 +00:00
LFS_ASSERT((file->flags & 3) != LFS_O_WRONLY);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
uint8_t *data = buffer;
lfs_size_t nsize = size;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_WRITING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// flush out any writes
int err = lfs_file_flush(lfs, file);
if (err) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_read -> %d", err);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (file->pos >= file->ctz.size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
// eof if past end
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_read -> %d", 0);
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return 0;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
size = lfs_min(size, file->ctz.size - file->pos);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
nsize = size;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
while (nsize > 0) {
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_READING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_find(lfs, NULL, &file->cache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->pos, &file->block, &file->off);
if (err) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_read -> %d", err);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
file->block = LFS_BLOCK_INLINE;
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
file->off = file->pos;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags |= LFS_F_READING;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
// read as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
if (file->flags & LFS_F_INLINE) {
int err = lfs_dir_getread(lfs, &file->m,
NULL, &file->cache, lfs->cfg->block_size,
LFS_MKTAG(0xfff, 0x1ff, 0),
LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0),
file->off, data, diff);
if (err) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_read -> %d", err);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
return err;
}
} else {
int err = lfs_bd_read(lfs,
NULL, &file->cache, lfs->cfg->block_size,
file->block, file->off, data, diff);
if (err) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_read -> %d", err);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
return err;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
data += diff;
nsize -= diff;
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_read -> %"PRId32, size);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
return size;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
const void *buffer, lfs_size_t size) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_write(%p, %p, %p, %"PRIu32")",
(void*)lfs, (void*)file, buffer, size);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Changed rdonly/wronly mistakes to assert Previously these returned LFS_ERR_BADF. But attempting to modify a file opened read-only, or reading a write-only flie, is a user error and should not occur in normal use. Changing this to an assert allows the logic to be omitted if the user disables asserts to reduce the code footprint (not suggested unless the user really really knows what they're doing).
2019-07-24 19:59:48 +00:00
LFS_ASSERT((file->flags & 3) != LFS_O_RDONLY);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
const uint8_t *data = buffer;
lfs_size_t nsize = size;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_READING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// drop any reads
int err = lfs_file_flush(lfs, file);
if (err) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_write -> %d", err);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if ((file->flags & LFS_O_APPEND) && file->pos < file->ctz.size) {
file->pos = file->ctz.size;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (file->pos + size > lfs->file_max) {
// Larger than file limit?
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_write -> %d", LFS_ERR_FBIG);
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
return LFS_ERR_FBIG;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > file->ctz.size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
// fill with zeros
lfs_off_t pos = file->pos;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->pos = file->ctz.size;
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
while (file->pos < pos) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_write -> %"PRId32, res);
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return res;
}
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if ((file->flags & LFS_F_INLINE) &&
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
lfs_max(file->pos+nsize, file->ctz.size) >
Fixed limit of inline files based on LFS_ATTR_MAX The maximum limit of inline files and attributes are unrelated, but were not at a point in littlefs v2 development. This should be checking against the bit-field limit in the littlefs tag. Found by lsilvaalmeida
2019-05-22 21:18:41 +00:00
lfs_min(0x3fe, lfs_min(
Fixed excessive waste from overly large inline files Before this, there were some safety limits, but there was no real default limit to the size of inline files other than the amount of RAM available. On PCs, this meant that inline files were free to fill up directory blocks to a little under the block size. However this is very wasteful in terms of storage space. Because of splitting limits to keep the compact runtime reasonable, each byte of an inline files uses 4x the amount. Fortunately we can find an optimal inline limit: Inline file waste for n bytes = 3n CTZ file waste for n bytes = B - n Where B = block size Solving for n = B/4 So the optimal inline limit is B/4. However, this assumes a perfect inline file and no metadata. We can decrease this to B/8 to give a bit more breathing room for directory+file metadata.
2019-01-14 23:53:41 +00:00
lfs->cfg->cache_size, lfs->cfg->block_size/8))) {
Bumped versions, cleaned up some TODOs and missing comments
2018-04-03 13:29:28 +00:00
// inline file doesn't fit anymore
Fixed issue where inline files were not cleaned up Due to the logging nature of metadata pairs, switching from inline files (type3 = 0x201) to CTZ skip-lists (type3 = 0x202) does not explicitly erase inline files, but instead leaves them up to compaction to omit. To save code size, this is handled by the same logic that deduplicates tags. Unfortunately, this wasn't working. Due to a relatively late change in v2 the struct's type field was changed to no longer be a part of determining a tag's "uniqueness". A part of this should have been the modification of directory traversal filtering to respect type-dependent uniqueness, but I missed this. The fix is to add in correct type-dependent filtering. Also there was some clean up necessary around removing delete tags during compaction and outlining files. Note that while this appears to conflict with the possibility of combining inline + ctz files, we still have the device-side-only LFS_TYPE_FROM tag that can be repurposed for 256 additional inline "chunks". Found by Johnxjj
2019-07-09 22:51:15 +00:00
int err = lfs_file_outline(lfs, file);
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
if (err) {
file->flags |= LFS_F_ERRED;
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_write -> %d", err);
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
return err;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
}
while (nsize > 0) {
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_WRITING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > 0) {
// find out which block we're extending from
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_find(lfs, NULL, &file->cache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->pos-1, &file->block, &file->off);
if (err) {
file->flags |= LFS_F_ERRED;
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_write -> %d", err);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
return err;
}
// mark cache as dirty since we may have read data into it
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_zero(lfs, &file->cache);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
// extend file with new blocks
lfs_alloc_ack(lfs);
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_extend(lfs, &file->cache, &lfs->rcache,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->block, file->pos,
&file->block, &file->off);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_write -> %d", err);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
file->block = LFS_BLOCK_INLINE;
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
file->off = file->pos;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
file->flags |= LFS_F_WRITING;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// program as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int err = lfs_bd_prog(lfs, &file->cache, &lfs->rcache, true,
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
file->block, file->off, data, diff);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_write -> %d", err);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
break;
relocate:
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
err = lfs_file_relocate(lfs, file);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_write -> %d", err);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
data += diff;
nsize -= diff;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_alloc_ack(lfs);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags &= ~LFS_F_ERRED;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_write -> %"PRId32, size);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
return size;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t off, int whence) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_seek(%p, %p, %"PRId32", %d)",
(void*)lfs, (void*)file, off, whence);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
// write out everything beforehand, may be noop if rdonly
int err = lfs_file_flush(lfs, file);
if (err) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_seek -> %d", err);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return err;
}
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
// find new pos
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
lfs_off_t npos = file->pos;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
if (whence == LFS_SEEK_SET) {
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
npos = off;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
} else if (whence == LFS_SEEK_CUR) {
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
npos = file->pos + off;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
} else if (whence == LFS_SEEK_END) {
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
npos = file->ctz.size + off;
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
}
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
if (npos > lfs->file_max) {
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
// file position out of range
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_seek -> %d", LFS_ERR_INVAL);
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
return LFS_ERR_INVAL;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
// update pos
file->pos = npos;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_seek -> %"PRId32, npos);
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
return npos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_truncate(%p, %p, %"PRIu32")",
(void*)lfs, (void*)file, size);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Changed rdonly/wronly mistakes to assert Previously these returned LFS_ERR_BADF. But attempting to modify a file opened read-only, or reading a write-only flie, is a user error and should not occur in normal use. Changing this to an assert allows the logic to be omitted if the user disables asserts to reduce the code footprint (not suggested unless the user really really knows what they're doing).
2019-07-24 19:59:48 +00:00
LFS_ASSERT((file->flags & 3) != LFS_O_RDONLY);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
Fix: length more than LFS_FILE_MAX should return error To make lfs_file_truncate inline with ftruncate function, when -ve or size more than maximum file size is passed to function it should return invalid parameter error. In LFS case LFS_ERR_INVAL. Signed-off-by: Ajay Bhargav <contact@rickeyworld.info>
2019-02-17 11:39:58 +00:00
if (size > LFS_FILE_MAX) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_truncate -> %d", LFS_ERR_INVAL);
Fix: length more than LFS_FILE_MAX should return error To make lfs_file_truncate inline with ftruncate function, when -ve or size more than maximum file size is passed to function it should return invalid parameter error. In LFS case LFS_ERR_INVAL. Signed-off-by: Ajay Bhargav <contact@rickeyworld.info>
2019-02-17 11:39:58 +00:00
return LFS_ERR_INVAL;
}
lfs: do not reposition seek pointer on truncate When using lfs_file_truncate() to make a file shorter the file block and off were incorrectly positioned at the new end, resulting in invalid data accessed when reading. Lift the seek pointer restoration to apply to both increasing and reducing truncates. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 16:58:19 +00:00
lfs_off_t pos = file->pos;
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
lfs_off_t oldsize = lfs_file_size(lfs, file);
if (size < oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
// need to flush since directly changing metadata
int err = lfs_file_flush(lfs, file);
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_truncate -> %d", err);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
return err;
}
// lookup new head in ctz skip list
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
err = lfs_ctz_find(lfs, NULL, &file->cache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
Fixed issue with writes following a truncate The problem was not setting the file state correctly after the truncate. To truncate < size, we end up using the cache to traverse the ctz skip-list far away from where our file->pos is. We can leave the last block in the cache in case we're going to append to the file, but if we do this we need to set up file->block+file->off to tell use where we are in the file, and set the LFS_F_READING flag to indicate that our cache contains read data. Note this is different than the LFS_F_DIRTY, which we need also. The purpose of the flags are as follows: - LFS_F_DIRTY - file ctz skip-list branch is out of sync with filesystem, need to update metadata - LFS_F_READING - file cache is in use for reading, need to drop cache - LFS_F_WRITING - file cache is in use for writing, need to write out cache to disk The difference between flags is subtle but important because read/prog caches are handled differently. Prog caches have asserts in place to catch programs without erases (the infamous pcache->block == 0xffffffff assert). Though maybe the names deserve an update... Found by ebinans
2019-05-22 19:24:05 +00:00
size, &file->block, &file->off);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_truncate -> %d", err);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
return err;
}
Fixed issue with writes following a truncate The problem was not setting the file state correctly after the truncate. To truncate < size, we end up using the cache to traverse the ctz skip-list far away from where our file->pos is. We can leave the last block in the cache in case we're going to append to the file, but if we do this we need to set up file->block+file->off to tell use where we are in the file, and set the LFS_F_READING flag to indicate that our cache contains read data. Note this is different than the LFS_F_DIRTY, which we need also. The purpose of the flags are as follows: - LFS_F_DIRTY - file ctz skip-list branch is out of sync with filesystem, need to update metadata - LFS_F_READING - file cache is in use for reading, need to drop cache - LFS_F_WRITING - file cache is in use for writing, need to write out cache to disk The difference between flags is subtle but important because read/prog caches are handled differently. Prog caches have asserts in place to catch programs without erases (the infamous pcache->block == 0xffffffff assert). Though maybe the names deserve an update... Found by ebinans
2019-05-22 19:24:05 +00:00
file->ctz.head = file->block;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.size = size;
Fixed issue with writes following a truncate The problem was not setting the file state correctly after the truncate. To truncate < size, we end up using the cache to traverse the ctz skip-list far away from where our file->pos is. We can leave the last block in the cache in case we're going to append to the file, but if we do this we need to set up file->block+file->off to tell use where we are in the file, and set the LFS_F_READING flag to indicate that our cache contains read data. Note this is different than the LFS_F_DIRTY, which we need also. The purpose of the flags are as follows: - LFS_F_DIRTY - file ctz skip-list branch is out of sync with filesystem, need to update metadata - LFS_F_READING - file cache is in use for reading, need to drop cache - LFS_F_WRITING - file cache is in use for writing, need to write out cache to disk The difference between flags is subtle but important because read/prog caches are handled differently. Prog caches have asserts in place to catch programs without erases (the infamous pcache->block == 0xffffffff assert). Though maybe the names deserve an update... Found by ebinans
2019-05-22 19:24:05 +00:00
file->flags |= LFS_F_DIRTY | LFS_F_READING;
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
} else if (size > oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
// flush+seek if not already at end
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
if (file->pos != oldsize) {
lfs_file_truncate: ensure lfs_file_seek return code is lsf_soff_t and cast error returns To ensure 16 bit devices do not invalidly truncate lfs_file_write return codes, change the return variable to be lfs_ssize_t which is the lfs_file_write return code and cast to int if it is a negative error code.
2019-10-01 04:20:43 +00:00
lfs_soff_t res = lfs_file_seek(lfs, file, 0, LFS_SEEK_END);
if (res < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_truncate -> %"PRId32, res);
lfs_file_truncate: ensure lfs_file_seek return code is lsf_soff_t and cast error returns To ensure 16 bit devices do not invalidly truncate lfs_file_write return codes, change the return variable to be lfs_ssize_t which is the lfs_file_write return code and cast to int if it is a negative error code.
2019-10-01 04:20:43 +00:00
return (int)res;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
}
// fill with zeros
while (file->pos < size) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_truncate -> %"PRId32, res);
lfs_file_truncate: ensure lfs_file_seek return code is lsf_soff_t and cast error returns To ensure 16 bit devices do not invalidly truncate lfs_file_write return codes, change the return variable to be lfs_ssize_t which is the lfs_file_write return code and cast to int if it is a negative error code.
2019-10-01 04:20:43 +00:00
return (int)res;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
}
lfs: do not reposition seek pointer on truncate When using lfs_file_truncate() to make a file shorter the file block and off were incorrectly positioned at the new end, resulting in invalid data accessed when reading. Lift the seek pointer restoration to apply to both increasing and reducing truncates. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 16:58:19 +00:00
}
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
lfs: do not reposition seek pointer on truncate When using lfs_file_truncate() to make a file shorter the file block and off were incorrectly positioned at the new end, resulting in invalid data accessed when reading. Lift the seek pointer restoration to apply to both increasing and reducing truncates. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 16:58:19 +00:00
// restore pos
lfs_file_truncate: ensure lfs_file_seek return code is lsf_soff_t and cast error returns To ensure 16 bit devices do not invalidly truncate lfs_file_write return codes, change the return variable to be lfs_ssize_t which is the lfs_file_write return code and cast to int if it is a negative error code.
2019-10-01 04:20:43 +00:00
lfs_soff_t res = lfs_file_seek(lfs, file, pos, LFS_SEEK_SET);
if (res < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_truncate -> %"PRId32, res);
lfs_file_truncate: ensure lfs_file_seek return code is lsf_soff_t and cast error returns To ensure 16 bit devices do not invalidly truncate lfs_file_write return codes, change the return variable to be lfs_ssize_t which is the lfs_file_write return code and cast to int if it is a negative error code.
2019-10-01 04:20:43 +00:00
return (int)res;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_truncate -> %d", 0);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
return 0;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_tell(lfs_t *lfs, lfs_file_t *file) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_tell(%p, %p)", (void*)lfs, (void*)file);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_tell -> %"PRId32, file->pos);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
int lfs_file_rewind(lfs_t *lfs, lfs_file_t *file) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_rewind(%p, %p)", (void*)lfs, (void*)file);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t res = lfs_file_seek(lfs, file, 0, LFS_SEEK_SET);
if (res < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_file_rewind -> %"PRId32, res);
lfs_file_rewind: Cast error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 04:22:25 +00:00
return (int)res;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_rewind -> %d", 0);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_size(%p, %p)", (void*)lfs, (void*)file);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
LFS_ASSERT(file->flags & LFS_F_OPENED);
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (file->flags & LFS_F_WRITING) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_size -> %"PRId32,
lfs_max(file->pos, file->ctz.size));
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return lfs_max(file->pos, file->ctz.size);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
} else {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_file_size -> %"PRId32, file->ctz.size);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return file->ctz.size;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Commentary typo fix
2018-01-30 19:07:37 +00:00
/// General fs operations ///
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
int lfs_stat(lfs_t *lfs, const char *path, struct lfs_info *info) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_stat(%p, \"%s\", %p)", (void*)lfs, path, (void*)info);
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &cwd, &path, NULL);
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_stat -> %"PRId32, tag);
lfs_stat: Cast error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 04:22:01 +00:00
return (int)tag;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
int err = lfs_dir_getinfo(lfs, &cwd, lfs_tag_id(tag), info);
LFS_TRACE("lfs_stat -> %d", err);
return err;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int lfs_remove(lfs_t *lfs, const char *path) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_remove(%p, \"%s\")", (void*)lfs, path);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_remove -> %d", err);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &cwd, &path, NULL);
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
if (tag < 0 || lfs_tag_id(tag) == 0x3ff) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_remove -> %"PRId32, (tag < 0) ? tag : LFS_ERR_INVAL);
lfs_remove: Cast tag/error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 03:56:51 +00:00
return (tag < 0) ? (int)tag : LFS_ERR_INVAL;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
struct lfs_mlist dir;
dir.next = lfs->mlist;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_DIR) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// must be empty before removal
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_block_t pair[2];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &cwd, LFS_MKTAG(0x700, 0x3ff, 0),
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tag_id(tag), 8), pair);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (res < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_remove -> %"PRId32, res);
lfs_remove: Cast tag/error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 03:56:51 +00:00
return (int)res;
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_fromle32(pair);
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_dir_fetch(lfs, &dir.m, pair);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_remove -> %d", err);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return err;
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (dir.m.count > 0 || dir.m.split) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_remove -> %d", LFS_ERR_NOTEMPTY);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return LFS_ERR_NOTEMPTY;
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// mark fs as orphaned
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, +1);
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// I know it's crazy but yes, dir can be changed by our parent's
// commit (if predecessor is child)
dir.type = 0;
dir.id = 0;
lfs->mlist = &dir;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// delete the entry
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &cwd, LFS_MKATTRS(
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
{LFS_MKTAG(LFS_TYPE_DELETE, lfs_tag_id(tag), 0), NULL}));
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->mlist = dir.next;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_remove -> %d", err);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return err;
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->mlist = dir.next;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_DIR) {
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// fix orphan
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -1);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_fs_pred(lfs, dir.m.pair, &cwd);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_remove -> %d", err);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_dir_drop(lfs, &cwd, &dir.m);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_remove -> %d", err);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_remove -> %d", 0);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return 0;
}
int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename(%p, \"%s\", \"%s\")", (void*)lfs, oldpath, newpath);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", err);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// find old entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t oldcwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t oldtag = lfs_dir_find(lfs, &oldcwd, &oldpath, NULL);
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
if (oldtag < 0 || lfs_tag_id(oldtag) == 0x3ff) {
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
LFS_TRACE("lfs_rename -> %"PRId32,
(oldtag < 0) ? oldtag : LFS_ERR_INVAL);
lfs_rename: Cast error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 03:51:52 +00:00
return (oldtag < 0) ? (int)oldtag : LFS_ERR_INVAL;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// find new entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t newcwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
uint16_t newid;
lfs_stag_t prevtag = lfs_dir_find(lfs, &newcwd, &newpath, &newid);
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
if ((prevtag < 0 || lfs_tag_id(prevtag) == 0x3ff) &&
!(prevtag == LFS_ERR_NOENT && newid != 0x3ff)) {
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
LFS_TRACE("lfs_rename -> %"PRId32,
(prevtag < 0) ? prevtag : LFS_ERR_INVAL);
lfs_rename: Cast error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 03:51:52 +00:00
return (prevtag < 0) ? (int)prevtag : LFS_ERR_INVAL;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// if we're in the same pair there's a few special cases...
bool samepair = (lfs_pair_cmp(oldcwd.pair, newcwd.pair) == 0);
uint16_t newoldid = lfs_tag_id(oldtag);
struct lfs_mlist prevdir;
prevdir.next = lfs->mlist;
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
if (prevtag == LFS_ERR_NOENT) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// check that name fits
lfs_size_t nlen = strlen(newpath);
Changed name of upper-limits from blah_size to blah_max This standardizes the naming between the LFS_BLAH_MAX macros and the blah_max configuration in the lfs_config structure.
2018-08-05 01:10:08 +00:00
if (nlen > lfs->name_max) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", LFS_ERR_NAMETOOLONG);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return LFS_ERR_NAMETOOLONG;
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// there is a small chance we are being renamed in the same
// directory/ to an id less than our old id, the global update
// to handle this is a bit messy
if (samepair && newid <= newoldid) {
newoldid += 1;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(prevtag) != lfs_tag_type3(oldtag)) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", LFS_ERR_ISDIR);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
return LFS_ERR_ISDIR;
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
} else if (samepair && newid == newoldid) {
// we're renaming to ourselves??
LFS_TRACE("lfs_rename -> %d", 0);
return 0;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(prevtag) == LFS_TYPE_DIR) {
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// must be empty before removal
lfs_block_t prevpair[2];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &newcwd, LFS_MKTAG(0x700, 0x3ff, 0),
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, newid, 8), prevpair);
if (res < 0) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_rename -> %"PRId32, res);
lfs_rename: Cast error return codes to int. For correctness, cast the lfs_stag_t variables to int when returning a negative error code.
2019-10-01 03:51:52 +00:00
return (int)res;
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
}
lfs_pair_fromle32(prevpair);
// must be empty before removal
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_dir_fetch(lfs, &prevdir.m, prevpair);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", err);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
return err;
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (prevdir.m.count > 0 || prevdir.m.split) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", LFS_ERR_NOTEMPTY);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
return LFS_ERR_NOTEMPTY;
}
// mark fs as orphaned
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, +1);
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// I know it's crazy but yes, dir can be changed by our parent's
// commit (if predecessor is child)
prevdir.type = 0;
prevdir.id = 0;
lfs->mlist = &prevdir;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (!samepair) {
lfs_fs_prepmove(lfs, newoldid, oldcwd.pair);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// move over all attributes
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &newcwd, LFS_MKATTRS(
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
{LFS_MKTAG_IF(prevtag != LFS_ERR_NOENT,
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
LFS_TYPE_DELETE, newid, 0), NULL},
{LFS_MKTAG(LFS_TYPE_CREATE, newid, 0), NULL},
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
{LFS_MKTAG(lfs_tag_type3(oldtag), newid, strlen(newpath)), newpath},
{LFS_MKTAG(LFS_FROM_MOVE, newid, lfs_tag_id(oldtag)), &oldcwd},
{LFS_MKTAG_IF(samepair,
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
LFS_TYPE_DELETE, newoldid, 0), NULL}));
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->mlist = prevdir.next;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", err);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return err;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// let commit clean up after move (if we're different! otherwise move
// logic already fixed it for us)
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
if (!samepair && lfs_gstate_hasmove(&lfs->gstate)) {
// prep gstate and delete move id
lfs_fs_prepmove(lfs, 0x3ff, NULL);
err = lfs_dir_commit(lfs, &oldcwd, LFS_MKATTRS(
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
{LFS_MKTAG(LFS_TYPE_DELETE, lfs_tag_id(oldtag), 0), NULL}));
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
if (err) {
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->mlist = prevdir.next;
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", err);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->mlist = prevdir.next;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (prevtag != LFS_ERR_NOENT && lfs_tag_type3(prevtag) == LFS_TYPE_DIR) {
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// fix orphan
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -1);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_fs_pred(lfs, prevdir.m.pair, &newcwd);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", err);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_dir_drop(lfs, &newcwd, &prevdir.m);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", err);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_rename -> %d", 0);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return 0;
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
lfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path,
uint8_t type, void *buffer, lfs_size_t size) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_getattr(%p, \"%s\", %"PRIu8", %p, %"PRIu32")",
(void*)lfs, path, type, buffer, size);
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &cwd, &path, NULL);
if (tag < 0) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_getattr -> %"PRId32, tag);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return tag;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
uint16_t id = lfs_tag_id(tag);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (id == 0x3ff) {
Dropped lfs_fs_getattr for the more implicit lfs_getattr("/") This was a pretty simple oversight on my part. Conceptually, there's no difference between lfs_fs_getattr and lfs_getattr("/"). Any operations on directories can be applied "globally" by referring to the root directory. Implementation wise, this actually fixes the "corner case" of storing attributes on the root directory, which is broken since the root directory doesn't have a related entry. Instead we need to use the root superblock for this purpose. Fewer functions means less code to document and maintain, so this is a nice benefit. Now we just have a single lfs_getattr/setattr/removeattr set of functions along with the ability to access attributes atomically in lfs_file_opencfg.
2018-08-12 04:05:52 +00:00
// special case for root
id = 0;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_getattr -> %d", err);
Dropped lfs_fs_getattr for the more implicit lfs_getattr("/") This was a pretty simple oversight on my part. Conceptually, there's no difference between lfs_fs_getattr and lfs_getattr("/"). Any operations on directories can be applied "globally" by referring to the root directory. Implementation wise, this actually fixes the "corner case" of storing attributes on the root directory, which is broken since the root directory doesn't have a related entry. Instead we need to use the root superblock for this purpose. Fewer functions means less code to document and maintain, so this is a nice benefit. Now we just have a single lfs_getattr/setattr/removeattr set of functions along with the ability to access attributes atomically in lfs_file_opencfg.
2018-08-12 04:05:52 +00:00
return err;
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
tag = lfs_dir_get(lfs, &cwd, LFS_MKTAG(0x7ff, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_USERATTR + type,
id, lfs_min(size, lfs->attr_max)),
Dropped lfs_fs_getattr for the more implicit lfs_getattr("/") This was a pretty simple oversight on my part. Conceptually, there's no difference between lfs_fs_getattr and lfs_getattr("/"). Any operations on directories can be applied "globally" by referring to the root directory. Implementation wise, this actually fixes the "corner case" of storing attributes on the root directory, which is broken since the root directory doesn't have a related entry. Instead we need to use the root superblock for this purpose. Fewer functions means less code to document and maintain, so this is a nice benefit. Now we just have a single lfs_getattr/setattr/removeattr set of functions along with the ability to access attributes atomically in lfs_file_opencfg.
2018-08-12 04:05:52 +00:00
buffer);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
if (tag < 0) {
if (tag == LFS_ERR_NOENT) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_getattr -> %d", LFS_ERR_NOATTR);
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
return LFS_ERR_NOATTR;
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_getattr -> %"PRId32, tag);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return tag;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
size = lfs_tag_size(tag);
LFS_TRACE("lfs_getattr -> %"PRId32, size);
return size;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
static int lfs_commitattr(lfs_t *lfs, const char *path,
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
uint8_t type, const void *buffer, lfs_size_t size) {
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &cwd, &path, NULL);
if (tag < 0) {
return tag;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
uint16_t id = lfs_tag_id(tag);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (id == 0x3ff) {
Dropped lfs_fs_getattr for the more implicit lfs_getattr("/") This was a pretty simple oversight on my part. Conceptually, there's no difference between lfs_fs_getattr and lfs_getattr("/"). Any operations on directories can be applied "globally" by referring to the root directory. Implementation wise, this actually fixes the "corner case" of storing attributes on the root directory, which is broken since the root directory doesn't have a related entry. Instead we need to use the root superblock for this purpose. Fewer functions means less code to document and maintain, so this is a nice benefit. Now we just have a single lfs_getattr/setattr/removeattr set of functions along with the ability to access attributes atomically in lfs_file_opencfg.
2018-08-12 04:05:52 +00:00
// special case for root
id = 0;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
}
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
return lfs_dir_commit(lfs, &cwd, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_USERATTR + type, id, size), buffer}));
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
int lfs_setattr(lfs_t *lfs, const char *path,
uint8_t type, const void *buffer, lfs_size_t size) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_setattr(%p, \"%s\", %"PRIu8", %p, %"PRIu32")",
(void*)lfs, path, type, buffer, size);
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
if (size > lfs->attr_max) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_setattr -> %d", LFS_ERR_NOSPC);
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
return LFS_ERR_NOSPC;
}
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
int err = lfs_commitattr(lfs, path, type, buffer, size);
LFS_TRACE("lfs_setattr -> %d", err);
return err;
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
}
int lfs_removeattr(lfs_t *lfs, const char *path, uint8_t type) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_removeattr(%p, \"%s\", %"PRIu8")", (void*)lfs, path, type);
int err = lfs_commitattr(lfs, path, type, NULL, 0x3ff);
LFS_TRACE("lfs_removeattr -> %d", err);
return err;
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
/// Filesystem operations ///
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
lfs->cfg = cfg;
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
int err = 0;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs: Validate lfs-cfg sizes before performing arithmetic logics with them
2019-08-31 13:57:56 +00:00
// validate that the lfs-cfg sizes were initiated properly before
// performing any arithmetic logics with them
LFS_ASSERT(lfs->cfg->read_size != 0);
LFS_ASSERT(lfs->cfg->prog_size != 0);
LFS_ASSERT(lfs->cfg->cache_size != 0);
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
// check that block size is a multiple of cache size is a multiple
// of prog and read sizes
LFS_ASSERT(lfs->cfg->cache_size % lfs->cfg->read_size == 0);
LFS_ASSERT(lfs->cfg->cache_size % lfs->cfg->prog_size == 0);
LFS_ASSERT(lfs->cfg->block_size % lfs->cfg->cache_size == 0);
// check that the block size is large enough to fit ctz pointers
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
LFS_ASSERT(4*lfs_npw2(0xffffffff / (lfs->cfg->block_size-2*4))
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
<= lfs->cfg->block_size);
Minor improvement to documentation over block_cycles Suggested by haneefmubarak
2019-07-29 01:42:13 +00:00
// block_cycles = 0 is no longer supported.
//
// block_cycles is the number of erase cycles before littlefs evicts
// metadata logs as a part of wear leveling. Suggested values are in the
// range of 100-1000, or set block_cycles to -1 to disable block-level
// wear-leveling.
Changed block_cycles disable from 0 to -1 As it is now, block_cycles = 0 disables wear leveling. This was a mistake as 0 is the "default" value for several other config options. It's even worse when migrating from v1 as it's easy to miss the addition of block_cycles and end up with a filesystem that is not actually wear-leveling. Clearly, block_cycles = 0 should do anything but disable wear-leveling. Here, I've changed block_cycles = 0 to assert. Forcing users to set a value for block_cycles (500 is suggested). block_cycles can be set to -1 to explicitly disable wear leveling if desired.
2019-07-17 22:05:20 +00:00
LFS_ASSERT(lfs->cfg->block_cycles != 0);
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup read cache
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->read_buffer) {
lfs->rcache.buffer = lfs->cfg->read_buffer;
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->rcache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->rcache.buffer) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = LFS_ERR_NOMEM;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
goto cleanup;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup program cache
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->prog_buffer) {
lfs->pcache.buffer = lfs->cfg->prog_buffer;
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->pcache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->pcache.buffer) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = LFS_ERR_NOMEM;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
goto cleanup;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Fixed information leaks through reused caches As a shortcut, littlefs never bother to zero any of the buffers is used. It didn't need to because it would always write out the entirety of the data it needed. Unfortunately, this, combined with the extra padding used to align buffers to the nearest prog size, would lead to uninitialized data getting written out to disk. This means unrelated file data could be written to different parts of storage, or worse, information leaked from the malloc calls could be written out to disk unnecessarily. found by rojer
2018-07-06 16:14:30 +00:00
// zero to avoid information leaks
lfs_cache_zero(lfs, &lfs->rcache);
lfs_cache_zero(lfs, &lfs->pcache);
lfs: correct alignment restriction on lookahead buffer The buffer need only be 32-bit aligned. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-07-31 01:01:51 +00:00
// setup lookahead, must be multiple of 64-bits, 32-bit aligned
Changed lookahead configuration unit to bytes instead of bits The fact that the lookahead buffer uses bits instead of bytes is an internal detail. Poking this through to the user API has caused a decent amount of confusion. Most buffers are provided as bytes and the inconsistency here can be surprising. The use of bytes instead of bits also makes us forward compatible in the case that we want to change the lookahead internal representation (hint segment list). Additionally, we change the configuration name to lookahead_size. This matches other configurations, such as cache_size and read_size, while also notifying the user that something important changed at compile time (by breaking).
2018-10-03 01:18:30 +00:00
LFS_ASSERT(lfs->cfg->lookahead_size > 0);
Added clarification on buffer alignment. In v2, the lookahead_buffer was changed from requiring 4-byte alignment to requiring 8-byte alignment. This was not documented as well as it could be, and as FabianInostroza noted, this also implies that lfs_malloc must provide 8-byte alignment. To protect against this, I've also added an assert on the alignment of both the lookahead_size and lookahead_buffer. found by FabianInostroza and amitv87
2019-04-09 23:07:44 +00:00
LFS_ASSERT(lfs->cfg->lookahead_size % 8 == 0 &&
lfs: correct alignment restriction on lookahead buffer The buffer need only be 32-bit aligned. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-07-31 01:01:51 +00:00
(uintptr_t)lfs->cfg->lookahead_buffer % 4 == 0);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
if (lfs->cfg->lookahead_buffer) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer = lfs->cfg->lookahead_buffer;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
} else {
Changed lookahead configuration unit to bytes instead of bits The fact that the lookahead buffer uses bits instead of bytes is an internal detail. Poking this through to the user API has caused a decent amount of confusion. Most buffers are provided as bytes and the inconsistency here can be surprising. The use of bytes instead of bits also makes us forward compatible in the case that we want to change the lookahead internal representation (hint segment list). Additionally, we change the configuration name to lookahead_size. This matches other configurations, such as cache_size and read_size, while also notifying the user that something important changed at compile time (by breaking).
2018-10-03 01:18:30 +00:00
lfs->free.buffer = lfs_malloc(lfs->cfg->lookahead_size);
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!lfs->free.buffer) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = LFS_ERR_NOMEM;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
goto cleanup;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that the size limits are sane
Changed attr_max to be specific to custom attributes While technically, both system and user attributes share the same disk limitations, that's not what attr_max represents when considered from the user's perspective. To the user, attr_max applies only to custom attributes. This means attr_max should not impact other configurable limitations, such as inline files, and the ordering should be reconsidered with what the user finds most important.
2018-10-02 23:28:37 +00:00
LFS_ASSERT(lfs->cfg->name_max <= LFS_NAME_MAX);
lfs->name_max = lfs->cfg->name_max;
if (!lfs->name_max) {
lfs->name_max = LFS_NAME_MAX;
}
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
LFS_ASSERT(lfs->cfg->file_max <= LFS_FILE_MAX);
lfs->file_max = lfs->cfg->file_max;
if (!lfs->file_max) {
lfs->file_max = LFS_FILE_MAX;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Changed name of upper-limits from blah_size to blah_max This standardizes the naming between the LFS_BLAH_MAX macros and the blah_max configuration in the lfs_config structure.
2018-08-05 01:10:08 +00:00
LFS_ASSERT(lfs->cfg->attr_max <= LFS_ATTR_MAX);
lfs->attr_max = lfs->cfg->attr_max;
if (!lfs->attr_max) {
lfs->attr_max = LFS_ATTR_MAX;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// setup default state
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
lfs->root[0] = LFS_BLOCK_NULL;
lfs->root[1] = LFS_BLOCK_NULL;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
lfs->mlist = NULL;
Added allocation randomization for dynamic wear-leveling This implements the second step of full dynamic wear-leveling, block allocation randomization. This is the key part the uniformly distributes wear across the filesystem, even through reboots. The entropy actually comes from the filesystem itself, by xoring together all of the CRCs in the metadata-pairs on the filesystem. While this sounds like a ridiculous operation, it's easy to do when we already scan the metadata-pairs at mount time. This gives us a random number we can use for block allocation. Unfortunately it's not a great general purpose random generator as the output only changes every filesystem write. Fortunately that's exactly when we need our allocator. --- Additionally, the randomization created a mess for the testing framework. Fortunately, this method of randomization is deterministic. A very useful property for reproducing bugs.
2018-08-09 14:06:17 +00:00
lfs->seed = 0;
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->gdisk = (lfs_gstate_t){0};
lfs->gstate = (lfs_gstate_t){0};
lfs->gdelta = (lfs_gstate_t){0};
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
#ifdef LFS_MIGRATE
lfs->lfs1 = NULL;
#endif
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
cleanup:
lfs_deinit(lfs);
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
return err;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
static int lfs_deinit(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// free allocated memory
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->cfg->read_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->rcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
if (!lfs->cfg->prog_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->pcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
if (!lfs->cfg->lookahead_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->free.buffer);
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
int lfs_format(lfs_t *lfs, const struct lfs_config *cfg) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_format(%p, %p {.context=%p, "
".read=%p, .prog=%p, .erase=%p, .sync=%p, "
".read_size=%"PRIu32", .prog_size=%"PRIu32", "
".block_size=%"PRIu32", .block_count=%"PRIu32", "
".block_cycles=%"PRIu32", .cache_size=%"PRIu32", "
".lookahead_size=%"PRIu32", .read_buffer=%p, "
".prog_buffer=%p, .lookahead_buffer=%p, "
".name_max=%"PRIu32", .file_max=%"PRIu32", "
".attr_max=%"PRIu32"})",
(void*)lfs, (void*)cfg, cfg->context,
(void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog,
(void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync,
cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count,
cfg->block_cycles, cfg->cache_size, cfg->lookahead_size,
cfg->read_buffer, cfg->prog_buffer, cfg->lookahead_buffer,
cfg->name_max, cfg->file_max, cfg->attr_max);
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
int err = 0;
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
err = lfs_init(lfs, cfg);
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_format -> %d", err);
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
return err;
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
// create free lookahead
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead_size);
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
lfs->free.off = 0;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
lfs->free.size = lfs_min(8*lfs->cfg->lookahead_size,
lfs->cfg->block_count);
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
lfs->free.i = 0;
lfs_alloc_ack(lfs);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
// create root dir
lfs_mdir_t root;
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
err = lfs_dir_alloc(lfs, &root);
if (err) {
goto cleanup;
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
// write one superblock
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
lfs_superblock_t superblock = {
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
.version = LFS_DISK_VERSION,
.block_size = lfs->cfg->block_size,
.block_count = lfs->cfg->block_count,
.name_max = lfs->name_max,
.file_max = lfs->file_max,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
.attr_max = lfs->attr_max,
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
};
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
lfs_superblock_tole32(&superblock);
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
err = lfs_dir_commit(lfs, &root, LFS_MKATTRS(
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
{LFS_MKTAG(LFS_TYPE_CREATE, 0, 0), NULL},
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
{LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, 8), "littlefs"},
{LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)),
&superblock}));
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (err) {
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
goto cleanup;
}
// sanity check that fetch works
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_dir_fetch(lfs, &root, (const lfs_block_t[2]){0, 1});
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
if (err) {
goto cleanup;
}
Added redundant compaction to lfs_format/lfs_migrate This ensures that both blocks in the superblock pair are written with the superblock info. While this does use an additional erase cycle, it prevents older versions of littlefs from accidentally being picked up in the case that the disk is mounted on a system that doesn't support the newer version. This does bring back the risk of picking up old littlefs versions on a disk that has been formatted with a filesystem that doesn't use block 2 (such as FAT), but this risk already exists, and moving between versions of littlefs is more likely with the recent v1 -> v2 update. Suggested by rojer
2019-05-28 23:16:51 +00:00
// force compaction to prevent accidentally mounting any
// older version of littlefs that may live on disk
root.erased = false;
err = lfs_dir_commit(lfs, &root, NULL, 0);
if (err) {
goto cleanup;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
cleanup:
lfs_deinit(lfs);
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_format -> %d", err);
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mount(%p, %p {.context=%p, "
".read=%p, .prog=%p, .erase=%p, .sync=%p, "
".read_size=%"PRIu32", .prog_size=%"PRIu32", "
".block_size=%"PRIu32", .block_count=%"PRIu32", "
".block_cycles=%"PRIu32", .cache_size=%"PRIu32", "
".lookahead_size=%"PRIu32", .read_buffer=%p, "
".prog_buffer=%p, .lookahead_buffer=%p, "
".name_max=%"PRIu32", .file_max=%"PRIu32", "
".attr_max=%"PRIu32"})",
(void*)lfs, (void*)cfg, cfg->context,
(void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog,
(void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync,
cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count,
cfg->block_cycles, cfg->cache_size, cfg->lookahead_size,
cfg->read_buffer, cfg->prog_buffer, cfg->lookahead_buffer,
cfg->name_max, cfg->file_max, cfg->attr_max);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
int err = lfs_init(lfs, cfg);
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mount -> %d", err);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return err;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// scan directory blocks for superblock and any global updates
lfs_mdir_t dir = {.tail = {0, 1}};
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
lfs_block_t cycle = 0;
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
while (!lfs_pair_isnull(dir.tail)) {
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
if (cycle >= lfs->cfg->block_count/2) {
// loop detected
err = LFS_ERR_CORRUPT;
goto cleanup;
}
cycle += 1;
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// fetch next block in tail list
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t tag = lfs_dir_fetchmatch(lfs, &dir, dir.tail,
LFS_MKTAG(0x7ff, 0x3ff, 0),
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, 8),
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
NULL,
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
lfs_dir_find_match, &(struct lfs_dir_find_match){
lfs, "littlefs", 8});
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
if (tag < 0) {
err = tag;
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
goto cleanup;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// has superblock?
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
if (tag && !lfs_tag_isdelete(tag)) {
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// update root
lfs->root[0] = dir.pair[0];
lfs->root[1] = dir.pair[1];
// grab superblock
lfs_superblock_t superblock;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
tag = lfs_dir_get(lfs, &dir, LFS_MKTAG(0x7ff, 0x3ff, 0),
Fixed issue with creating files named "littlefs" A rather humorous issue, we accidentally ended up mixing our file namespace with our superblocks. This meant if we created a file named "littlefs" it would reference the superblock and all sorts of things would break. Fixing this also highlighted another issue, the fact that the superblock always needs to come before any file entries in the directory. I didn't account for this in the initial B-tree design, but we need a higher ordering for superblocks + children + files than just name. To fix this I added ordering information in the 2 bits currently unused in the tag type. Though note that the size of these fields are flexible. 9-bit type field: [--- 9 ---] [1|- 3 -|- 2 -|- 3 -] ^ ^ ^ ^- type-specific info | | \------- ordering info | \------------- subtype \----------------- user bit
2018-10-04 21:25:19 +00:00
LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)),
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
&superblock);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
if (tag < 0) {
err = tag;
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
goto cleanup;
}
lfs_superblock_fromle32(&superblock);
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// check version
uint16_t major_version = (0xffff & (superblock.version >> 16));
uint16_t minor_version = (0xffff & (superblock.version >> 0));
if ((major_version != LFS_DISK_VERSION_MAJOR ||
minor_version > LFS_DISK_VERSION_MINOR)) {
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_ERROR("Invalid version v%"PRIu16".%"PRIu16,
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
major_version, minor_version);
err = LFS_ERR_INVAL;
goto cleanup;
}
Added root entry and expanding superblocks Expanding superblocks has been on my wishlist for a while. The basic idea is that instead of maintaining a fixed offset blocks {0, 1} to the the root directory (1 pointer), we maintain a dynamically sized linked-list of superblocks that point to the actual root. If the number of writes to the root exceeds some value, we increase the size of the superblock linked-list. This can leverage existing metadata-pair operations. The revision count for metadata-pairs provides some knowledge on how much wear we've put on the superblock, and the threaded linked-list can also be reused for this purpose. This means superblock expansion is both optional and cheap to implement. Expanding superblocks helps both extremely small and extremely large filesystem (extreme being relative of course). On the small end, we can actually collapse the superblock into the root directory and drop the hard requirement of 4-blocks for the superblock. On the large end, our superblock will now last longer than the rest of the filesystem. Each time we expand, the number of cycles until the superblock dies is increased by a power. Before we were stuck with this layout: level cycles limit layout 1 E^2 390 MiB s0 -> root Now we expand every time a fixed offset is exceeded: level cycles limit layout 0 E 4 KiB s0+root 1 E^2 390 MiB s0 -> root 2 E^3 37 TiB s0 -> s1 -> root 3 E^4 3.6 EiB s0 -> s1 -> s2 -> root ... Where the cycles are the number of cycles before death, and the limit is the worst-case size a filesystem where early superblock death becomes a concern (all writes to root using this formula: E^|s| = E*B, E = erase cycles = 100000, B = block count, assuming 4096 byte blocks). Note we can also store copies of the superblock entry on the expanded superblocks. This may help filesystem recover tools in the future.
2018-08-06 18:30:51 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// check superblock configuration
if (superblock.name_max) {
if (superblock.name_max > lfs->name_max) {
LFS_ERROR("Unsupported name_max (%"PRIu32" > %"PRIu32")",
superblock.name_max, lfs->name_max);
err = LFS_ERR_INVAL;
goto cleanup;
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
lfs->name_max = superblock.name_max;
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
if (superblock.file_max) {
if (superblock.file_max > lfs->file_max) {
LFS_ERROR("Unsupported file_max (%"PRIu32" > %"PRIu32")",
superblock.file_max, lfs->file_max);
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
err = LFS_ERR_INVAL;
goto cleanup;
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
lfs->file_max = superblock.file_max;
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
}
Changed attr_max to be specific to custom attributes While technically, both system and user attributes share the same disk limitations, that's not what attr_max represents when considered from the user's perspective. To the user, attr_max applies only to custom attributes. This means attr_max should not impact other configurable limitations, such as inline files, and the ordering should be reconsidered with what the user finds most important.
2018-10-02 23:28:37 +00:00
if (superblock.attr_max) {
if (superblock.attr_max > lfs->attr_max) {
LFS_ERROR("Unsupported attr_max (%"PRIu32" > %"PRIu32")",
superblock.attr_max, lfs->attr_max);
err = LFS_ERR_INVAL;
goto cleanup;
}
lfs->attr_max = superblock.attr_max;
}
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// has gstate?
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
err = lfs_dir_getgstate(lfs, &dir, &lfs->gstate);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
goto cleanup;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
}
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
}
Fixed issue with creating files named "littlefs" A rather humorous issue, we accidentally ended up mixing our file namespace with our superblocks. This meant if we created a file named "littlefs" it would reference the superblock and all sorts of things would break. Fixing this also highlighted another issue, the fact that the superblock always needs to come before any file entries in the directory. I didn't account for this in the initial B-tree design, but we need a higher ordering for superblocks + children + files than just name. To fix this I added ordering information in the 2 bits currently unused in the tag type. Though note that the size of these fields are flexible. 9-bit type field: [--- 9 ---] [1|- 3 -|- 2 -|- 3 -] ^ ^ ^ ^- type-specific info | | \------- ordering info | \------------- subtype \----------------- user bit
2018-10-04 21:25:19 +00:00
// found superblock?
if (lfs_pair_isnull(lfs->root)) {
err = LFS_ERR_INVAL;
goto cleanup;
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// update littlefs with gstate
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (!lfs_gstate_iszero(&lfs->gstate)) {
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Found pending gstate 0x%08"PRIx32"%08"PRIx32"%08"PRIx32,
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->gstate.tag,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs->gstate.pair[0],
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->gstate.pair[1]);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->gstate.tag += !lfs_tag_isvalid(lfs->gstate.tag);
lfs->gdisk = lfs->gstate;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Removed unnecessary randomization of offsets in lfs_alloc_reset On first read, randomizing the allocators offset may seem appropriate for lfs_alloc_reset. However, it ends up using the filesystem-fed pseudorandom seed in situations it wasn't designed for. As noted by gtaska, the combination of using xors for feeding the seed and multiple traverses of the same CRCs can cause the seed to flip to zeros with concerning frequency. Removed the randomization from lfs_alloc_reset, leaving it in only lfs_mount. Found by gtaska
2020-11-20 06:18:13 +00:00
// setup free lookahead, to distribute allocations uniformly across
// boots, we start the allocator at a random location
lfs->free.off = lfs->seed % lfs->cfg->block_count;
lfs_alloc_drop(lfs);
Added allocation randomization for dynamic wear-leveling This implements the second step of full dynamic wear-leveling, block allocation randomization. This is the key part the uniformly distributes wear across the filesystem, even through reboots. The entropy actually comes from the filesystem itself, by xoring together all of the CRCs in the metadata-pairs on the filesystem. While this sounds like a ridiculous operation, it's easy to do when we already scan the metadata-pairs at mount time. This gives us a random number we can use for block allocation. Unfortunately it's not a great general purpose random generator as the output only changes every filesystem write. Fortunately that's exactly when we need our allocator. --- Additionally, the randomization created a mess for the testing framework. Fortunately, this method of randomization is deterministic. A very useful property for reproducing bugs.
2018-08-09 14:06:17 +00:00
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mount -> %d", 0);
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return 0;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
cleanup:
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_unmount(lfs);
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_mount -> %d", err);
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
return err;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
int lfs_unmount(lfs_t *lfs) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_unmount(%p)", (void*)lfs);
int err = lfs_deinit(lfs);
LFS_TRACE("lfs_unmount -> %d", err);
return err;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
/// Filesystem filesystem operations ///
Fixed lfs_fs_size doubling metadata-pairs This was caused by the previous fix for allocations during lfs_fs_deorphan in this branch. To catch half-orphans during block allocations we needed to duplicate all metadata-pairs reported to lfs_fs_traverse. Unfortunately this causes lfs_fs_size to report 2x the number of metadata-pairs, which would undoubtably confuse users. The fix here is inelegantly simple, just do a different traversale for allocations and size measurements. It reuses the same code but touches slightly different sets of blocks. Unfortunately, this causes the public lfs_fs_traverse and lfs_fs_size functions to split in how they report blocks. This is technically allowed, since lfs_fs_traverse may report blocks multiple times due to CoW behavior, however it's undesirable and I'm sure there will be some confusion. But I don't have a better solution, so from this point lfs_fs_traverse will be reporting 2x metadata-blocks and shouldn't be used for finding the number of available blocks on the filesystem.
2020-02-09 15:05:37 +00:00
int lfs_fs_traverseraw(lfs_t *lfs,
int (*cb)(void *data, lfs_block_t block), void *data,
bool includeorphans) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// iterate over metadata pairs
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir = {.tail = {0, 1}};
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
#ifdef LFS_MIGRATE
// also consider v1 blocks during migration
if (lfs->lfs1) {
int err = lfs1_traverse(lfs, cb, data);
if (err) {
return err;
}
dir.tail[0] = lfs->root[0];
dir.tail[1] = lfs->root[1];
}
#endif
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
lfs_block_t cycle = 0;
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
while (!lfs_pair_isnull(dir.tail)) {
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
if (cycle >= lfs->cfg->block_count/2) {
// loop detected
return LFS_ERR_CORRUPT;
}
cycle += 1;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
for (int i = 0; i < 2; i++) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int err = cb(data, dir.tail[i]);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
// iterate through ids in directory
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
for (uint16_t id = 0; id < dir.count; id++) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
struct lfs_ctz ctz;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t tag = lfs_dir_get(lfs, &dir, LFS_MKTAG(0x700, 0x3ff, 0),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, id, sizeof(ctz)), &ctz);
if (tag < 0) {
if (tag == LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
continue;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return tag;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_ctz_fromle32(&ctz);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_CTZSTRUCT) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
err = lfs_ctz_traverse(lfs, NULL, &lfs->rcache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
ctz.head, ctz.size, cb, data);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Fixed lfs_fs_size doubling metadata-pairs This was caused by the previous fix for allocations during lfs_fs_deorphan in this branch. To catch half-orphans during block allocations we needed to duplicate all metadata-pairs reported to lfs_fs_traverse. Unfortunately this causes lfs_fs_size to report 2x the number of metadata-pairs, which would undoubtably confuse users. The fix here is inelegantly simple, just do a different traversale for allocations and size measurements. It reuses the same code but touches slightly different sets of blocks. Unfortunately, this causes the public lfs_fs_traverse and lfs_fs_size functions to split in how they report blocks. This is technically allowed, since lfs_fs_traverse may report blocks multiple times due to CoW behavior, however it's undesirable and I'm sure there will be some confusion. But I don't have a better solution, so from this point lfs_fs_traverse will be reporting 2x metadata-blocks and shouldn't be used for finding the number of available blocks on the filesystem.
2020-02-09 15:05:37 +00:00
} else if (includeorphans &&
Fixed more bugs, mostly related to ENOSPC on different geometries Fixes: - Fixed reproducability issue when we can't read a directory revision - Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size - Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync - Fixed cleanup issue if we run out of space while extending a CTZ skip-list - Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan Also: - Added cycle-detection to readtree.py - Allowed pseudo-C expressions in test conditions (and it's beautifully hacky, see line 187 of test.py) - Better handling of ctrl-C during test runs - Added build-only mode to test.py - Limited stdout of test failures to 5 lines unless in verbose mode Explanation of fixes below 1. Fixed reproducability issue when we can't read a directory revision An interesting subtlety of the block-device layer is that the block-device is allowed to return LFS_ERR_CORRUPT on reads to untouched blocks. This can easily happen if a user is using ECC or some sort of CMAC on their blocks. Normally we never run into this, except for the optimization around directory revisions where we use uninitialized data to start our revision count. We correctly handle this case by ignoring whats on disk if the read fails, but end up using unitialized RAM instead. This is not an issue for normal use, though it can lead to a small information leak. However it creates a big problem for reproducability, which is very helpful for debugging. I ended up running into a case where the RAM values for the revision count was different, causing two identical runs to wear-level at different times, leading to one version running out of space before a bug occured because it expanded the superblock early. 2. Fixed incorrect erase assumption if lfs_dir_fetch exceeds block size This could be caused if the previous tag was a valid commit and we lost power causing a partially written tag as the start of a new commit. Fortunately we already have a separate condition for exceeding the block size, so we can force that case to always treat the mdir as unerased. 3. Fixed cleanup issue caused by lfs_fs_relocate failing when trying to outline a file in lfs_file_sync Most operations involving metadata-pairs treat the mdir struct as entirely temporary and throw it out if any error occurs. Except for lfs_file_sync since the mdir is also a part of the file struct. This is relevant because of a cleanup issue in lfs_dir_compact that usually doesn't have side-effects. The issue is that lfs_fs_relocate can fail. It needs to allocate new blocks to relocate to, and as the disk reaches its end of life, it can fail with ENOSPC quite often. If lfs_fs_relocate fails, the containing lfs_dir_compact would return immediately without restoring the previous state of the mdir. If a new commit comes in on the same mdir, the old state left there could corrupt the filesystem. It's interesting to note this is forced to happen in lfs_file_sync, since it always tries to outline the file if it gets ENOSPC (ENOSPC can mean both no blocks to allocate and that the mdir is full). I'm not actually sure this bit of code is necessary anymore, we may be able to remove it. 4. Fixed cleanup issue if we run out of space while extending a CTZ skip-list The actually CTZ skip-list logic itself hasn't been touched in more than a year at this point, so I was surprised to find a bug here. But it turns out the CTZ skip-list could be put in an invalid state if we run out of space while trying to extend the skip-list. This only becomes a problem if we keep the file open, clean up some space elsewhere, and then continue to write to the open file without modifying it. Fortunately an easy fix. 5. Fixed missing half-orphans when allocating blocks during lfs_fs_deorphan This was a really interesting bug. Normally, we don't have to worry about allocations, since we force consistency before we are allowed to allocate blocks. But what about the deorphan operation itself? Don't we need to allocate blocks if we relocate while deorphaning? It turns out the deorphan operation can lead to allocating blocks while there's still orphans and half-orphans on the threaded linked-list. Orphans aren't an issue, but half-orphans may contain references to blocks in the outdated half, which doesn't get scanned during the normal allocation pass. Fortunately we already fetch directory entries to check CTZ lists, so we can also check half-orphans here. However this causes lfs_fs_traverse to duplicate all metadata-pairs, not sure what to do about this yet.
2020-01-29 07:45:19 +00:00
lfs_tag_type3(tag) == LFS_TYPE_DIRSTRUCT) {
for (int i = 0; i < 2; i++) {
err = cb(data, (&ctz.head)[i]);
if (err) {
return err;
}
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
}
// iterate over any open files
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
for (lfs_file_t *f = (lfs_file_t*)lfs->mlist; f; f = f->next) {
if (f->type != LFS_TYPE_REG) {
continue;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if ((f->flags & LFS_F_DIRTY) && !(f->flags & LFS_F_INLINE)) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_traverse(lfs, &f->cache, &lfs->rcache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
f->ctz.head, f->ctz.size, cb, data);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
if ((f->flags & LFS_F_WRITING) && !(f->flags & LFS_F_INLINE)) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_traverse(lfs, &f->cache, &lfs->rcache,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
f->block, f->pos, cb, data);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
return 0;
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Fixed lfs_fs_size doubling metadata-pairs This was caused by the previous fix for allocations during lfs_fs_deorphan in this branch. To catch half-orphans during block allocations we needed to duplicate all metadata-pairs reported to lfs_fs_traverse. Unfortunately this causes lfs_fs_size to report 2x the number of metadata-pairs, which would undoubtably confuse users. The fix here is inelegantly simple, just do a different traversale for allocations and size measurements. It reuses the same code but touches slightly different sets of blocks. Unfortunately, this causes the public lfs_fs_traverse and lfs_fs_size functions to split in how they report blocks. This is technically allowed, since lfs_fs_traverse may report blocks multiple times due to CoW behavior, however it's undesirable and I'm sure there will be some confusion. But I don't have a better solution, so from this point lfs_fs_traverse will be reporting 2x metadata-blocks and shouldn't be used for finding the number of available blocks on the filesystem.
2020-02-09 15:05:37 +00:00
int lfs_fs_traverse(lfs_t *lfs,
int (*cb)(void *data, lfs_block_t block), void *data) {
LFS_TRACE("lfs_fs_traverse(%p, %p, %p)",
(void*)lfs, (void*)(uintptr_t)cb, data);
int err = lfs_fs_traverseraw(lfs, cb, data, true);
LFS_TRACE("lfs_fs_traverse -> %d", 0);
return err;
}
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_pred(lfs_t *lfs,
const lfs_block_t pair[2], lfs_mdir_t *pdir) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// iterate over all directory directory entries
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
pdir->tail[0] = 0;
pdir->tail[1] = 1;
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
lfs_block_t cycle = 0;
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
while (!lfs_pair_isnull(pdir->tail)) {
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
if (cycle >= lfs->cfg->block_count/2) {
// loop detected
return LFS_ERR_CORRUPT;
}
cycle += 1;
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
if (lfs_pair_cmp(pdir->tail, pair) == 0) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return 0;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, pdir, pdir->tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return LFS_ERR_NOENT;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
struct lfs_fs_parent_match {
lfs_t *lfs;
const lfs_block_t pair[2];
};
static int lfs_fs_parent_match(void *data,
lfs_tag_t tag, const void *buffer) {
struct lfs_fs_parent_match *find = data;
lfs_t *lfs = find->lfs;
const struct lfs_diskoff *disk = buffer;
(void)tag;
lfs_block_t child[2];
int err = lfs_bd_read(lfs,
&lfs->pcache, &lfs->rcache, lfs->cfg->block_size,
disk->block, disk->off, &child, sizeof(child));
if (err) {
return err;
}
lfs_pair_fromle32(child);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return (lfs_pair_cmp(child, find->pair) == 0) ? LFS_CMP_EQ : LFS_CMP_LT;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t pair[2],
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
lfs_mdir_t *parent) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
// use fetchmatch with callback to find pairs
parent->tail[0] = 0;
parent->tail[1] = 1;
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
lfs_block_t cycle = 0;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
while (!lfs_pair_isnull(parent->tail)) {
Added "evil" tests and detecion/recovery from bad pointers and infinite loops These two features have been much requested by users, and have even had several PRs proposed to fix these in several cases. Before this, these error conditions usually were caught by internal asserts, however asserts prevented users from implementing their own workarounds. It's taken me a while to provide/accept a useful recovery mechanism (returning LFS_ERR_CORRUPT instead of asserting) because my original thinking was that these error conditions only occur due to bugs in the filesystem, and these bugs should be fixed properly. While I still think this is mostly true, the point has been made clear that being able to recover from these conditions is definitely worth the code cost. Hopefully this new behaviour helps the longevity of devices even if the storage code fails. Another, less important, reason I didn't want to accept fixes for these situations was the lack of tests that prove the code's value. This has been fixed with the new testing framework thanks to the additional of "internal tests" which can call C static functions and really take advantage of the internal information of the filesystem.
2020-02-24 04:55:52 +00:00
if (cycle >= lfs->cfg->block_count/2) {
// loop detected
return LFS_ERR_CORRUPT;
}
cycle += 1;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_fetchmatch(lfs, parent, parent->tail,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x7ff, 0, 0x3ff),
LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 0, 8),
NULL,
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_fs_parent_match, &(struct lfs_fs_parent_match){
lfs, {pair[0], pair[1]}});
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (tag && tag != LFS_ERR_NOENT) {
Added root entry and expanding superblocks Expanding superblocks has been on my wishlist for a while. The basic idea is that instead of maintaining a fixed offset blocks {0, 1} to the the root directory (1 pointer), we maintain a dynamically sized linked-list of superblocks that point to the actual root. If the number of writes to the root exceeds some value, we increase the size of the superblock linked-list. This can leverage existing metadata-pair operations. The revision count for metadata-pairs provides some knowledge on how much wear we've put on the superblock, and the threaded linked-list can also be reused for this purpose. This means superblock expansion is both optional and cheap to implement. Expanding superblocks helps both extremely small and extremely large filesystem (extreme being relative of course). On the small end, we can actually collapse the superblock into the root directory and drop the hard requirement of 4-blocks for the superblock. On the large end, our superblock will now last longer than the rest of the filesystem. Each time we expand, the number of cycles until the superblock dies is increased by a power. Before we were stuck with this layout: level cycles limit layout 1 E^2 390 MiB s0 -> root Now we expand every time a fixed offset is exceeded: level cycles limit layout 0 E 4 KiB s0+root 1 E^2 390 MiB s0 -> root 2 E^3 37 TiB s0 -> s1 -> root 3 E^4 3.6 EiB s0 -> s1 -> s2 -> root ... Where the cycles are the number of cycles before death, and the limit is the worst-case size a filesystem where early superblock death becomes a concern (all writes to root using this formula: E^|s| = E*B, E = erase cycles = 100000, B = block count, assuming 4096 byte blocks). Note we can also store copies of the superblock entry on the expanded superblocks. This may help filesystem recover tools in the future.
2018-08-06 18:30:51 +00:00
return tag;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Modified results from find-like functions to use tags Tags offer all of the necessary info from the find functions (which makes sense, this is the structure that stores the info on disk). Passing around a single tag instead of separate id and type fields simplifies the internal functions while leverages the tag's compactness.
2018-07-11 12:18:30 +00:00
return LFS_ERR_NOENT;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_relocate(lfs_t *lfs,
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
const lfs_block_t oldpair[2], lfs_block_t newpair[2]) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// update internal root
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
if (lfs_pair_cmp(oldpair, lfs->root) == 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
lfs->root[0] = newpair[0];
lfs->root[1] = newpair[1];
}
// update internally tracked dirs
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
if (lfs_pair_cmp(oldpair, d->m.pair) == 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
d->m.pair[0] = newpair[0];
d->m.pair[1] = newpair[1];
}
Fixed issue with directories falling out of date after block relocation This is caused by dir->head not being updated when dir->m.pair may be. This causes the two to fall out of sync and later dir rewinds to fail. This bug stems all the way back from the first commits of littlefs, so it's surprising it has avoided detection for this long. Perhaps because lfs_dir_rewind is not used often.
2019-11-14 20:25:42 +00:00
if (d->type == LFS_TYPE_DIR &&
lfs_pair_cmp(oldpair, ((lfs_dir_t*)d)->head) == 0) {
((lfs_dir_t*)d)->head[0] = newpair[0];
((lfs_dir_t*)d)->head[1] = newpair[1];
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// find parent
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t parent;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_stag_t tag = lfs_fs_parent(lfs, oldpair, &parent);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag < 0 && tag != LFS_ERR_NOENT) {
return tag;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag != LFS_ERR_NOENT) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// update disk, this creates a desync
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, +1);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
// fix pending move in this pair? this looks like an optimization but
// is in fact _required_ since relocating may outdate the move.
uint16_t moveid = 0x3ff;
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (lfs_gstate_hasmovehere(&lfs->gstate, parent.pair)) {
moveid = lfs_tag_id(lfs->gstate.tag);
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
LFS_DEBUG("Fixing move while relocating "
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
"{0x%"PRIx32", 0x%"PRIx32"} 0x%"PRIx16"\n",
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
parent.pair[0], parent.pair[1], moveid);
lfs_fs_prepmove(lfs, 0x3ff, NULL);
if (moveid < lfs_tag_id(tag)) {
tag -= LFS_MKTAG(0, 1, 0);
}
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_tole32(newpair);
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
int err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS(
{LFS_MKTAG_IF(moveid != 0x3ff,
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
LFS_TYPE_DELETE, moveid, 0), NULL},
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
{tag, newpair}));
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_fromle32(newpair);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// next step, clean up orphans
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -1);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
// find pred
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_pred(lfs, oldpair, &parent);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// if we can't find dir, it must be new
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err != LFS_ERR_NOENT) {
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
// fix pending move in this pair? this looks like an optimization but
// is in fact _required_ since relocating may outdate the move.
uint16_t moveid = 0x3ff;
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (lfs_gstate_hasmovehere(&lfs->gstate, parent.pair)) {
moveid = lfs_tag_id(lfs->gstate.tag);
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
LFS_DEBUG("Fixing move while relocating "
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
"{0x%"PRIx32", 0x%"PRIx32"} 0x%"PRIx16"\n",
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
parent.pair[0], parent.pair[1], moveid);
lfs_fs_prepmove(lfs, 0x3ff, NULL);
}
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// replace bad pair, either we clean up desync, or no desync occured
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_tole32(newpair);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS(
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
{LFS_MKTAG_IF(moveid != 0x3ff,
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
LFS_TYPE_DELETE, moveid, 0), NULL},
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
{LFS_MKTAG(LFS_TYPE_TAIL + parent.split, 0x3ff, 8), newpair}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(newpair);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (err) {
return err;
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return 0;
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static void lfs_fs_preporphans(lfs_t *lfs, int8_t orphans) {
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
LFS_ASSERT(lfs_tag_size(lfs->gstate.tag) > 0 || orphans >= 0);
lfs->gstate.tag += orphans;
lfs->gstate.tag = ((lfs->gstate.tag & ~LFS_MKTAG(0x800, 0, 0)) |
((uint32_t)lfs_gstate_hasorphans(&lfs->gstate) << 31));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
static void lfs_fs_prepmove(lfs_t *lfs,
uint16_t id, const lfs_block_t pair[2]) {
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->gstate.tag = ((lfs->gstate.tag & ~LFS_MKTAG(0x7ff, 0x3ff, 0)) |
((id != 0x3ff) ? LFS_MKTAG(LFS_TYPE_DELETE, id, 0) : 0));
lfs->gstate.pair[0] = (id != 0x3ff) ? pair[0] : 0;
lfs->gstate.pair[1] = (id != 0x3ff) ? pair[1] : 0;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
static int lfs_fs_demove(lfs_t *lfs) {
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
if (!lfs_gstate_hasmove(&lfs->gdisk)) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
return 0;
}
// Fix bad moves
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Fixing move {0x%"PRIx32", 0x%"PRIx32"} 0x%"PRIx16,
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs->gdisk.pair[0],
lfs->gdisk.pair[1],
lfs_tag_id(lfs->gdisk.tag));
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// fetch and delete the moved entry
lfs_mdir_t movedir;
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
int err = lfs_dir_fetch(lfs, &movedir, lfs->gdisk.pair);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
return err;
}
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
// prep gstate and delete move id
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
uint16_t moveid = lfs_tag_id(lfs->gdisk.tag);
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
lfs_fs_prepmove(lfs, 0x3ff, NULL);
err = lfs_dir_commit(lfs, &movedir, LFS_MKATTRS(
Fix -Wmissing-field-initializers
2020-04-03 01:05:55 +00:00
{LFS_MKTAG(LFS_TYPE_DELETE, moveid, 0), NULL}));
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
return err;
}
return 0;
}
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
static int lfs_fs_deorphan(lfs_t *lfs) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (!lfs_gstate_hasorphans(&lfs->gstate)) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
return 0;
}
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// Fix any orphans
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
lfs_mdir_t pdir = {.split = true, .tail = {0, 1}};
lfs_mdir_t dir;
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// iterate over all directory directory entries
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
while (!lfs_pair_isnull(pdir.tail)) {
int err = lfs_dir_fetch(lfs, &dir, pdir.tail);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// check head blocks for orphans
if (!pdir.split) {
// check if we have a parent
lfs_mdir_t parent;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_stag_t tag = lfs_fs_parent(lfs, pdir.tail, &parent);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (tag < 0 && tag != LFS_ERR_NOENT) {
return tag;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (tag == LFS_ERR_NOENT) {
// we are an orphan
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Fixing orphan {0x%"PRIx32", 0x%"PRIx32"}",
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
pdir.tail[0], pdir.tail[1]);
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
err = lfs_dir_drop(lfs, &pdir, &dir);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// refetch tail
continue;
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
lfs_block_t pair[2];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &parent,
LFS_MKTAG(0x7ff, 0x3ff, 0), tag, pair);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (res < 0) {
return res;
}
lfs_pair_fromle32(pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (!lfs_pair_sync(pair, pdir.tail)) {
// we have desynced
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Fixing half-orphan {0x%"PRIx32", 0x%"PRIx32"} "
"-> {0x%"PRIx32", 0x%"PRIx32"}",
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
pdir.tail[0], pdir.tail[1], pair[0], pair[1]);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_tole32(pair);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &pdir, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), pair}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(pair);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
// refetch tail
continue;
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
}
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added tests for power-cycled-relocations and fixed the bugs that fell out The power-cycled-relocation test with random renames has been the most aggressive test applied to littlefs so far, with: - Random nested directory creation - Random nested directory removal - Random nested directory renames (this could make the threaded linked-list very interesting) - Relocating blocks every write (maximum wear-leveling) - Incrementally cycling power every write Also added a couple other tests to test_orphans and test_relocations. The good news is the added testing worked well, it found quite a number of complex and subtle bugs that have been difficult to find. 1. It's actually possible for our parent to be relocated and go out of sync in lfs_mkdir. This can happen if our predecessor's predecessor is our parent as we are threading ourselves into the filesystem's threaded list. (note this doesn't happen if our predecessor _is_ our parent, as we then update our parent in a single commit). This is annoying because it only happens if our parent is a long (>1 pair) directory, otherwise we wouldn't need to catch relocations. Fortunately we can reuse the internal open file/dir linked-list to catch relocations easily, as long as we're careful to unhook our parent whenever lfs_mkdir returns. 2. Even more surprising, it's possible for the child in lfs_remove to be relocated while we delete the entry from our parent. This can happen if we are our own parent's predecessor, since we need to be updated then if our parent relocates. Fortunately we can also hook into the open linked-list here. Note this same issue was present in lfs_rename. Fortunately, this means now all fetched dirs are hooked into the open linked-list if they are needed across a commit. This means we shouldn't need assumptions about tree movement for correctness. 3. lfs_rename("deja/vu", "deja/vu") with the same source and destination was broken and tried to delete the entry twice. 4. Managing gstate deltas when we lose power during relocations was broken. And unfortunately complicated. The issue happens when we lose power during a relocation while removing a directory. When we remove a directory, we need to move the contents of its gstate delta to another directory or we'll corrupt littlefs gstate. (gstate is an xor of all deltas on the filesystem). We used to just xor the gstate into our parent's gstate, however this isn't correct. The gstate isn't built out of the directory tree, but rather out of the threaded linked-list (which exists to make collecting this gstate efficient). Because we have to remove our dir in two operations, there's a point were both the updated parent and child can exist in threaded linked-list and duplicate the child's gstate delta. .--------. ->| parent |-. | gstate | | .-| a |-' | '--------' | X <- child is orphaned | .--------. '>| child |-> | gstate | | a | '--------' What we need to do is save our child's gstate and only give it to our predecessor, since this finalizes the removal of the child. However we still need to make valid updates to the gstate to mark that we've created an orphan when we start removing the child. This led to a small rework of how the gstate is handled. Now we have a separation of the gpending state that should be written out ASAP and the gdelta state that is collected from orphans awaiting deletion. 5. lfs_deorphan wasn't actually able to handle deorphaning/desyncing more than one orphan after a power-cycle. Having more than one orphan is very rare, but of course very possible. Fortunately this was just a mistake with using a break the in the deorphan, perhaps left from v1 where multiple orphans weren't possible? Note that we use a continue to force a refetch of the orphaned block. This is needed in the case of a half-orphan, since the fetched half-orphan may have an outdated tail pointer.
2020-01-22 04:18:19 +00:00
pdir = dir;
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// mark orphans as fixed
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -lfs_gstate_getorphans(&lfs->gstate));
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
return 0;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
static int lfs_fs_forceconsistency(lfs_t *lfs) {
int err = lfs_fs_demove(lfs);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
err = lfs_fs_deorphan(lfs);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
return 0;
}
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
static int lfs_fs_size_count(void *p, lfs_block_t block) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
(void)block;
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
lfs_size_t *size = p;
*size += 1;
return 0;
}
lfs_ssize_t lfs_fs_size(lfs_t *lfs) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_fs_size(%p)", (void*)lfs);
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
lfs_size_t size = 0;
Fixed lfs_fs_size doubling metadata-pairs This was caused by the previous fix for allocations during lfs_fs_deorphan in this branch. To catch half-orphans during block allocations we needed to duplicate all metadata-pairs reported to lfs_fs_traverse. Unfortunately this causes lfs_fs_size to report 2x the number of metadata-pairs, which would undoubtably confuse users. The fix here is inelegantly simple, just do a different traversale for allocations and size measurements. It reuses the same code but touches slightly different sets of blocks. Unfortunately, this causes the public lfs_fs_traverse and lfs_fs_size functions to split in how they report blocks. This is technically allowed, since lfs_fs_traverse may report blocks multiple times due to CoW behavior, however it's undesirable and I'm sure there will be some confusion. But I don't have a better solution, so from this point lfs_fs_traverse will be reporting 2x metadata-blocks and shouldn't be used for finding the number of available blocks on the filesystem.
2020-02-09 15:05:37 +00:00
int err = lfs_fs_traverseraw(lfs, lfs_fs_size_count, &size, false);
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
if (err) {
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_fs_size -> %d", err);
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
return err;
}
Fix some LFS_TRACE format specifiers
2019-11-22 04:29:57 +00:00
LFS_TRACE("lfs_fs_size -> %d", err);
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
return size;
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
}
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
#ifdef LFS_MIGRATE
////// Migration from littelfs v1 below this //////
/// Version info ///
// Software library version
// Major (top-nibble), incremented on backwards incompatible changes
// Minor (bottom-nibble), incremented on feature additions
#define LFS1_VERSION 0x00010007
#define LFS1_VERSION_MAJOR (0xffff & (LFS1_VERSION >> 16))
#define LFS1_VERSION_MINOR (0xffff & (LFS1_VERSION >> 0))
// Version of On-disk data structures
// Major (top-nibble), incremented on backwards incompatible changes
// Minor (bottom-nibble), incremented on feature additions
#define LFS1_DISK_VERSION 0x00010001
#define LFS1_DISK_VERSION_MAJOR (0xffff & (LFS1_DISK_VERSION >> 16))
#define LFS1_DISK_VERSION_MINOR (0xffff & (LFS1_DISK_VERSION >> 0))
/// v1 Definitions ///
// File types
enum lfs1_type {
LFS1_TYPE_REG = 0x11,
LFS1_TYPE_DIR = 0x22,
LFS1_TYPE_SUPERBLOCK = 0x2e,
};
typedef struct lfs1 {
lfs_block_t root[2];
} lfs1_t;
typedef struct lfs1_entry {
lfs_off_t off;
struct lfs1_disk_entry {
uint8_t type;
uint8_t elen;
uint8_t alen;
uint8_t nlen;
union {
struct {
lfs_block_t head;
lfs_size_t size;
} file;
lfs_block_t dir[2];
} u;
} d;
} lfs1_entry_t;
typedef struct lfs1_dir {
struct lfs1_dir *next;
lfs_block_t pair[2];
lfs_off_t off;
lfs_block_t head[2];
lfs_off_t pos;
struct lfs1_disk_dir {
uint32_t rev;
lfs_size_t size;
lfs_block_t tail[2];
} d;
} lfs1_dir_t;
typedef struct lfs1_superblock {
lfs_off_t off;
struct lfs1_disk_superblock {
uint8_t type;
uint8_t elen;
uint8_t alen;
uint8_t nlen;
lfs_block_t root[2];
uint32_t block_size;
uint32_t block_count;
uint32_t version;
char magic[8];
} d;
} lfs1_superblock_t;
/// Low-level wrappers v1->v2 ///
Make lfs1_crc static so it doesn't conflict with prefixed LFS1 code When LFS1 code is present and LFS_MIGRATE is enabled
2019-05-16 16:51:22 +00:00
static void lfs1_crc(uint32_t *crc, const void *buffer, size_t size) {
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
*crc = lfs_crc(*crc, buffer, size);
}
static int lfs1_bd_read(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, void *buffer, lfs_size_t size) {
// if we ever do more than writes to alternating pairs,
// this may need to consider pcache
return lfs_bd_read(lfs, &lfs->pcache, &lfs->rcache, size,
block, off, buffer, size);
}
static int lfs1_bd_crc(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
int err = lfs1_bd_read(lfs, block, off+i, &c, 1);
if (err) {
return err;
}
lfs1_crc(crc, &c, 1);
}
return 0;
}
/// Endian swapping functions ///
static void lfs1_dir_fromle32(struct lfs1_disk_dir *d) {
d->rev = lfs_fromle32(d->rev);
d->size = lfs_fromle32(d->size);
d->tail[0] = lfs_fromle32(d->tail[0]);
d->tail[1] = lfs_fromle32(d->tail[1]);
}
static void lfs1_dir_tole32(struct lfs1_disk_dir *d) {
d->rev = lfs_tole32(d->rev);
d->size = lfs_tole32(d->size);
d->tail[0] = lfs_tole32(d->tail[0]);
d->tail[1] = lfs_tole32(d->tail[1]);
}
static void lfs1_entry_fromle32(struct lfs1_disk_entry *d) {
d->u.dir[0] = lfs_fromle32(d->u.dir[0]);
d->u.dir[1] = lfs_fromle32(d->u.dir[1]);
}
static void lfs1_entry_tole32(struct lfs1_disk_entry *d) {
d->u.dir[0] = lfs_tole32(d->u.dir[0]);
d->u.dir[1] = lfs_tole32(d->u.dir[1]);
}
static void lfs1_superblock_fromle32(struct lfs1_disk_superblock *d) {
d->root[0] = lfs_fromle32(d->root[0]);
d->root[1] = lfs_fromle32(d->root[1]);
d->block_size = lfs_fromle32(d->block_size);
d->block_count = lfs_fromle32(d->block_count);
d->version = lfs_fromle32(d->version);
}
///// Metadata pair and directory operations ///
static inline lfs_size_t lfs1_entry_size(const lfs1_entry_t *entry) {
return 4 + entry->d.elen + entry->d.alen + entry->d.nlen;
}
static int lfs1_dir_fetch(lfs_t *lfs,
lfs1_dir_t *dir, const lfs_block_t pair[2]) {
// copy out pair, otherwise may be aliasing dir
const lfs_block_t tpair[2] = {pair[0], pair[1]};
bool valid = false;
// check both blocks for the most recent revision
for (int i = 0; i < 2; i++) {
struct lfs1_disk_dir test;
int err = lfs1_bd_read(lfs, tpair[i], 0, &test, sizeof(test));
lfs1_dir_fromle32(&test);
if (err) {
if (err == LFS_ERR_CORRUPT) {
continue;
}
return err;
}
if (valid && lfs_scmp(test.rev, dir->d.rev) < 0) {
continue;
}
if ((0x7fffffff & test.size) < sizeof(test)+4 ||
(0x7fffffff & test.size) > lfs->cfg->block_size) {
continue;
}
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
uint32_t crc = 0xffffffff;
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
lfs1_dir_tole32(&test);
lfs1_crc(&crc, &test, sizeof(test));
lfs1_dir_fromle32(&test);
err = lfs1_bd_crc(lfs, tpair[i], sizeof(test),
(0x7fffffff & test.size) - sizeof(test), &crc);
if (err) {
if (err == LFS_ERR_CORRUPT) {
continue;
}
return err;
}
if (crc != 0) {
continue;
}
valid = true;
// setup dir in case it's valid
dir->pair[0] = tpair[(i+0) % 2];
dir->pair[1] = tpair[(i+1) % 2];
dir->off = sizeof(dir->d);
dir->d = test;
}
if (!valid) {
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_ERROR("Corrupted dir pair at {0x%"PRIx32", 0x%"PRIx32"}",
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
tpair[0], tpair[1]);
return LFS_ERR_CORRUPT;
}
return 0;
}
static int lfs1_dir_next(lfs_t *lfs, lfs1_dir_t *dir, lfs1_entry_t *entry) {
while (dir->off + sizeof(entry->d) > (0x7fffffff & dir->d.size)-4) {
if (!(0x80000000 & dir->d.size)) {
entry->off = dir->off;
return LFS_ERR_NOENT;
}
int err = lfs1_dir_fetch(lfs, dir, dir->d.tail);
if (err) {
return err;
}
dir->off = sizeof(dir->d);
dir->pos += sizeof(dir->d) + 4;
}
int err = lfs1_bd_read(lfs, dir->pair[0], dir->off,
&entry->d, sizeof(entry->d));
lfs1_entry_fromle32(&entry->d);
if (err) {
return err;
}
entry->off = dir->off;
dir->off += lfs1_entry_size(entry);
dir->pos += lfs1_entry_size(entry);
return 0;
}
/// littlefs v1 specific operations ///
int lfs1_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data) {
if (lfs_pair_isnull(lfs->lfs1->root)) {
return 0;
}
// iterate over metadata pairs
lfs1_dir_t dir;
lfs1_entry_t entry;
lfs_block_t cwd[2] = {0, 1};
while (true) {
for (int i = 0; i < 2; i++) {
int err = cb(data, cwd[i]);
if (err) {
return err;
}
}
int err = lfs1_dir_fetch(lfs, &dir, cwd);
if (err) {
return err;
}
// iterate over contents
while (dir.off + sizeof(entry.d) <= (0x7fffffff & dir.d.size)-4) {
err = lfs1_bd_read(lfs, dir.pair[0], dir.off,
&entry.d, sizeof(entry.d));
lfs1_entry_fromle32(&entry.d);
if (err) {
return err;
}
dir.off += lfs1_entry_size(&entry);
if ((0x70 & entry.d.type) == (0x70 & LFS1_TYPE_REG)) {
err = lfs_ctz_traverse(lfs, NULL, &lfs->rcache,
entry.d.u.file.head, entry.d.u.file.size, cb, data);
if (err) {
return err;
}
}
}
// we also need to check if we contain a threaded v2 directory
lfs_mdir_t dir2 = {.split=true, .tail={cwd[0], cwd[1]}};
while (dir2.split) {
err = lfs_dir_fetch(lfs, &dir2, dir2.tail);
if (err) {
break;
}
for (int i = 0; i < 2; i++) {
err = cb(data, dir2.pair[i]);
if (err) {
return err;
}
}
}
cwd[0] = dir.d.tail[0];
cwd[1] = dir.d.tail[1];
if (lfs_pair_isnull(cwd)) {
break;
}
}
return 0;
}
static int lfs1_moved(lfs_t *lfs, const void *e) {
if (lfs_pair_isnull(lfs->lfs1->root)) {
return 0;
}
// skip superblock
lfs1_dir_t cwd;
int err = lfs1_dir_fetch(lfs, &cwd, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
// iterate over all directory directory entries
lfs1_entry_t entry;
while (!lfs_pair_isnull(cwd.d.tail)) {
err = lfs1_dir_fetch(lfs, &cwd, cwd.d.tail);
if (err) {
return err;
}
while (true) {
err = lfs1_dir_next(lfs, &cwd, &entry);
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err == LFS_ERR_NOENT) {
break;
}
if (!(0x80 & entry.d.type) &&
memcmp(&entry.d.u, e, sizeof(entry.d.u)) == 0) {
return true;
}
}
}
return false;
}
/// Filesystem operations ///
static int lfs1_mount(lfs_t *lfs, struct lfs1 *lfs1,
const struct lfs_config *cfg) {
int err = 0;
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
err = lfs_init(lfs, cfg);
if (err) {
return err;
}
lfs->lfs1 = lfs1;
lfs: use meaningful names for magic block identifiers The difference between 0xffffffff and 0xfffffffe is too subtle. Use names that reflect what the value represents. Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2019-08-03 14:17:47 +00:00
lfs->lfs1->root[0] = LFS_BLOCK_NULL;
lfs->lfs1->root[1] = LFS_BLOCK_NULL;
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
// setup free lookahead
lfs->free.off = 0;
lfs->free.size = 0;
lfs->free.i = 0;
lfs_alloc_ack(lfs);
// load superblock
lfs1_dir_t dir;
lfs1_superblock_t superblock;
err = lfs1_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
if (err && err != LFS_ERR_CORRUPT) {
goto cleanup;
}
if (!err) {
err = lfs1_bd_read(lfs, dir.pair[0], sizeof(dir.d),
&superblock.d, sizeof(superblock.d));
lfs1_superblock_fromle32(&superblock.d);
if (err) {
goto cleanup;
}
lfs->lfs1->root[0] = superblock.d.root[0];
lfs->lfs1->root[1] = superblock.d.root[1];
}
if (err || memcmp(superblock.d.magic, "littlefs", 8) != 0) {
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_ERROR("Invalid superblock at {0x%"PRIx32", 0x%"PRIx32"}",
0, 1);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
err = LFS_ERR_CORRUPT;
goto cleanup;
}
uint16_t major_version = (0xffff & (superblock.d.version >> 16));
uint16_t minor_version = (0xffff & (superblock.d.version >> 0));
if ((major_version != LFS1_DISK_VERSION_MAJOR ||
minor_version > LFS1_DISK_VERSION_MINOR)) {
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_ERROR("Invalid version v%d.%d", major_version, minor_version);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
err = LFS_ERR_INVAL;
goto cleanup;
}
return 0;
}
cleanup:
lfs_deinit(lfs);
return err;
}
static int lfs1_unmount(lfs_t *lfs) {
return lfs_deinit(lfs);
}
/// v1 migration ///
int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_migrate(%p, %p {.context=%p, "
".read=%p, .prog=%p, .erase=%p, .sync=%p, "
".read_size=%"PRIu32", .prog_size=%"PRIu32", "
".block_size=%"PRIu32", .block_count=%"PRIu32", "
".block_cycles=%"PRIu32", .cache_size=%"PRIu32", "
".lookahead_size=%"PRIu32", .read_buffer=%p, "
".prog_buffer=%p, .lookahead_buffer=%p, "
".name_max=%"PRIu32", .file_max=%"PRIu32", "
".attr_max=%"PRIu32"})",
(void*)lfs, (void*)cfg, cfg->context,
(void*)(uintptr_t)cfg->read, (void*)(uintptr_t)cfg->prog,
(void*)(uintptr_t)cfg->erase, (void*)(uintptr_t)cfg->sync,
cfg->read_size, cfg->prog_size, cfg->block_size, cfg->block_count,
cfg->block_cycles, cfg->cache_size, cfg->lookahead_size,
cfg->read_buffer, cfg->prog_buffer, cfg->lookahead_buffer,
cfg->name_max, cfg->file_max, cfg->attr_max);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
struct lfs1 lfs1;
int err = lfs1_mount(lfs, &lfs1, cfg);
if (err) {
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_migrate -> %d", err);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
return err;
}
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
// iterate through each directory, copying over entries
// into new directory
lfs1_dir_t dir1;
lfs_mdir_t dir2;
dir1.d.tail[0] = lfs->lfs1->root[0];
dir1.d.tail[1] = lfs->lfs1->root[1];
while (!lfs_pair_isnull(dir1.d.tail)) {
// iterate old dir
err = lfs1_dir_fetch(lfs, &dir1, dir1.d.tail);
if (err) {
goto cleanup;
}
// create new dir and bind as temporary pretend root
err = lfs_dir_alloc(lfs, &dir2);
if (err) {
goto cleanup;
}
dir2.rev = dir1.d.rev;
Added v1->v2 migration into CI Also fixed issue where migration would not handle large dirs due to v1 iteration changing the pair of the directory.
2019-04-02 03:12:08 +00:00
dir1.head[0] = dir1.pair[0];
dir1.head[1] = dir1.pair[1];
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
lfs->root[0] = dir2.pair[0];
lfs->root[1] = dir2.pair[1];
err = lfs_dir_commit(lfs, &dir2, NULL, 0);
if (err) {
goto cleanup;
}
while (true) {
lfs1_entry_t entry1;
err = lfs1_dir_next(lfs, &dir1, &entry1);
if (err && err != LFS_ERR_NOENT) {
goto cleanup;
}
if (err == LFS_ERR_NOENT) {
break;
}
// check that entry has not been moved
if (entry1.d.type & 0x80) {
int moved = lfs1_moved(lfs, &entry1.d.u);
if (moved < 0) {
err = moved;
goto cleanup;
}
if (moved) {
continue;
}
entry1.d.type &= ~0x80;
}
Use LFS_F_OPENED flag to protect against use of not opened or closed file.
2019-07-21 09:34:53 +00:00
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
// also fetch name
char name[LFS_NAME_MAX+1];
memset(name, 0, sizeof(name));
err = lfs1_bd_read(lfs, dir1.pair[0],
entry1.off + 4+entry1.d.elen+entry1.d.alen,
name, entry1.d.nlen);
if (err) {
goto cleanup;
}
bool isdir = (entry1.d.type == LFS1_TYPE_DIR);
// create entry in new dir
err = lfs_dir_fetch(lfs, &dir2, lfs->root);
if (err) {
goto cleanup;
}
uint16_t id;
err = lfs_dir_find(lfs, &dir2, &(const char*){name}, &id);
if (!(err == LFS_ERR_NOENT && id != 0x3ff)) {
err = (err < 0) ? err : LFS_ERR_EXIST;
goto cleanup;
}
lfs1_entry_tole32(&entry1.d);
err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS(
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
{LFS_MKTAG(LFS_TYPE_CREATE, id, 0)},
{LFS_MKTAG_IF_ELSE(isdir,
LFS_TYPE_DIR, id, entry1.d.nlen,
LFS_TYPE_REG, id, entry1.d.nlen),
name},
{LFS_MKTAG_IF_ELSE(isdir,
LFS_TYPE_DIRSTRUCT, id, sizeof(entry1.d.u),
LFS_TYPE_CTZSTRUCT, id, sizeof(entry1.d.u)),
&entry1.d.u}));
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
lfs1_entry_fromle32(&entry1.d);
if (err) {
goto cleanup;
}
}
if (!lfs_pair_isnull(dir1.d.tail)) {
// find last block and update tail to thread into fs
err = lfs_dir_fetch(lfs, &dir2, lfs->root);
if (err) {
goto cleanup;
}
while (dir2.split) {
err = lfs_dir_fetch(lfs, &dir2, dir2.tail);
if (err) {
goto cleanup;
}
}
lfs_pair_tole32(dir2.pair);
err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS(
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
{LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), dir1.d.tail}));
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
lfs_pair_fromle32(dir2.pair);
if (err) {
goto cleanup;
}
}
// Copy over first block to thread into fs. Unfortunately
// if this fails there is not much we can do.
Last minute tweaks to debug scripts - Standardized littlefs debug statements to use hex prefixes and brackets for printing pairs. - Removed the entry behavior for readtree and made -t the default. This is because 1. the CTZ skip-list parsing was broken, which is not surprising, and 2. the entry parsing was more complicated than useful. This functionality may be better implemented as a proper filesystem read script, complete with directory tree dumping. - Changed test.py's --gdb argument to take [init, main, assert], this matches the names of the stages in C's startup. - Added printing of tail to all mdir dumps in readtree/readmdir. - Added a print for if any mdirs are corrupted in readtree. - Added debug script side-effects to .gitignore.
2020-03-30 02:19:33 +00:00
LFS_DEBUG("Migrating {0x%"PRIx32", 0x%"PRIx32"} "
"-> {0x%"PRIx32", 0x%"PRIx32"}",
Added v1->v2 migration into CI Also fixed issue where migration would not handle large dirs due to v1 iteration changing the pair of the directory.
2019-04-02 03:12:08 +00:00
lfs->root[0], lfs->root[1], dir1.head[0], dir1.head[1]);
err = lfs_bd_erase(lfs, dir1.head[1]);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
if (err) {
goto cleanup;
}
err = lfs_dir_fetch(lfs, &dir2, lfs->root);
if (err) {
goto cleanup;
}
for (lfs_off_t i = 0; i < dir2.off; i++) {
uint8_t dat;
err = lfs_bd_read(lfs,
NULL, &lfs->rcache, dir2.off,
dir2.pair[0], i, &dat, 1);
if (err) {
goto cleanup;
}
err = lfs_bd_prog(lfs,
&lfs->pcache, &lfs->rcache, true,
Added v1->v2 migration into CI Also fixed issue where migration would not handle large dirs due to v1 iteration changing the pair of the directory.
2019-04-02 03:12:08 +00:00
dir1.head[1], i, &dat, 1);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
if (err) {
goto cleanup;
}
}
Fixed missing cache flush in lfs_migrate The data written to the prog cache would make littlefs internally consistent, but because this was never written to disk, the filesystem would become unmountable. Unfortunately, this wasn't found during testing because caches automatically flush if data is written up to a program boundary (maybe this was a mistake?). Found by rojer
2019-05-28 16:35:22 +00:00
err = lfs_bd_flush(lfs, &lfs->pcache, &lfs->rcache, true);
if (err) {
goto cleanup;
}
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
}
// Create new superblock. This marks a successful migration!
err = lfs1_dir_fetch(lfs, &dir1, (const lfs_block_t[2]){0, 1});
if (err) {
goto cleanup;
}
dir2.pair[0] = dir1.pair[0];
dir2.pair[1] = dir1.pair[1];
dir2.rev = dir1.d.rev;
dir2.off = sizeof(dir2.rev);
Fixed CRC check when prog_size causes multiple CRCs per commit This is a bit of a strange case that can be caused by storage with very large prog sizes, such as NAND flash. We only have 10 bits to store the size of our padding, so when the prog_size gets larger than 1024 bytes, we have to use multiple padding tags to commit to the next prog_size boundary. This causes some complication for the new logic that checks CRCs in case our block becomes "readonly" and contains existing commits that just happen to match our new commit size. Here we just check the CRC of the first commit. This isn't perfect but does protect against pure "readonly" blocks.
2020-02-10 04:43:20 +00:00
dir2.etag = 0xffffffff;
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
dir2.count = 0;
dir2.tail[0] = lfs->lfs1->root[0];
dir2.tail[1] = lfs->lfs1->root[1];
dir2.erased = false;
dir2.split = true;
lfs_superblock_t superblock = {
.version = LFS_DISK_VERSION,
.block_size = lfs->cfg->block_size,
.block_count = lfs->cfg->block_count,
.name_max = lfs->name_max,
.file_max = lfs->file_max,
.attr_max = lfs->attr_max,
};
lfs_superblock_tole32(&superblock);
err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS(
Fixed issues with neighbor updates during moves The root of the problem was some assumptions about what tags could be sent to lfs_dir_commit. - The first assumption is that there could be only one splice (create/delete) tag at a time, which is trivially broken by the core commit in lfs_rename. - The second assumption is that there is at most one create and one delete in a single commit. This is less obvious but turns out to not be true in the case that we rename a file such that it overwrites another file in the same directory (1 delete for source file, 1 delete for destination). - The third assumption was that there was an ordering to the delete/creates passed to lfs_dir_commit. It may be possible to force all deletes to follow creates by rearranging the tags in lfs_rename, but this risks overflowing tag ids. The way the lfs_dir_commit first collected the "deletetag" and "createtag" broke all three of these assumptions. And because we lose the ordering information we can no longer apply the directory changes to open files correctly. The file ids may be shifted in a way that doesn't reflect the actual operations on disk. These problems were made worst by lfs_dir_commit cleaning up moves implicitly, which also creates deletes implicitly. While cleaning up moves in lfs_dir_commit may save some code size, it makes the commit logic much more difficult to implement correctly. This bug turned into pulling out a dead tree stump, roots and all. I ended up reworking how lfs_dir_commit updates open files so that it has less assumptions, now it just traverses the commit tags multiple times in order to update file ids after a successful commit in the correct order. This also got rid of the dir copy by carefully updating split dirs after all files have an up-to-date copy of the original dir. I also just removed the implicit move cleanup. It turns out the only commits that can occur before we have cleaned up the move is in lfs_fs_relocate, so it was simple enough to explicitly handle this case when we update our parent and pred during a relocate. Cases where we may need to fix moves: - In lfs_rename when we move a file/dir - In lfs_demove if we lose power - In lfs_fs_relocate if we have to relocate our parent and we find it had a pending move (or else the move will be outdated) - In lfs_fs_relocate if we have to relocate our predecessor and we find it had a pending move (or else the move will be outdated) Note the two cases in lfs_fs_relocate may be recursive. But lfs_fs_relocate can only trigger other lfs_fs_relocates so it's not possible for pending moves to spill out into other filesystem commits And of couse, I added several tests to cover these situations. Hopefully the rename-with-open-files logic should be fairly locked down now. found with initial fix by eastmoutain
2020-01-20 23:35:45 +00:00
{LFS_MKTAG(LFS_TYPE_CREATE, 0, 0)},
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
{LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, 8), "littlefs"},
{LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)),
&superblock}));
if (err) {
goto cleanup;
}
// sanity check that fetch works
err = lfs_dir_fetch(lfs, &dir2, (const lfs_block_t[2]){0, 1});
if (err) {
goto cleanup;
}
Added redundant compaction to lfs_format/lfs_migrate This ensures that both blocks in the superblock pair are written with the superblock info. While this does use an additional erase cycle, it prevents older versions of littlefs from accidentally being picked up in the case that the disk is mounted on a system that doesn't support the newer version. This does bring back the risk of picking up old littlefs versions on a disk that has been formatted with a filesystem that doesn't use block 2 (such as FAT), but this risk already exists, and moving between versions of littlefs is more likely with the recent v1 -> v2 update. Suggested by rojer
2019-05-28 23:16:51 +00:00
// force compaction to prevent accidentally mounting v1
dir2.erased = false;
err = lfs_dir_commit(lfs, &dir2, NULL, 0);
if (err) {
goto cleanup;
}
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
}
cleanup:
lfs1_unmount(lfs);
Added trace statements through LFS_YES_TRACE To use, compile and run with LFS_YES_TRACE defined: make CFLAGS+=-DLFS_YES_TRACE=1 test_format The name LFS_YES_TRACE was chosen to match the LFS_NO_DEBUG and LFS_NO_WARN defines for the similar levels of output. The YES is necessary to avoid a conflict with the actual LFS_TRACE macro that gets emitting. LFS_TRACE can also be defined directly to provide a custom trace formatter. Hopefully having trace statements at the littlefs C API helps debugging and reproducing issues.
2019-05-31 09:40:19 +00:00
LFS_TRACE("lfs_migrate -> %d", err);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
return err;
}
#endif
Reference in New Issue Copy Permalink