openharmony/third_party_littlefs
1
0
Fork 0
mirror of https://gitee.com/openharmony/third_party_littlefs synced 2024-11-23 14:59:50 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
df2e676562
third_party_littlefs/lfs.c

4473 lines
129 KiB
C
Raw Normal View History

Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
/*
* The little filesystem
*
Updated copyright Due to employee contract Per ARM license remains under Apache 2.0
2017-10-13 01:27:33 +00:00
* Copyright (c) 2017 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
*/
#include "lfs.h"
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
#include "lfs_util.h"
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// Caching block device operations ///
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
static inline void lfs_cache_drop(lfs_t *lfs, lfs_cache_t *rcache) {
// do not zero, cheaper if cache is readonly or only going to be
// written with identical data (during relocates)
(void)lfs;
rcache->block = 0xffffffff;
}
static inline void lfs_cache_zero(lfs_t *lfs, lfs_cache_t *pcache) {
// zero to avoid information leak
Fixed buffer overflow due to mistaking prog_size for cache_size found by ajaybhargav
2019-04-12 13:41:42 +00:00
memset(pcache->buffer, 0xff, lfs->cfg->cache_size);
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
pcache->block = 0xffffffff;
}
static int lfs_bd_read(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache, lfs_size_t hint,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_block_t block, lfs_off_t off,
void *buffer, lfs_size_t size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
if (off+size > lfs->cfg->block_size) {
return LFS_ERR_CORRUPT;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
lfs_size_t diff = size;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (pcache && block == pcache->block &&
off < pcache->off + pcache->size) {
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
if (off >= pcache->off) {
// is already in pcache?
diff = lfs_min(diff, pcache->size - (off-pcache->off));
memcpy(data, &pcache->buffer[off-pcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
// pcache takes priority
diff = lfs_min(diff, pcache->off-off);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (block == rcache->block &&
off < rcache->off + rcache->size) {
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
if (off >= rcache->off) {
// is already in rcache?
diff = lfs_min(diff, rcache->size - (off-rcache->off));
memcpy(data, &rcache->buffer[off-rcache->off], diff);
data += diff;
off += diff;
size -= diff;
continue;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
// rcache takes priority
diff = lfs_min(diff, rcache->off-off);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
// load to cache, first condition can no longer fail
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
rcache->block = block;
Fixed issue with reads ignoring the pcache The downside of smarter caching is that now there are more complicated corner cases to consider. Here we weren't considering our pcaches when aligning reads to the rcache. This meant if things were unaligned, we would read a cache-line that overlaps the pcache and then proceed to ignore whatever we overlapped. This fix is to determine the limit of an rcache read not from cache alignment but from the available caches, which we check anyways to find cached data.
2018-10-02 20:42:07 +00:00
rcache->off = lfs_aligndown(off, lfs->cfg->read_size);
Fixed read cache amount based on hint and offset Found by apmorton
2019-05-21 22:21:52 +00:00
rcache->size = lfs_min(
lfs_min(
lfs_alignup(off+hint, lfs->cfg->read_size),
lfs->cfg->block_size)
- rcache->off,
lfs->cfg->cache_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
int err = lfs->cfg->read(lfs->cfg, rcache->block,
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
rcache->off, rcache->buffer, rcache->size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
enum {
LFS_CMP_EQ = 0,
LFS_CMP_LT = 1,
LFS_CMP_GT = 2,
};
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
static int lfs_bd_cmp(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache, lfs_size_t hint,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_block_t block, lfs_off_t off,
const void *buffer, lfs_size_t size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
const uint8_t *data = buffer;
for (lfs_off_t i = 0; i < size; i++) {
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
uint8_t dat;
int err = lfs_bd_read(lfs,
pcache, rcache, hint-i,
block, off+i, &dat, 1);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (err) {
return err;
}
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
if (dat != data[i]) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return (dat < data[i]) ? LFS_CMP_LT : LFS_CMP_GT;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return LFS_CMP_EQ;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
static int lfs_bd_flush(lfs_t *lfs,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_cache_t *pcache, lfs_cache_t *rcache, bool validate) {
Fixed inline files when inline_max == cache_size The initial implementation of inline files was thrown together fairly quicky, however it has worked well so far and there hasn't been much reason to change it. One shortcut was to trick file writes into thinking they are writing to imaginary blocks. This works well and reuses most of the file code paths, as long as we don't flush the imaginary block out to disk. Initially we did this by limiting inline_max to cache_max-1, ensuring that the cache never fills up and gets flushed. This was a rather dirty hack, the better solution, implemented here, is to handle the representation of an "imaginary" block correctly all the way down into the cache layer. So now for files specifically, the value -1 represents a null pointer, and the value -2 represents an "imaginary" block. This may become a problem if the number of blocks approaches the max, however this -2 value is never written to disk and can be changed in the future without breaking compatibility.
2018-08-13 19:08:30 +00:00
if (pcache->block != 0xffffffff && pcache->block != 0xfffffffe) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(pcache->block < lfs->cfg->block_count);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_size_t diff = lfs_alignup(pcache->size, lfs->cfg->prog_size);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs->cfg->prog(lfs->cfg, pcache->block,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
pcache->off, pcache->buffer, diff);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (validate) {
// check data on disk
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, rcache);
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int res = lfs_bd_cmp(lfs,
NULL, rcache, diff,
pcache->block, pcache->off, pcache->buffer, diff);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (res < 0) {
return res;
}
if (res != LFS_CMP_EQ) {
return LFS_ERR_CORRUPT;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
}
Fixed information leaks through reused caches As a shortcut, littlefs never bother to zero any of the buffers is used. It didn't need to because it would always write out the entirety of the data it needed. Unfortunately, this, combined with the extra padding used to align buffers to the nearest prog size, would lead to uninitialized data getting written out to disk. This means unrelated file data could be written to different parts of storage, or worse, information leaked from the malloc calls could be written out to disk unnecessarily. found by rojer
2018-07-06 16:14:30 +00:00
lfs_cache_zero(lfs, pcache);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
return 0;
}
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
static int lfs_bd_sync(lfs_t *lfs,
lfs_cache_t *pcache, lfs_cache_t *rcache, bool validate) {
lfs_cache_drop(lfs, rcache);
int err = lfs_bd_flush(lfs, pcache, rcache, validate);
if (err) {
return err;
}
return lfs->cfg->sync(lfs->cfg);
}
static int lfs_bd_prog(lfs_t *lfs,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_cache_t *pcache, lfs_cache_t *rcache, bool validate,
lfs_block_t block, lfs_off_t off,
const void *buffer, lfs_size_t size) {
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
const uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
LFS_ASSERT(off + size <= lfs->cfg->block_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (block == pcache->block &&
off >= pcache->off &&
off < pcache->off + lfs->cfg->cache_size) {
// already fits in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->cfg->cache_size - (off-pcache->off));
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
memcpy(&pcache->buffer[off-pcache->off], data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Fixed issues with reading and caching inline files Kind of a two-fold issue. One, the programming to the middle of inline files was causing the cache to get updated to a half programmed state. While fine, as all programs do occur in order in a block, this is less efficient when writing to inline files as it would cause the inline file to need to be reread even if it fits in the cache. Two, the rereading of the inline file was broken and passed the file's tag all the way to where a user would expect an error. This was easy to fix but adds to the reasons we should have test coverage information. Found by ebinans
2019-05-31 05:58:48 +00:00
pcache->size = lfs_max(pcache->size, off - pcache->off);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (pcache->size == lfs->cfg->cache_size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// eagerly flush out pcache if we fill up
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int err = lfs_bd_flush(lfs, pcache, rcache, validate);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// pcache must have been flushed, either by programming and
// entire block or manually flushing the pcache
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(pcache->block == 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// prepare pcache, first condition can no longer fail
pcache->block = block;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
pcache->off = lfs_aligndown(off, lfs->cfg->prog_size);
pcache->size = 0;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_erase(lfs_t *lfs, lfs_block_t block) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->erase(lfs->cfg, block);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
/// Small type-level utilities ///
// operations on block pairs
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline void lfs_pair_swap(lfs_block_t pair[2]) {
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
lfs_block_t t = pair[0];
pair[0] = pair[1];
pair[1] = t;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline bool lfs_pair_isnull(const lfs_block_t pair[2]) {
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
return pair[0] == 0xffffffff || pair[1] == 0xffffffff;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline int lfs_pair_cmp(
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
return !(paira[0] == pairb[0] || paira[1] == pairb[1] ||
paira[0] == pairb[1] || paira[1] == pairb[0]);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline bool lfs_pair_sync(
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
return (paira[0] == pairb[0] && paira[1] == pairb[1]) ||
(paira[0] == pairb[1] && paira[1] == pairb[0]);
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline void lfs_pair_fromle32(lfs_block_t pair[2]) {
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
pair[0] = lfs_fromle32(pair[0]);
pair[1] = lfs_fromle32(pair[1]);
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static inline void lfs_pair_tole32(lfs_block_t pair[2]) {
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
pair[0] = lfs_tole32(pair[0]);
pair[1] = lfs_tole32(pair[1]);
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// operations on 32-bit entry tags
typedef uint32_t lfs_tag_t;
typedef int32_t lfs_stag_t;
Added root entry and expanding superblocks Expanding superblocks has been on my wishlist for a while. The basic idea is that instead of maintaining a fixed offset blocks {0, 1} to the the root directory (1 pointer), we maintain a dynamically sized linked-list of superblocks that point to the actual root. If the number of writes to the root exceeds some value, we increase the size of the superblock linked-list. This can leverage existing metadata-pair operations. The revision count for metadata-pairs provides some knowledge on how much wear we've put on the superblock, and the threaded linked-list can also be reused for this purpose. This means superblock expansion is both optional and cheap to implement. Expanding superblocks helps both extremely small and extremely large filesystem (extreme being relative of course). On the small end, we can actually collapse the superblock into the root directory and drop the hard requirement of 4-blocks for the superblock. On the large end, our superblock will now last longer than the rest of the filesystem. Each time we expand, the number of cycles until the superblock dies is increased by a power. Before we were stuck with this layout: level cycles limit layout 1 E^2 390 MiB s0 -> root Now we expand every time a fixed offset is exceeded: level cycles limit layout 0 E 4 KiB s0+root 1 E^2 390 MiB s0 -> root 2 E^3 37 TiB s0 -> s1 -> root 3 E^4 3.6 EiB s0 -> s1 -> s2 -> root ... Where the cycles are the number of cycles before death, and the limit is the worst-case size a filesystem where early superblock death becomes a concern (all writes to root using this formula: E^|s| = E*B, E = erase cycles = 100000, B = block count, assuming 4096 byte blocks). Note we can also store copies of the superblock entry on the expanded superblocks. This may help filesystem recover tools in the future.
2018-08-06 18:30:51 +00:00
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
#define LFS_MKTAG(type, id, size) \
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
(((lfs_tag_t)(type) << 20) | ((lfs_tag_t)(id) << 10) | (lfs_tag_t)(size))
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline bool lfs_tag_isvalid(lfs_tag_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return !(tag & 0x80000000);
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static inline bool lfs_tag_isdelete(lfs_tag_t tag) {
return ((int32_t)(tag << 22) >> 22) == -1;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static inline uint16_t lfs_tag_type1(lfs_tag_t tag) {
return (tag & 0x70000000) >> 20;
}
static inline uint16_t lfs_tag_type3(lfs_tag_t tag) {
return (tag & 0x7ff00000) >> 20;
Added support for deleting attributes littlefs has a mechanism for deleting file entries, but it doesn't have a mechanism for deleting individual tags. This _is_ sufficient for a filesystem, but limits our flexibility. Deleting attributes would be useful in the custom attribute API and for future improvements (hint the child pointers in B-trees). However, deleteing attributes is tricky. We can't just omit the attribute, since we can only add new tags. Additionally, we need a way to track what attributes have been deleted during compaction, which currently relies on writing out attributes to disk. The solution here is pretty nifty. First we have to come up with a way to represent a "deleted" attribute. Rather than adding an additional bit to the already squished tag structure, we use a -1 length field, specifically 0xfff. Now we can commit a delete attribute, and this deleted tag acts as a place holder during compacts. However our delete tag will never leave our metadata log. We need some way to discard our delete tag if we know it's the only representation of that tag on the metadata log. Ah! We know it's the only tag if it's in the first commit on the metadata log. So we add an additional bit to the CRC entry to indicate if we're on the first commit, and use that to decide if we need to keep delete tags around. Now we have working tag deletion. Interestingly enough, tag deletion is actually indirectly more efficient than entry deletion, since compacting entries requires multiple passes, whereas tag deletion gets cleaned up lazily. However we can't adopt the same strategy in entry deletion because of the compact ordering of entries. Tag deletion works because tag types are unique and static. Managing entry deletion in this manner would require static id allocation, which would cause problems when creating files, running out of space, and disallow arbitrary insertions of files.
2018-09-09 22:48:11 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static inline uint8_t lfs_tag_chunk(lfs_tag_t tag) {
return (tag & 0x0ff00000) >> 20;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static inline int8_t lfs_tag_splice(lfs_tag_t tag) {
return (int8_t)lfs_tag_chunk(tag);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline uint16_t lfs_tag_id(lfs_tag_t tag) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return (tag & 0x000ffc00) >> 10;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline lfs_size_t lfs_tag_size(lfs_tag_t tag) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return tag & 0x000003ff;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline lfs_size_t lfs_tag_dsize(lfs_tag_t tag) {
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
return sizeof(tag) + lfs_tag_size(tag + lfs_tag_isdelete(tag));
Added support for deleting attributes littlefs has a mechanism for deleting file entries, but it doesn't have a mechanism for deleting individual tags. This _is_ sufficient for a filesystem, but limits our flexibility. Deleting attributes would be useful in the custom attribute API and for future improvements (hint the child pointers in B-trees). However, deleteing attributes is tricky. We can't just omit the attribute, since we can only add new tags. Additionally, we need a way to track what attributes have been deleted during compaction, which currently relies on writing out attributes to disk. The solution here is pretty nifty. First we have to come up with a way to represent a "deleted" attribute. Rather than adding an additional bit to the already squished tag structure, we use a -1 length field, specifically 0xfff. Now we can commit a delete attribute, and this deleted tag acts as a place holder during compacts. However our delete tag will never leave our metadata log. We need some way to discard our delete tag if we know it's the only representation of that tag on the metadata log. Ah! We know it's the only tag if it's in the first commit on the metadata log. So we add an additional bit to the CRC entry to indicate if we're on the first commit, and use that to decide if we need to keep delete tags around. Now we have working tag deletion. Interestingly enough, tag deletion is actually indirectly more efficient than entry deletion, since compacting entries requires multiple passes, whereas tag deletion gets cleaned up lazily. However we can't adopt the same strategy in entry deletion because of the compact ordering of entries. Tag deletion works because tag types are unique and static. Managing entry deletion in this manner would require static id allocation, which would cause problems when creating files, running out of space, and disallow arbitrary insertions of files.
2018-09-09 22:48:11 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// operations on attributes in attribute lists
struct lfs_mattr {
lfs_tag_t tag;
const void *buffer;
};
struct lfs_diskoff {
lfs_block_t block;
lfs_off_t off;
};
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
#define LFS_MKATTRS(...) \
(struct lfs_mattr[]){__VA_ARGS__}, \
sizeof((struct lfs_mattr[]){__VA_ARGS__}) / sizeof(struct lfs_mattr)
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// operations on global state
static inline void lfs_gstate_xor(struct lfs_gstate *a,
const struct lfs_gstate *b) {
for (int i = 0; i < 3; i++) {
((uint32_t*)a)[i] ^= ((const uint32_t*)b)[i];
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static inline bool lfs_gstate_iszero(const struct lfs_gstate *a) {
for (int i = 0; i < 3; i++) {
if (((uint32_t*)a)[i] != 0) {
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
return false;
}
}
return true;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static inline bool lfs_gstate_hasorphans(const struct lfs_gstate *a) {
return lfs_tag_size(a->tag);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static inline uint8_t lfs_gstate_getorphans(const struct lfs_gstate *a) {
return lfs_tag_size(a->tag);
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static inline bool lfs_gstate_hasmove(const struct lfs_gstate *a) {
return lfs_tag_type1(a->tag);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static inline bool lfs_gstate_hasmovehere(const struct lfs_gstate *a,
const lfs_block_t *pair) {
return lfs_tag_type1(a->tag) && lfs_pair_cmp(a->pair, pair) == 0;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
static inline void lfs_gstate_xororphans(struct lfs_gstate *a,
const struct lfs_gstate *b, bool orphans) {
a->tag ^= LFS_MKTAG(0x800, 0, 0) & (b->tag ^ (orphans << 31));
}
static inline void lfs_gstate_xormove(struct lfs_gstate *a,
const struct lfs_gstate *b, uint16_t id, const lfs_block_t pair[2]) {
a->tag ^= LFS_MKTAG(0x7ff, 0x3ff, 0) & (b->tag ^ (
(id != 0x3ff) ? LFS_MKTAG(LFS_TYPE_DELETE, id, 0) : 0));
a->pair[0] ^= b->pair[0] ^ ((id != 0x3ff) ? pair[0] : 0);
a->pair[1] ^= b->pair[1] ^ ((id != 0x3ff) ? pair[1] : 0);
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static inline void lfs_gstate_fromle32(struct lfs_gstate *a) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
a->tag = lfs_fromle32(a->tag);
a->pair[0] = lfs_fromle32(a->pair[0]);
a->pair[1] = lfs_fromle32(a->pair[1]);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
static inline void lfs_gstate_tole32(struct lfs_gstate *a) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
a->tag = lfs_tole32(a->tag);
a->pair[0] = lfs_tole32(a->pair[0]);
a->pair[1] = lfs_tole32(a->pair[1]);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// other endianness operations
static void lfs_ctz_fromle32(struct lfs_ctz *ctz) {
ctz->head = lfs_fromle32(ctz->head);
ctz->size = lfs_fromle32(ctz->size);
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static void lfs_ctz_tole32(struct lfs_ctz *ctz) {
ctz->head = lfs_tole32(ctz->head);
ctz->size = lfs_tole32(ctz->size);
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline void lfs_superblock_fromle32(lfs_superblock_t *superblock) {
superblock->version = lfs_fromle32(superblock->version);
superblock->block_size = lfs_fromle32(superblock->block_size);
superblock->block_count = lfs_fromle32(superblock->block_count);
Changed attr_max to be specific to custom attributes While technically, both system and user attributes share the same disk limitations, that's not what attr_max represents when considered from the user's perspective. To the user, attr_max applies only to custom attributes. This means attr_max should not impact other configurable limitations, such as inline files, and the ordering should be reconsidered with what the user finds most important.
2018-10-02 23:28:37 +00:00
superblock->name_max = lfs_fromle32(superblock->name_max);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
superblock->file_max = lfs_fromle32(superblock->file_max);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
superblock->attr_max = lfs_fromle32(superblock->attr_max);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static inline void lfs_superblock_tole32(lfs_superblock_t *superblock) {
superblock->version = lfs_tole32(superblock->version);
superblock->block_size = lfs_tole32(superblock->block_size);
superblock->block_count = lfs_tole32(superblock->block_count);
Changed attr_max to be specific to custom attributes While technically, both system and user attributes share the same disk limitations, that's not what attr_max represents when considered from the user's perspective. To the user, attr_max applies only to custom attributes. This means attr_max should not impact other configurable limitations, such as inline files, and the ordering should be reconsidered with what the user finds most important.
2018-10-02 23:28:37 +00:00
superblock->name_max = lfs_tole32(superblock->name_max);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
superblock->file_max = lfs_tole32(superblock->file_max);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
superblock->attr_max = lfs_tole32(superblock->attr_max);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
Added support for entry insertion Currently unused, the insertion of new file entries in arbitrary locations in a metadata-pair is very easy to add into the existing metadata logging. The only tricky things: 1. Name tags must strictly precede any tags related to a file. We can pull this off during a compact, but must make two passes. One for the name tag, one for the file. Though a benefit of this is that now our scans during moves can exit early upon finding the name tag. 1. We need to handle name tags appearing out of order. This makes name tags symmetric to deletes, although it doesn't seem like we can leverage this fact very well. Note this also means we need to make the superblock tag a type of name tag.
2018-09-09 14:01:06 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
/// Internal operations predeclared here ///
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir,
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
const struct lfs_mattr *attrs, int attrcount);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_compact(lfs_t *lfs,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_mdir_t *dir, const struct lfs_mattr *attrs, int attrcount,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_mdir_t *source, uint16_t begin, uint16_t end);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
static int lfs_file_relocate(lfs_t *lfs, lfs_file_t *file);
static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file);
static void lfs_fs_preporphans(lfs_t *lfs, int8_t orphans);
static void lfs_fs_prepmove(lfs_t *lfs,
uint16_t id, const lfs_block_t pair[2]);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static int lfs_fs_pred(lfs_t *lfs, const lfs_block_t dir[2],
lfs_mdir_t *pdir);
static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t dir[2],
lfs_mdir_t *parent);
static int lfs_fs_relocate(lfs_t *lfs,
const lfs_block_t oldpair[2], lfs_block_t newpair[2]);
static int lfs_fs_forceconsistency(lfs_t *lfs);
static int lfs_deinit(lfs_t *lfs);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
#ifdef LFS_MIGRATE
static int lfs1_traverse(lfs_t *lfs,
int (*cb)(void*, lfs_block_t), void *data);
#endif
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
/// Block allocator ///
static int lfs_alloc_lookahead(void *p, lfs_block_t block) {
lfs_t *lfs = (lfs_t*)p;
lfs_block_t off = ((block - lfs->free.off)
+ lfs->cfg->block_count) % lfs->cfg->block_count;
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (off < lfs->free.size) {
lfs->free.buffer[off / 32] |= 1U << (off % 32);
}
return 0;
}
static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) {
while (true) {
while (lfs->free.i != lfs->free.size) {
lfs_block_t off = lfs->free.i;
lfs->free.i += 1;
lfs->free.ack -= 1;
if (!(lfs->free.buffer[off / 32] & (1U << (off % 32)))) {
// found a free block
*block = (lfs->free.off + off) % lfs->cfg->block_count;
// eagerly find next off so an alloc ack can
// discredit old lookahead blocks
while (lfs->free.i != lfs->free.size &&
(lfs->free.buffer[lfs->free.i / 32]
& (1U << (lfs->free.i % 32)))) {
lfs->free.i += 1;
lfs->free.ack -= 1;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
return 0;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Added support for entry insertion Currently unused, the insertion of new file entries in arbitrary locations in a metadata-pair is very easy to add into the existing metadata logging. The only tricky things: 1. Name tags must strictly precede any tags related to a file. We can pull this off during a compact, but must make two passes. One for the name tag, one for the file. Though a benefit of this is that now our scans during moves can exit early upon finding the name tag. 1. We need to handle name tags appearing out of order. This makes name tags symmetric to deletes, although it doesn't seem like we can leverage this fact very well. Note this also means we need to make the superblock tag a type of name tag.
2018-09-09 14:01:06 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// check if we have looked at all blocks since last ack
if (lfs->free.ack == 0) {
LFS_WARN("No more free space %"PRIu32,
lfs->free.i + lfs->free.off);
return LFS_ERR_NOSPC;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->free.off = (lfs->free.off + lfs->free.size)
% lfs->cfg->block_count;
Changed lookahead configuration unit to bytes instead of bits The fact that the lookahead buffer uses bits instead of bytes is an internal detail. Poking this through to the user API has caused a decent amount of confusion. Most buffers are provided as bytes and the inconsistency here can be surprising. The use of bytes instead of bits also makes us forward compatible in the case that we want to change the lookahead internal representation (hint segment list). Additionally, we change the configuration name to lookahead_size. This matches other configurations, such as cache_size and read_size, while also notifying the user that something important changed at compile time (by breaking).
2018-10-03 01:18:30 +00:00
lfs->free.size = lfs_min(8*lfs->cfg->lookahead_size, lfs->free.ack);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->free.i = 0;
// find mask of free blocks from tree
Changed lookahead configuration unit to bytes instead of bits The fact that the lookahead buffer uses bits instead of bytes is an internal detail. Poking this through to the user API has caused a decent amount of confusion. Most buffers are provided as bytes and the inconsistency here can be surprising. The use of bytes instead of bits also makes us forward compatible in the case that we want to change the lookahead internal representation (hint segment list). Additionally, we change the configuration name to lookahead_size. This matches other configurations, such as cache_size and read_size, while also notifying the user that something important changed at compile time (by breaking).
2018-10-03 01:18:30 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead_size);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
int err = lfs_fs_traverse(lfs, lfs_alloc_lookahead, lfs);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
return err;
}
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
static void lfs_alloc_ack(lfs_t *lfs) {
lfs->free.ack = lfs->cfg->block_count;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
/// Metadata pair and directory operations ///
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
static lfs_stag_t lfs_dir_getslice(lfs_t *lfs, const lfs_mdir_t *dir,
lfs_tag_t gmask, lfs_tag_t gtag,
lfs_off_t goff, void *gbuffer, lfs_size_t gsize) {
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_off_t off = dir->off;
lfs_tag_t ntag = dir->etag;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t gdiff = 0;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (lfs_gstate_hasmovehere(&lfs->gstate, dir->pair) &&
lfs_tag_id(gtag) <= lfs_tag_id(lfs->gstate.tag)) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// synthetic moves
gdiff -= LFS_MKTAG(0, 1, 0);
}
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// iterate over dir block backwards (for faster lookups)
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
while (off >= sizeof(lfs_tag_t) + lfs_tag_dsize(ntag)) {
off -= lfs_tag_dsize(ntag);
lfs_tag_t tag = ntag;
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(ntag),
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->pair[0], off, &ntag, sizeof(ntag));
if (err) {
return err;
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
ntag = (lfs_frombe32(ntag) ^ tag) & 0x7fffffff;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_id(gmask) != 0 &&
lfs_tag_type1(tag) == LFS_TYPE_SPLICE &&
lfs_tag_id(tag) <= lfs_tag_id(gtag - gdiff)) {
if (tag == (LFS_MKTAG(LFS_TYPE_CREATE, 0, 0) |
(LFS_MKTAG(0, 0x3ff, 0) & (gtag - gdiff)))) {
// found where we were created
return LFS_ERR_NOENT;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// move around splices
gdiff += LFS_MKTAG(0, lfs_tag_splice(tag), 0);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if ((gmask & tag) == (gmask & (gtag - gdiff))) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (lfs_tag_isdelete(tag)) {
return LFS_ERR_NOENT;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
lfs_size_t diff = lfs_min(lfs_tag_size(tag), gsize);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, diff,
dir->pair[0], off+sizeof(tag)+goff, gbuffer, diff);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (err) {
return err;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
memset((uint8_t*)gbuffer + diff, 0, gsize - diff);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return tag + gdiff;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return LFS_ERR_NOENT;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
static lfs_stag_t lfs_dir_get(lfs_t *lfs, const lfs_mdir_t *dir,
lfs_tag_t gmask, lfs_tag_t gtag, void *buffer) {
return lfs_dir_getslice(lfs, dir,
gmask, gtag,
0, buffer, lfs_tag_size(gtag));
}
static int lfs_dir_getread(lfs_t *lfs, const lfs_mdir_t *dir,
const lfs_cache_t *pcache, lfs_cache_t *rcache, lfs_size_t hint,
lfs_tag_t gmask, lfs_tag_t gtag,
lfs_off_t off, void *buffer, lfs_size_t size) {
uint8_t *data = buffer;
if (off+size > lfs->cfg->block_size) {
return LFS_ERR_CORRUPT;
}
while (size > 0) {
lfs_size_t diff = size;
if (pcache && pcache->block == 0xfffffffe &&
off < pcache->off + pcache->size) {
if (off >= pcache->off) {
// is already in pcache?
diff = lfs_min(diff, pcache->size - (off-pcache->off));
memcpy(data, &pcache->buffer[off-pcache->off], diff);
data += diff;
off += diff;
size -= diff;
continue;
}
// pcache takes priority
diff = lfs_min(diff, pcache->off-off);
}
if (rcache->block == 0xfffffffe &&
off < rcache->off + rcache->size) {
if (off >= rcache->off) {
// is already in rcache?
diff = lfs_min(diff, rcache->size - (off-rcache->off));
memcpy(data, &rcache->buffer[off-rcache->off], diff);
data += diff;
off += diff;
size -= diff;
continue;
}
// rcache takes priority
diff = lfs_min(diff, rcache->off-off);
}
// load to cache, first condition can no longer fail
rcache->block = 0xfffffffe;
rcache->off = lfs_aligndown(off, lfs->cfg->read_size);
rcache->size = lfs_min(lfs_alignup(off+hint, lfs->cfg->read_size),
lfs->cfg->cache_size);
int err = lfs_dir_getslice(lfs, dir, gmask, gtag,
rcache->off, rcache->buffer, rcache->size);
Fixed issues with reading and caching inline files Kind of a two-fold issue. One, the programming to the middle of inline files was causing the cache to get updated to a half programmed state. While fine, as all programs do occur in order in a block, this is less efficient when writing to inline files as it would cause the inline file to need to be reread even if it fits in the cache. Two, the rereading of the inline file was broken and passed the file's tag all the way to where a user would expect an error. This was easy to fix but adds to the reasons we should have test coverage information. Found by ebinans
2019-05-31 05:58:48 +00:00
if (err < 0) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
return err;
}
}
return 0;
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_traverse_filter(void *p,
lfs_tag_t tag, const void *buffer) {
lfs_tag_t *filtertag = p;
(void)buffer;
// check for redundancy
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
uint32_t mask = LFS_MKTAG(0x7ff, 0x3ff, 0);
if ((mask & tag) == (mask & *filtertag) ||
(mask & tag) == (LFS_MKTAG(LFS_TYPE_DELETE, 0, 0) |
(LFS_MKTAG(0, 0x3ff, 0) & *filtertag))) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
return true;
}
// check if we need to adjust for created/deleted tags
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type1(tag) == LFS_TYPE_SPLICE &&
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_tag_id(tag) <= lfs_tag_id(*filtertag)) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
*filtertag += LFS_MKTAG(0, lfs_tag_splice(tag), 0);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
return false;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
static int lfs_dir_traverse(lfs_t *lfs,
const lfs_mdir_t *dir, lfs_off_t off, lfs_tag_t ptag,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
const struct lfs_mattr *attrs, int attrcount, bool hasseenmove,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_tag_t tmask, lfs_tag_t ttag,
uint16_t begin, uint16_t end, int16_t diff,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
int (*cb)(void *data, lfs_tag_t tag, const void *buffer), void *data) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// iterate over directory and attrs
while (true) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_tag_t tag;
const void *buffer;
struct lfs_diskoff disk;
if (off+lfs_tag_dsize(ptag) < dir->off) {
off += lfs_tag_dsize(ptag);
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(tag),
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
dir->pair[0], off, &tag, sizeof(tag));
if (err) {
return err;
}
tag = (lfs_frombe32(tag) ^ ptag) | 0x80000000;
disk.block = dir->pair[0];
disk.off = off+sizeof(lfs_tag_t);
buffer = &disk;
ptag = tag;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
} else if (attrcount > 0) {
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
tag = attrs[0].tag;
buffer = attrs[0].buffer;
attrs += 1;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
attrcount -= 1;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
} else if (!hasseenmove &&
lfs_gstate_hasmovehere(&lfs->gpending, dir->pair)) {
// Wait, we have pending move? Handle this here (we need to
// or else we risk letting moves fall out of date)
tag = lfs->gpending.tag & LFS_MKTAG(0x7ff, 0x3ff, 0);
buffer = NULL;
hasseenmove = true;
} else {
return 0;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
}
lfs_tag_t mask = LFS_MKTAG(0x7ff, 0, 0);
if ((mask & tmask & tag) != (mask & tmask & ttag)) {
continue;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
}
// do we need to filter? inlining the filtering logic here allows
// for some minor optimizations
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_id(tmask) != 0) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// scan for duplicates and update tag based on creates/deletes
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int filter = lfs_dir_traverse(lfs,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
dir, off, ptag, attrs, attrcount, hasseenmove,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
0, 0, 0, 0, 0,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_dir_traverse_filter, &tag);
if (filter < 0) {
return filter;
}
if (filter) {
continue;
}
// in filter range?
if (!(lfs_tag_id(tag) >= begin && lfs_tag_id(tag) < end)) {
continue;
}
}
// handle special cases for mcu-side operations
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
if (lfs_tag_type3(tag) == LFS_FROM_NOOP) {
// do nothing
} else if (lfs_tag_type3(tag) == LFS_FROM_MOVE) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
uint16_t fromid = lfs_tag_size(tag);
uint16_t toid = lfs_tag_id(tag);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_traverse(lfs,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
buffer, 0, 0xffffffff, NULL, 0, true,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x600, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_STRUCT, 0, 0),
fromid, fromid+1, toid-fromid+diff,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
cb, data);
if (err) {
return err;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(tag) == LFS_FROM_USERATTRS) {
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
for (unsigned i = 0; i < lfs_tag_size(tag); i++) {
const struct lfs_attr *a = buffer;
int err = cb(data, LFS_MKTAG(LFS_TYPE_USERATTR + a[i].type,
lfs_tag_id(tag) + diff, a[i].size), a[i].buffer);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
return err;
}
}
} else {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = cb(data, tag + LFS_MKTAG(0, diff, 0), buffer);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
return err;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
static lfs_stag_t lfs_dir_fetchmatch(lfs_t *lfs,
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
lfs_mdir_t *dir, const lfs_block_t pair[2],
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_tag_t fmask, lfs_tag_t ftag, uint16_t *id,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
int (*cb)(void *data, lfs_tag_t tag, const void *buffer), void *data) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// we can find tag very efficiently during a fetch, since we're already
// scanning the entire directory
lfs_stag_t besttag = -1;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// find the block with the most recent revision
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
uint32_t revs[2] = {0, 0};
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
int r = 0;
for (int i = 0; i < 2; i++) {
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(revs[i]),
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
pair[i], 0, &revs[i], sizeof(revs[i]));
revs[i] = lfs_fromle32(revs[i]);
if (err && err != LFS_ERR_CORRUPT) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return err;
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
if (err != LFS_ERR_CORRUPT &&
lfs_scmp(revs[i], revs[(i+1)%2]) > 0) {
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
r = i;
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->pair[0] = pair[(r+0)%2];
dir->pair[1] = pair[(r+1)%2];
dir->rev = revs[(r+0)%2];
dir->off = 0; // nonzero = found some commits
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// now scan tags to fetch the actual dir and find possible match
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (int i = 0; i < 2; i++) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_off_t off = 0;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_tag_t ptag = 0xffffffff;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
uint16_t tempcount = 0;
lfs_block_t temptail[2] = {0xffffffff, 0xffffffff};
bool tempsplit = false;
lfs_stag_t tempbesttag = besttag;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->rev = lfs_tole32(dir->rev);
uint32_t crc = lfs_crc(0xffffffff, &dir->rev, sizeof(dir->rev));
dir->rev = lfs_fromle32(dir->rev);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
while (true) {
// extract next tag
lfs_tag_t tag;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
off += lfs_tag_dsize(ptag);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, lfs->cfg->block_size,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->pair[0], off, &tag, sizeof(tag));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err == LFS_ERR_CORRUPT) {
// can't continue?
dir->erased = false;
break;
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return err;
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
crc = lfs_crc(crc, &tag, sizeof(tag));
Tweaked tag endianness to catch power-loss after <1 word is written There was an interesting subtlety with the existing layout of tags that could become a problem in the future. Basically, littlefs avoids writing to any region of storage it is not absolutely sure has been erased beforehand. This is a part of limiting the number of assumptions about storage. It's possible a storage technology can't support writes without erases in a way that is undetectable at write time (Maybe changing a bit without an erase decreases the longevity of the information stored on the bit). But the existing layout had a very tiny corner case where this wasn't true. Consider the location of the valid bit in the tag struct: [1|--- 31 ---] ^--- valid bit The responsibility of this bit is to indicate if an attempt has been made to write the following commit. If it is not set (the specific value is dependent on a previous read and identified by the preceeding commit), the assumption is that it is safe to write to the next region because it has been erased previously. If it is set, we check if the next commit is valid, if it isn't (because of CRC failure, likely due to power-loss), we discard the commit. But because an attempt has been made to write to that storage, we must then do a compaction to move to the other block in the metadata-pair. This plan looks good on paper, but what does it look like on storage? The problem is that words in littlefs are in little-endian. So on storage the tag actually looks like this: [- 8 -|- 8 -|- 8 -|1|- 7 -] ^-- valid bit This means that we don't actually set the valid bit before writing the tag! We write the lower bytes first. If we lose power, we may have written 3 bytes without this fact being detectable. We could restructure the tag structure to store the valid bit lower, however because none of the fields are 7 bits, this would make the extraction more costly, and we then lose the ability to check this valid bit with a sign comparison. The simple solution is to just store the tag in big-endian. A small benefit is that this will actually have a negative code cost on big-endian machines. This mixture of endiannesses is frustrating, however it is a pragmatic solution with only a 20-byte code size cost.
2018-10-22 21:42:30 +00:00
tag = lfs_frombe32(tag) ^ ptag;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// next commit not yet programmed or we're not in valid range
if (!lfs_tag_isvalid(tag) ||
off + lfs_tag_dsize(tag) > lfs->cfg->block_size) {
Added better handling of large program sizes (> 1024) The issue here is how commits handle padding to the nearest program size. This is done by exploiting the size field of the LFS_TYPE_CRC tag that completes the commit. Unfortunately, during developement, the size field shrank in size to make room for more type information, limiting the size field to 1024. Normally this isn't a problem, as very rarely do program sizes exceed 1024 bytes. However, using a simulated block device, user earlephilhower found that exceeding 1024 caused littlefs to crash. To make this corner case behave in a more user friendly manner, I've modified this situtation to treat >1024 program sizes as small commits that don't match the prog size. As a part of this, littlefs also needed to understand that non-matching commits indicate an "unerased" dir block, which would be needed for portability (something which notably lacks testing). This raises the question of if the tag size field size needs to be reconsidered, but to change that at this point would need a new major version. found by earlephilhower
2019-04-09 21:06:43 +00:00
dir->erased = (lfs_tag_type1(ptag) == LFS_TYPE_CRC &&
dir->off % lfs->cfg->prog_size == 0);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
break;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
ptag = tag;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type1(tag) == LFS_TYPE_CRC) {
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// check the crc attr
uint32_t dcrc;
err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, lfs->cfg->block_size,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->pair[0], off+sizeof(tag), &dcrc, sizeof(dcrc));
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
dir->erased = false;
break;
}
return err;
}
dcrc = lfs_fromle32(dcrc);
if (crc != dcrc) {
dir->erased = false;
break;
}
// reset the next bit if we need to
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
ptag ^= (lfs_tag_chunk(tag) & 1U) << 31;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// toss our crc into the filesystem seed for
// pseudorandom numbers
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->seed ^= crc;
// update with what's found so far
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
besttag = tempbesttag;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
dir->off = off + lfs_tag_dsize(tag);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->etag = ptag;
dir->count = tempcount;
dir->tail[0] = temptail[0];
dir->tail[1] = temptail[1];
dir->split = tempsplit;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// reset crc
crc = 0xffffffff;
continue;
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// crc the entry first, hopefully leaving it in the cache
for (lfs_off_t j = sizeof(tag); j < lfs_tag_dsize(tag); j++) {
uint8_t dat;
err = lfs_bd_read(lfs,
NULL, &lfs->rcache, lfs->cfg->block_size,
dir->pair[0], off+j, &dat, 1);
if (err) {
if (err == LFS_ERR_CORRUPT) {
dir->erased = false;
break;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return err;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
crc = lfs_crc(crc, &dat, 1);
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// directory modification tags?
if (lfs_tag_type1(tag) == LFS_TYPE_NAME) {
// increase count of files if necessary
if (lfs_tag_id(tag) >= tempcount) {
tempcount = lfs_tag_id(tag) + 1;
}
} else if (lfs_tag_type1(tag) == LFS_TYPE_SPLICE) {
tempcount += lfs_tag_splice(tag);
if (tag == (LFS_MKTAG(LFS_TYPE_DELETE, 0, 0) |
(LFS_MKTAG(0, 0x3ff, 0) & tempbesttag))) {
tempbesttag |= 0x80000000;
} else if (tempbesttag != -1 &&
lfs_tag_id(tag) <= lfs_tag_id(tempbesttag)) {
tempbesttag += LFS_MKTAG(0, lfs_tag_splice(tag), 0);
}
} else if (lfs_tag_type1(tag) == LFS_TYPE_TAIL) {
tempsplit = (lfs_tag_chunk(tag) & 1);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, lfs->cfg->block_size,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
dir->pair[0], off+sizeof(tag), &temptail, 8);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
dir->erased = false;
break;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_pair_fromle32(temptail);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// found a match for our fetcher?
if ((fmask & tag) == (fmask & ftag)) {
int res = cb(data, tag, &(struct lfs_diskoff){
dir->pair[0], off+sizeof(tag)});
if (res < 0) {
if (res == LFS_ERR_CORRUPT) {
dir->erased = false;
break;
}
return res;
}
if (res == LFS_CMP_EQ) {
// found a match
tempbesttag = tag;
} else if (res == LFS_CMP_GT &&
lfs_tag_id(tag) <= lfs_tag_id(tempbesttag)) {
// found a greater match, keep track to keep things sorted
tempbesttag = tag | 0x80000000;
}
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
// consider what we have good enough
if (dir->off > 0) {
// synthetic move
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (lfs_gstate_hasmovehere(&lfs->gstate, dir->pair)) {
if (lfs_tag_id(lfs->gstate.tag) == lfs_tag_id(besttag)) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
besttag |= 0x80000000;
} else if (besttag != -1 &&
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_tag_id(lfs->gstate.tag) < lfs_tag_id(besttag)) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
besttag -= LFS_MKTAG(0, 1, 0);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// found tag? or found best id?
if (id) {
*id = lfs_min(lfs_tag_id(besttag), dir->count);
}
if (lfs_tag_isvalid(besttag)) {
return besttag;
} else if (lfs_tag_id(besttag) < dir->count) {
return LFS_ERR_NOENT;
} else {
return 0;
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// failed, try the other block?
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_pair_swap(dir->pair);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
dir->rev = revs[(r+1)%2];
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
LFS_ERROR("Corrupted dir pair at %"PRIu32" %"PRIu32,
dir->pair[0], dir->pair[1]);
return LFS_ERR_CORRUPT;
}
static int lfs_dir_fetch(lfs_t *lfs,
lfs_mdir_t *dir, const lfs_block_t pair[2]) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// note, mask=-1, tag=0 can never match a tag since this
// pattern has the invalid bit set
return lfs_dir_fetchmatch(lfs, dir, pair, -1, 0, NULL, NULL, NULL);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static int lfs_dir_getgstate(lfs_t *lfs, const lfs_mdir_t *dir,
struct lfs_gstate *gstate) {
struct lfs_gstate temp;
lfs_stag_t res = lfs_dir_get(lfs, dir, LFS_MKTAG(0x7ff, 0, 0),
LFS_MKTAG(LFS_TYPE_MOVESTATE, 0, sizeof(temp)), &temp);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (res < 0 && res != LFS_ERR_NOENT) {
return res;
}
if (res != LFS_ERR_NOENT) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// xor together to find resulting gstate
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs_gstate_fromle32(&temp);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_gstate_xor(gstate, &temp);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
return 0;
}
static int lfs_dir_getinfo(lfs_t *lfs, lfs_mdir_t *dir,
uint16_t id, struct lfs_info *info) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (id == 0x3ff) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// special case for root
strcpy(info->name, "/");
info->type = LFS_TYPE_DIR;
return 0;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t tag = lfs_dir_get(lfs, dir, LFS_MKTAG(0x780, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_NAME, id, lfs->name_max+1), info->name);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (tag < 0) {
return tag;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
info->type = lfs_tag_type3(tag);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
struct lfs_ctz ctz;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
tag = lfs_dir_get(lfs, dir, LFS_MKTAG(0x700, 0x3ff, 0),
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, id, sizeof(ctz)), &ctz);
if (tag < 0) {
return tag;
}
lfs_ctz_fromle32(&ctz);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_CTZSTRUCT) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
info->size = ctz.size;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(tag) == LFS_TYPE_INLINESTRUCT) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
info->size = lfs_tag_size(tag);
}
return 0;
}
struct lfs_dir_find_match {
lfs_t *lfs;
const void *name;
lfs_size_t size;
};
static int lfs_dir_find_match(void *data,
lfs_tag_t tag, const void *buffer) {
struct lfs_dir_find_match *name = data;
lfs_t *lfs = name->lfs;
const struct lfs_diskoff *disk = buffer;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// compare with disk
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_size_t diff = lfs_min(name->size, lfs_tag_size(tag));
int res = lfs_bd_cmp(lfs,
NULL, &lfs->rcache, diff,
disk->block, disk->off, name->name, diff);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (res != LFS_CMP_EQ) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return res;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// only equal if our size is still the same
if (name->size != lfs_tag_size(tag)) {
return (name->size < lfs_tag_size(tag)) ? LFS_CMP_LT : LFS_CMP_GT;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// found a match!
return LFS_CMP_EQ;
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
static int lfs_dir_find(lfs_t *lfs, lfs_mdir_t *dir,
const char **path, uint16_t *id) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// we reduce path to a single name if we can find it
const char *name = *path;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (id) {
*id = 0x3ff;
}
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// default to root dir
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t tag = LFS_MKTAG(LFS_TYPE_DIR, 0x3ff, 0);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
dir->tail[0] = lfs->root[0];
dir->tail[1] = lfs->root[1];
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
while (true) {
nextname:
// skip slashes
name += strspn(name, "/");
lfs_size_t namelen = strcspn(name, "/");
// skip '.' and root '..'
if ((namelen == 1 && memcmp(name, ".", 1) == 0) ||
(namelen == 2 && memcmp(name, "..", 2) == 0)) {
name += namelen;
goto nextname;
}
// skip if matched by '..' in name
const char *suffix = name + namelen;
lfs_size_t sufflen;
int depth = 1;
while (true) {
suffix += strspn(suffix, "/");
sufflen = strcspn(suffix, "/");
if (sufflen == 0) {
break;
}
if (sufflen == 2 && memcmp(suffix, "..", 2) == 0) {
depth -= 1;
if (depth == 0) {
name = suffix + sufflen;
goto nextname;
}
} else {
depth += 1;
}
suffix += sufflen;
}
// found path
if (name[0] == '\0') {
return tag;
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
// update what we've found so far
*path = name;
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// only continue if we hit a directory
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) != LFS_TYPE_DIR) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
return LFS_ERR_NOTDIR;
}
// grab the entry data
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_id(tag) != 0x3ff) {
lfs_stag_t res = lfs_dir_get(lfs, dir, LFS_MKTAG(0x700, 0x3ff, 0),
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tag_id(tag), 8), dir->tail);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (res < 0) {
return res;
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_pair_fromle32(dir->tail);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
// find entry matching name
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
while (true) {
tag = lfs_dir_fetchmatch(lfs, dir, dir->tail,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x780, 0, 0),
LFS_MKTAG(LFS_TYPE_NAME, 0, namelen),
// are we last name?
(strchr(name, '/') == NULL) ? id : NULL,
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_dir_find_match, &(struct lfs_dir_find_match){
lfs, name, namelen});
if (tag < 0) {
return tag;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (tag) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
break;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (!dir->split) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return LFS_ERR_NOENT;
}
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
}
// to next name
name += namelen;
}
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// commit logic
struct lfs_commit {
lfs_block_t block;
lfs_off_t off;
lfs_tag_t ptag;
uint32_t crc;
lfs_off_t begin;
lfs_off_t end;
};
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_commitprog(lfs_t *lfs, struct lfs_commit *commit,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
const void *buffer, lfs_size_t size) {
int err = lfs_bd_prog(lfs,
&lfs->pcache, &lfs->rcache, false,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
commit->block, commit->off ,
(const uint8_t*)buffer, size);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
return err;
}
commit->crc = lfs_crc(commit->crc, buffer, size);
commit->off += size;
return 0;
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_commitattr(lfs_t *lfs, struct lfs_commit *commit,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_tag_t tag, const void *buffer) {
// check if we fit
lfs_size_t dsize = lfs_tag_dsize(tag);
if (commit->off + dsize > commit->end) {
return LFS_ERR_NOSPC;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
// write out tag
Tweaked tag endianness to catch power-loss after <1 word is written There was an interesting subtlety with the existing layout of tags that could become a problem in the future. Basically, littlefs avoids writing to any region of storage it is not absolutely sure has been erased beforehand. This is a part of limiting the number of assumptions about storage. It's possible a storage technology can't support writes without erases in a way that is undetectable at write time (Maybe changing a bit without an erase decreases the longevity of the information stored on the bit). But the existing layout had a very tiny corner case where this wasn't true. Consider the location of the valid bit in the tag struct: [1|--- 31 ---] ^--- valid bit The responsibility of this bit is to indicate if an attempt has been made to write the following commit. If it is not set (the specific value is dependent on a previous read and identified by the preceeding commit), the assumption is that it is safe to write to the next region because it has been erased previously. If it is set, we check if the next commit is valid, if it isn't (because of CRC failure, likely due to power-loss), we discard the commit. But because an attempt has been made to write to that storage, we must then do a compaction to move to the other block in the metadata-pair. This plan looks good on paper, but what does it look like on storage? The problem is that words in littlefs are in little-endian. So on storage the tag actually looks like this: [- 8 -|- 8 -|- 8 -|1|- 7 -] ^-- valid bit This means that we don't actually set the valid bit before writing the tag! We write the lower bytes first. If we lose power, we may have written 3 bytes without this fact being detectable. We could restructure the tag structure to store the valid bit lower, however because none of the fields are 7 bits, this would make the extraction more costly, and we then lose the ability to check this valid bit with a sign comparison. The simple solution is to just store the tag in big-endian. A small benefit is that this will actually have a negative code cost on big-endian machines. This mixture of endiannesses is frustrating, however it is a pragmatic solution with only a 20-byte code size cost.
2018-10-22 21:42:30 +00:00
lfs_tag_t ntag = lfs_tobe32((tag & 0x7fffffff) ^ commit->ptag);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
int err = lfs_dir_commitprog(lfs, commit, &ntag, sizeof(ntag));
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
return err;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (!(tag & 0x80000000)) {
// from memory
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitprog(lfs, commit, buffer, dsize-sizeof(tag));
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
return err;
}
} else {
// from disk
const struct lfs_diskoff *disk = buffer;
for (lfs_off_t i = 0; i < dsize-sizeof(tag); i++) {
// rely on caching to make this efficient
uint8_t dat;
err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, dsize-sizeof(tag)-i,
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
disk->block, disk->off+i, &dat, 1);
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
if (err) {
return err;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitprog(lfs, commit, &dat, 1);
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
if (err) {
return err;
Added support for entry insertion Currently unused, the insertion of new file entries in arbitrary locations in a metadata-pair is very easy to add into the existing metadata logging. The only tricky things: 1. Name tags must strictly precede any tags related to a file. We can pull this off during a compact, but must make two passes. One for the name tag, one for the file. Though a benefit of this is that now our scans during moves can exit early upon finding the name tag. 1. We need to handle name tags appearing out of order. This makes name tags symmetric to deletes, although it doesn't seem like we can leverage this fact very well. Note this also means we need to make the superblock tag a type of name tag.
2018-09-09 14:01:06 +00:00
}
}
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
}
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
commit->ptag = tag & 0x7fffffff;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
return 0;
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_commitcrc(lfs_t *lfs, struct lfs_commit *commit) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// align to program units
lfs_off_t off = lfs_alignup(commit->off + 2*sizeof(uint32_t),
lfs->cfg->prog_size);
// read erased state from next program unit
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_tag_t tag;
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(tag),
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
commit->block, off, &tag, sizeof(tag));
if (err && err != LFS_ERR_CORRUPT) {
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// build crc tag
Tweaked tag endianness to catch power-loss after <1 word is written There was an interesting subtlety with the existing layout of tags that could become a problem in the future. Basically, littlefs avoids writing to any region of storage it is not absolutely sure has been erased beforehand. This is a part of limiting the number of assumptions about storage. It's possible a storage technology can't support writes without erases in a way that is undetectable at write time (Maybe changing a bit without an erase decreases the longevity of the information stored on the bit). But the existing layout had a very tiny corner case where this wasn't true. Consider the location of the valid bit in the tag struct: [1|--- 31 ---] ^--- valid bit The responsibility of this bit is to indicate if an attempt has been made to write the following commit. If it is not set (the specific value is dependent on a previous read and identified by the preceeding commit), the assumption is that it is safe to write to the next region because it has been erased previously. If it is set, we check if the next commit is valid, if it isn't (because of CRC failure, likely due to power-loss), we discard the commit. But because an attempt has been made to write to that storage, we must then do a compaction to move to the other block in the metadata-pair. This plan looks good on paper, but what does it look like on storage? The problem is that words in littlefs are in little-endian. So on storage the tag actually looks like this: [- 8 -|- 8 -|- 8 -|1|- 7 -] ^-- valid bit This means that we don't actually set the valid bit before writing the tag! We write the lower bytes first. If we lose power, we may have written 3 bytes without this fact being detectable. We could restructure the tag structure to store the valid bit lower, however because none of the fields are 7 bits, this would make the extraction more costly, and we then lose the ability to check this valid bit with a sign comparison. The simple solution is to just store the tag in big-endian. A small benefit is that this will actually have a negative code cost on big-endian machines. This mixture of endiannesses is frustrating, however it is a pragmatic solution with only a 20-byte code size cost.
2018-10-22 21:42:30 +00:00
bool reset = ~lfs_frombe32(tag) >> 31;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
tag = LFS_MKTAG(LFS_TYPE_CRC + reset, 0x3ff,
off - (commit->off+sizeof(lfs_tag_t)));
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// write out crc
uint32_t footer[2];
Tweaked tag endianness to catch power-loss after <1 word is written There was an interesting subtlety with the existing layout of tags that could become a problem in the future. Basically, littlefs avoids writing to any region of storage it is not absolutely sure has been erased beforehand. This is a part of limiting the number of assumptions about storage. It's possible a storage technology can't support writes without erases in a way that is undetectable at write time (Maybe changing a bit without an erase decreases the longevity of the information stored on the bit). But the existing layout had a very tiny corner case where this wasn't true. Consider the location of the valid bit in the tag struct: [1|--- 31 ---] ^--- valid bit The responsibility of this bit is to indicate if an attempt has been made to write the following commit. If it is not set (the specific value is dependent on a previous read and identified by the preceeding commit), the assumption is that it is safe to write to the next region because it has been erased previously. If it is set, we check if the next commit is valid, if it isn't (because of CRC failure, likely due to power-loss), we discard the commit. But because an attempt has been made to write to that storage, we must then do a compaction to move to the other block in the metadata-pair. This plan looks good on paper, but what does it look like on storage? The problem is that words in littlefs are in little-endian. So on storage the tag actually looks like this: [- 8 -|- 8 -|- 8 -|1|- 7 -] ^-- valid bit This means that we don't actually set the valid bit before writing the tag! We write the lower bytes first. If we lose power, we may have written 3 bytes without this fact being detectable. We could restructure the tag structure to store the valid bit lower, however because none of the fields are 7 bits, this would make the extraction more costly, and we then lose the ability to check this valid bit with a sign comparison. The simple solution is to just store the tag in big-endian. A small benefit is that this will actually have a negative code cost on big-endian machines. This mixture of endiannesses is frustrating, however it is a pragmatic solution with only a 20-byte code size cost.
2018-10-22 21:42:30 +00:00
footer[0] = lfs_tobe32(tag ^ commit->ptag);
Changed lfs_crc to match more common API In looking at the common CRC APIs out there, this seemed the most common. At least more common than the current modified-in-place pointer API. It also seems to have a slightly better code footprint. I'm blaming pointer optimization issues. One downside is that lfs_crc can't report errors, however it was already assumed that lfs_crc can not error.
2018-08-07 19:59:44 +00:00
commit->crc = lfs_crc(commit->crc, &footer[0], sizeof(footer[0]));
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
footer[1] = lfs_tole32(commit->crc);
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
err = lfs_bd_prog(lfs,
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
&lfs->pcache, &lfs->rcache, false,
commit->block, commit->off, &footer, sizeof(footer));
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
return err;
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
commit->off += sizeof(tag)+lfs_tag_size(tag);
Modified valid bit to provide an early check on all tags The valid bit present in tags is a requirement to properly detect the end of commits in metadata logs. The way it works is that the CRC entry is allowed to specify what is needed from the next tag's valid bit. If it's incorrect, we've reached the end of the commit. We then set the valid bit to indicate when we tried to program a new commit. If we lose power, this commit will still be thrown out by a bad checksum. However, the valid bit is unused outside of the CRC entry. Here we turn on the valid bit for all tags, which means we have a decent chance of exiting early if we hit a half-written commit. We still need to guarantee detection of the valid bit on commits following the CRC entry, so we allow the CRC entry to flip the expected valid bit. The only tricky part is what valid bit we expect by default, since this is used on the first commit on a metadata log. Here we default to a 1, which gives us the fastest exit on blocks that erase to 0. This is because blocks that erase to 1s will implicitly flip the valid bit of the next tag, allowing us to exit on the next tag. If we defaulted to 0, we could exit faster on disks that erase to 1, but would need to scan the entire block on disks that erase to 0 before we realize a CRC commit is never coming.
2018-08-24 22:45:45 +00:00
commit->ptag = tag ^ (reset << 31);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// flush buffers
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
err = lfs_bd_sync(lfs, &lfs->pcache, &lfs->rcache, false);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
return err;
}
// successful commit, check checksum to make sure
uint32_t crc = 0xffffffff;
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
lfs_size_t size = commit->off - lfs_tag_size(tag) - commit->begin;
for (lfs_off_t i = 0; i < size; i++) {
// leave it up to caching to make this efficient
uint8_t dat;
err = lfs_bd_read(lfs,
NULL, &lfs->rcache, size-i,
commit->block, commit->begin+i, &dat, 1);
if (err) {
return err;
}
crc = lfs_crc(crc, &dat, 1);
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
return err;
}
if (crc != commit->crc) {
return LFS_ERR_CORRUPT;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
return 0;
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
static int lfs_dir_alloc(lfs_t *lfs, lfs_mdir_t *dir) {
Added root entry and expanding superblocks Expanding superblocks has been on my wishlist for a while. The basic idea is that instead of maintaining a fixed offset blocks {0, 1} to the the root directory (1 pointer), we maintain a dynamically sized linked-list of superblocks that point to the actual root. If the number of writes to the root exceeds some value, we increase the size of the superblock linked-list. This can leverage existing metadata-pair operations. The revision count for metadata-pairs provides some knowledge on how much wear we've put on the superblock, and the threaded linked-list can also be reused for this purpose. This means superblock expansion is both optional and cheap to implement. Expanding superblocks helps both extremely small and extremely large filesystem (extreme being relative of course). On the small end, we can actually collapse the superblock into the root directory and drop the hard requirement of 4-blocks for the superblock. On the large end, our superblock will now last longer than the rest of the filesystem. Each time we expand, the number of cycles until the superblock dies is increased by a power. Before we were stuck with this layout: level cycles limit layout 1 E^2 390 MiB s0 -> root Now we expand every time a fixed offset is exceeded: level cycles limit layout 0 E 4 KiB s0+root 1 E^2 390 MiB s0 -> root 2 E^3 37 TiB s0 -> s1 -> root 3 E^4 3.6 EiB s0 -> s1 -> s2 -> root ... Where the cycles are the number of cycles before death, and the limit is the worst-case size a filesystem where early superblock death becomes a concern (all writes to root using this formula: E^|s| = E*B, E = erase cycles = 100000, B = block count, assuming 4096 byte blocks). Note we can also store copies of the superblock entry on the expanded superblocks. This may help filesystem recover tools in the future.
2018-08-06 18:30:51 +00:00
// allocate pair of dir blocks (backwards, so we write block 1 first)
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
for (int i = 0; i < 2; i++) {
int err = lfs_alloc(lfs, &dir->pair[(i+1)%2]);
if (err) {
return err;
}
}
// rather than clobbering one of the blocks we just pretend
// the revision may be valid
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int err = lfs_bd_read(lfs,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
NULL, &lfs->rcache, sizeof(dir->rev),
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
dir->pair[0], 0, &dir->rev, sizeof(dir->rev));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->rev = lfs_fromle32(dir->rev);
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
if (err && err != LFS_ERR_CORRUPT) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return err;
}
Fixed corner case in block_cycles eviction logic The problem was when we allocate a dir-pair, it's possible for the revision count to immediately overflow and the dir-pair be evicted and returned to the unused blocks without being written even once. In the case that block_cycles = 1, this made it impossible to ever create a dir-pair, even in lfs_format. I've also added a bit of logic to lfs_dir_alloc that will prevent any immediate evictions because of the revision count. found by TheLoneWolfling
2019-01-30 03:53:56 +00:00
// make sure we don't immediately evict
dir->rev += dir->rev & 1;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// set defaults
dir->off = sizeof(dir->rev);
Modified valid bit to provide an early check on all tags The valid bit present in tags is a requirement to properly detect the end of commits in metadata logs. The way it works is that the CRC entry is allowed to specify what is needed from the next tag's valid bit. If it's incorrect, we've reached the end of the commit. We then set the valid bit to indicate when we tried to program a new commit. If we lose power, this commit will still be thrown out by a bad checksum. However, the valid bit is unused outside of the CRC entry. Here we turn on the valid bit for all tags, which means we have a decent chance of exiting early if we hit a half-written commit. We still need to guarantee detection of the valid bit on commits following the CRC entry, so we allow the CRC entry to flip the expected valid bit. The only tricky part is what valid bit we expect by default, since this is used on the first commit on a metadata log. Here we default to a 1, which gives us the fastest exit on blocks that erase to 0. This is because blocks that erase to 1s will implicitly flip the valid bit of the next tag, allowing us to exit on the next tag. If we defaulted to 0, we could exit faster on disks that erase to 1, but would need to scan the entire block on disks that erase to 0 before we realize a CRC commit is never coming.
2018-08-24 22:45:45 +00:00
dir->etag = 0xffffffff;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->count = 0;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
dir->tail[0] = 0xffffffff;
dir->tail[1] = 0xffffffff;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->erased = false;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
dir->split = false;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// don't write out yet, let caller take care of that
return 0;
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static int lfs_dir_drop(lfs_t *lfs, lfs_mdir_t *dir, lfs_mdir_t *tail) {
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
// steal state
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
int err = lfs_dir_getgstate(lfs, tail, &lfs->gdelta);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
return err;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// steal tail
lfs_pair_tole32(tail->tail);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, dir, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_TAIL + tail->split, 0x3ff, 8), tail->tail}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(tail->tail);
if (err) {
return err;
}
return 0;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
static int lfs_dir_split(lfs_t *lfs,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_mdir_t *dir, const struct lfs_mattr *attrs, int attrcount,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_mdir_t *source, uint16_t split, uint16_t end) {
// create tail directory
lfs_mdir_t tail;
int err = lfs_dir_alloc(lfs, &tail);
if (err) {
return err;
}
tail.split = dir->split;
tail.tail[0] = dir->tail[0];
tail.tail[1] = dir->tail[1];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
err = lfs_dir_compact(lfs, &tail, attrs, attrcount, source, split, end);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
return err;
}
dir->tail[0] = tail.pair[0];
dir->tail[1] = tail.pair[1];
dir->split = true;
// update root if needed
if (lfs_pair_cmp(dir->pair, lfs->root) == 0 && split == 0) {
lfs->root[0] = tail.pair[0];
lfs->root[1] = tail.pair[1];
}
return 0;
}
static int lfs_dir_commit_size(void *p, lfs_tag_t tag, const void *buffer) {
lfs_size_t *size = p;
(void)buffer;
*size += lfs_tag_dsize(tag);
return 0;
}
struct lfs_dir_commit_commit {
lfs_t *lfs;
struct lfs_commit *commit;
};
static int lfs_dir_commit_commit(void *p, lfs_tag_t tag, const void *buffer) {
struct lfs_dir_commit_commit *commit = p;
return lfs_dir_commitattr(commit->lfs, commit->commit, tag, buffer);
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_dir_compact(lfs_t *lfs,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_mdir_t *dir, const struct lfs_mattr *attrs, int attrcount,
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *source, uint16_t begin, uint16_t end) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// save some state in case block is bad
const lfs_block_t oldpair[2] = {dir->pair[1], dir->pair[0]};
bool relocated = false;
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
bool exhausted = false;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Fixed issue with long names causing unbounded recursion This was caused by any commit containing entries large enough to _always_ force a compaction. This would cause littlefs to think that it would need to split infinitely because there was no base case. The fix here is pretty simple: treat any commit with only a single entry as unsplittable. This forces littlefs to first try overcompacting (fitting more in a block than what has optimal runtime), and then failing that return LFS_ERR_NOSPC for higher layers to handle. found by TheLoneWolfling
2019-01-31 20:54:47 +00:00
// should we split?
while (end - begin > 1) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// find size
lfs_size_t size = 0;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_traverse(lfs,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
source, 0, 0xffffffff, attrs, attrcount, false,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x400, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_NAME, 0, 0),
begin, end, -begin,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_dir_commit_size, &size);
if (err) {
return err;
}
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// space is complicated, we need room for tail, crc, gstate,
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
// cleanup delete, and we cap at half a block to give room
Fixed issue with long names causing unbounded recursion This was caused by any commit containing entries large enough to _always_ force a compaction. This would cause littlefs to think that it would need to split infinitely because there was no base case. The fix here is pretty simple: treat any commit with only a single entry as unsplittable. This forces littlefs to first try overcompacting (fitting more in a block than what has optimal runtime), and then failing that return LFS_ERR_NOSPC for higher layers to handle. found by TheLoneWolfling
2019-01-31 20:54:47 +00:00
// for metadata updates.
Artificially limited number of file ids per metadata block This is an expirement to determine which field in the tag structure is the most critical: tag id or tag size. This came from looking at NAND storage and discussions around behaviour of large prog_sizes. Initial exploration indicates that prog_sizes around 2KiB are not _that_ uncommon, and the 1KiB limitation is surprising. It's possible to increase the lfs_tag size to 12-bits (4096), but at the cost of only 8-bit ids (256). [---- 32 ----] a [1|-3-|-- 8 --|-- 10 --|-- 10 --] b [1|-3-|-- 8 --|-- 8 --|-- 12 --] This requires more investigation, but in order to allow us to change the tag sizes with minimal impact I've artificially limited the number of file ids to 0xfe (255) different file ids per metadata pair. If 12-bit lengths turn out to be a bad idea, we can remove the artificial limit without backwards incompatible changes. To avoid breaking users already on v2-alpha, this change will refuse _creating_ file ids > 255, but should read file ids > 255 without issues.
2019-04-09 23:56:53 +00:00
if (end - begin < 0xff &&
size <= lfs_min(lfs->cfg->block_size - 36,
lfs_alignup(lfs->cfg->block_size/2,
lfs->cfg->prog_size))) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
break;
}
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// can't fit, need to split, we should really be finding the
// largest size that fits with a small binary search, but right now
// it's not worth the code size
uint16_t split = (end - begin) / 2;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
err = lfs_dir_split(lfs, dir, attrs, attrcount,
source, begin+split, end);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
// if we fail to split, we may be able to overcompact, unless
// we're too big for even the full block, in which case our
// only option is to error
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (err == LFS_ERR_NOSPC && size <= lfs->cfg->block_size - 36) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
break;
}
return err;
}
end = begin + split;
}
// increment revision count
dir->rev += 1;
Fixed corner case in block_cycles eviction logic The problem was when we allocate a dir-pair, it's possible for the revision count to immediately overflow and the dir-pair be evicted and returned to the unused blocks without being written even once. In the case that block_cycles = 1, this made it impossible to ever create a dir-pair, even in lfs_format. I've also added a bit of logic to lfs_dir_alloc that will prevent any immediate evictions because of the revision count. found by TheLoneWolfling
2019-01-30 03:53:56 +00:00
if (lfs->cfg->block_cycles &&
(dir->rev % (lfs->cfg->block_cycles+1) == 0)) {
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (lfs_pair_cmp(dir->pair, (const lfs_block_t[2]){0, 1}) == 0) {
// oh no! we're writing too much to the superblock,
// should we expand?
lfs_ssize_t res = lfs_fs_size(lfs);
if (res < 0) {
return res;
}
// do we have extra space? littlefs can't reclaim this space
// by itself, so expand cautiously
if ((lfs_size_t)res < lfs->cfg->block_count/2) {
LFS_DEBUG("Expanding superblock at rev %"PRIu32, dir->rev);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_split(lfs, dir, attrs, attrcount,
source, begin, end);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err && err != LFS_ERR_NOSPC) {
return err;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// welp, we tried, if we ran out of space there's not much
// we can do, we'll error later if we've become frozen
if (!err) {
end = begin;
}
}
} else {
// we're writing too much, time to relocate
exhausted = true;
goto relocate;
}
}
// begin loop to commit compaction to blocks until a compact sticks
while (true) {
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// There's nothing special about our global delta, so feed it into
// our local global delta
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
int err = lfs_dir_getgstate(lfs, dir, &lfs->gdelta);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
if (err) {
return err;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// setup commit state
struct lfs_commit commit = {
.block = dir->pair[1],
.off = 0,
.ptag = 0xffffffff,
.crc = 0xffffffff,
.begin = 0,
.end = lfs->cfg->block_size - 8,
};
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
// erase block to write to
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
err = lfs_bd_erase(lfs, dir->pair[1]);
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
// write out header
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
dir->rev = lfs_tole32(dir->rev);
err = lfs_dir_commitprog(lfs, &commit,
&dir->rev, sizeof(dir->rev));
dir->rev = lfs_fromle32(dir->rev);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
// traverse the directory, this time writing out all unique tags
err = lfs_dir_traverse(lfs,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
source, 0, 0xffffffff, attrs, attrcount, false,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x400, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_NAME, 0, 0),
begin, end, -begin,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_dir_commit_commit, &(struct lfs_dir_commit_commit){
lfs, &commit});
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
return err;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// commit tail, which may be new after last size check
if (!lfs_pair_isnull(dir->tail)) {
lfs_pair_tole32(dir->tail);
err = lfs_dir_commitattr(lfs, &commit,
LFS_MKTAG(LFS_TYPE_TAIL + dir->split, 0x3ff, 8),
dir->tail);
lfs_pair_fromle32(dir->tail);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Modified lfs_dir_compact to avoid redundant erases during split The commit machine in littlefs has three stages: commit, compact, and then split. First we try to append our commit to the metadata log, if that fails we try to compact the metadata log to remove duplicates and make room for the commit, if that still fails we split the metadata into two metadata-pairs and try again. Each stage is less efficient but also less frequent. However, in the case that we're filling up a directory with new files, such as the bootstrap process in setting up a new system, we must pass through all three stages rather quickly in order to get enough metadata-pairs to hold all of our files. This means we'll compact, split, and then need to compact again. This creates more erases than is needed in the optimal case, which can be a big cost on disks with an expensive erase operation. In theory, we can actually avoid this redundant erase by reusing the data we wrote out in the first attempt to compact. In practice, this trick is very complicated to pull off. 1. We may need to cache a half-completed program while we write out the new metadata-pair. We need to write out the second pair first in order to get our new tail before we complete our first metadata-pair. This requires two pcaches, which we don't have The solution here is to just drop our cache and reconstruct what if would have been. This needs to be perfect down to the byte level because we don't have knowledge of where our cache lines are. 2. We may have written out entries that are then moved to the new metadata-pair. The solution here isn't pretty but it works, we just add a delete tag for any entry that was moved over. In the end the solution ends up a bit hacky, with different layers poked through the commit logic in order to manage writes at the byte level from where we manage splits. But it works fairly well and saves erases.
2018-08-21 02:45:11 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (!relocated && !lfs_gstate_iszero(&lfs->gdelta)) {
// commit any globals, unless we're relocating,
// in which case our parent will steal our globals
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs_gstate_tole32(&lfs->gdelta);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitattr(lfs, &commit,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
LFS_MKTAG(LFS_TYPE_MOVESTATE, 0x3ff,
sizeof(lfs->gdelta)), &lfs->gdelta);
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs_gstate_fromle32(&lfs->gdelta);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
}
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitcrc(lfs, &commit);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
// successful compaction, swap dir pair to indicate most recent
lfs_pair_swap(dir->pair);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
dir->count = end - begin;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
dir->off = commit.off;
dir->etag = commit.ptag;
Added better handling of large program sizes (> 1024) The issue here is how commits handle padding to the nearest program size. This is done by exploiting the size field of the LFS_TYPE_CRC tag that completes the commit. Unfortunately, during developement, the size field shrank in size to make room for more type information, limiting the size field to 1024. Normally this isn't a problem, as very rarely do program sizes exceed 1024 bytes. However, using a simulated block device, user earlephilhower found that exceeding 1024 caused littlefs to crash. To make this corner case behave in a more user friendly manner, I've modified this situtation to treat >1024 program sizes as small commits that don't match the prog size. As a part of this, littlefs also needed to understand that non-matching commits indicate an "unerased" dir block, which would be needed for portability (something which notably lacks testing). This raises the question of if the tag size field size needs to be reconsidered, but to change that at this point would need a new major version. found by earlephilhower
2019-04-09 21:06:43 +00:00
dir->erased = (dir->off % lfs->cfg->prog_size == 0);
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
// note we able to have already handled move here
if (lfs_gstate_hasmovehere(&lfs->gpending, dir->pair)) {
lfs_gstate_xormove(&lfs->gpending,
&lfs->gpending, 0x3ff, NULL);
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
break;
relocate:
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// commit was corrupted, drop caches and prepare to relocate block
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
relocated = true;
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, &lfs->pcache);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (!exhausted) {
LFS_DEBUG("Bad block at %"PRIu32, dir->pair[1]);
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// can't relocate superblock, filesystem is now frozen
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
if (lfs_pair_cmp(oldpair, (const lfs_block_t[2]){0, 1}) == 0) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
LFS_WARN("Superblock %"PRIu32" has become unwritable", oldpair[1]);
Changed unwritable superblock to ENOSPC for consistency While ECORRUPT is not a wrong error code, it doesn't match other instances of hitting a corrupt block during write. During writes, if blocks are detected as corrupt their data is evicted and moved to a new clean block. This means that at the end of a disk's lifetime, exhaustion errors will be reported as ENOSPC when littlefs can't find any new block to store the data. This has the benefit of matching behaviour when a new file is written and no more blocks can be found, due to either a small disk or corrupted blocks on disk. To littlefs it's like the disk shrinks in size over time.
2018-08-13 19:20:40 +00:00
return LFS_ERR_NOSPC;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
// relocate half of pair
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
int err = lfs_alloc(lfs, &dir->pair[1]);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err && (err != LFS_ERR_NOSPC && !exhausted)) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
return err;
}
continue;
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (!relocated) {
lfs->gstate = lfs->gpending;
lfs->gdelta = (struct lfs_gstate){0};
} else {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// update references if we relocated
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
LFS_DEBUG("Relocating %"PRIu32" %"PRIu32" to %"PRIu32" %"PRIu32,
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]);
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
int err = lfs_fs_relocate(lfs, oldpair, dir->pair);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
return err;
}
}
return 0;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir,
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
const struct lfs_mattr *attrs, int attrcount) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
// check for any inline files that aren't RAM backed and
// forcefully evict them, needed for filesystem consistency
for (lfs_file_t *f = (lfs_file_t*)lfs->mlist; f; f = f->next) {
if (dir != &f->m && lfs_pair_cmp(f->m.pair, dir->pair) == 0 &&
f->type == LFS_TYPE_REG && (f->flags & LFS_F_INLINE) &&
f->ctz.size > lfs->cfg->cache_size) {
f->flags &= ~LFS_F_READING;
f->off = 0;
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
lfs_alloc_ack(lfs);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
int err = lfs_file_relocate(lfs, f);
if (err) {
return err;
}
err = lfs_file_flush(lfs, f);
if (err) {
return err;
}
}
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// calculate changes to the directory
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_tag_t deletetag = 0xffffffff;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_tag_t createtag = 0xffffffff;
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
for (int i = 0; i < attrcount; i++) {
if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_CREATE) {
createtag = attrs[i].tag;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
dir->count += 1;
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
} else if (lfs_tag_type3(attrs[i].tag) == LFS_TYPE_DELETE) {
deletetag = attrs[i].tag;
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
LFS_ASSERT(dir->count > 0);
dir->count -= 1;
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
} else if (lfs_tag_type1(attrs[i].tag) == LFS_TYPE_TAIL) {
dir->tail[0] = ((lfs_block_t*)attrs[i].buffer)[0];
dir->tail[1] = ((lfs_block_t*)attrs[i].buffer)[1];
dir->split = (lfs_tag_chunk(attrs[i].tag) & 1);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(dir->tail);
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
}
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// do we have a pending move?
if (lfs_gstate_hasmovehere(&lfs->gpending, dir->pair)) {
deletetag = lfs->gpending.tag & LFS_MKTAG(0x7ff, 0x3ff, 0);
LFS_ASSERT(dir->count > 0);
dir->count -= 1;
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
// mark gdelta so we reflect the move we will fix
lfs_gstate_xormove(&lfs->gdelta, &lfs->gpending, 0x3ff, NULL);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
// should we actually drop the directory block?
if (lfs_tag_isvalid(deletetag) && dir->count == 0) {
lfs_mdir_t pdir;
int err = lfs_fs_pred(lfs, dir->pair, &pdir);
if (err && err != LFS_ERR_NOENT) {
return err;
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
}
Added support for entry insertion Currently unused, the insertion of new file entries in arbitrary locations in a metadata-pair is very easy to add into the existing metadata logging. The only tricky things: 1. Name tags must strictly precede any tags related to a file. We can pull this off during a compact, but must make two passes. One for the name tag, one for the file. Though a benefit of this is that now our scans during moves can exit early upon finding the name tag. 1. We need to handle name tags appearing out of order. This makes name tags symmetric to deletes, although it doesn't seem like we can leverage this fact very well. Note this also means we need to make the superblock tag a type of name tag.
2018-09-09 14:01:06 +00:00
Fixed issues found by more aggressive rename tests - Fixed underflow issue caused by search id shortcuts that would result in early termination from lfs_dir_get - Fixed issue where entry file delete would toss out the best id during lfs_dir_fetchmatch - Fixed globals going out of date when canceling in same metadata-pair - Fixed early removal of metadata-pair when attribute list contains creates after deletes bring dir->count to zero
2018-10-21 16:25:48 +00:00
if (err != LFS_ERR_NOENT && pdir.split) {
return lfs_dir_drop(lfs, &pdir, dir);
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
}
Artificially limited number of file ids per metadata block This is an expirement to determine which field in the tag structure is the most critical: tag id or tag size. This came from looking at NAND storage and discussions around behaviour of large prog_sizes. Initial exploration indicates that prog_sizes around 2KiB are not _that_ uncommon, and the 1KiB limitation is surprising. It's possible to increase the lfs_tag size to 12-bits (4096), but at the cost of only 8-bit ids (256). [---- 32 ----] a [1|-3-|-- 8 --|-- 10 --|-- 10 --] b [1|-3-|-- 8 --|-- 8 --|-- 12 --] This requires more investigation, but in order to allow us to change the tag sizes with minimal impact I've artificially limited the number of file ids to 0xfe (255) different file ids per metadata pair. If 12-bit lengths turn out to be a bad idea, we can remove the artificial limit without backwards incompatible changes. To avoid breaking users already on v2-alpha, this change will refuse _creating_ file ids > 255, but should read file ids > 255 without issues.
2019-04-09 23:56:53 +00:00
if (dir->erased || dir->count >= 0xff) {
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// try to commit
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
struct lfs_commit commit = {
.block = dir->pair[0],
.off = dir->off,
.ptag = dir->etag,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
.crc = 0xffffffff,
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
.begin = dir->off,
.end = lfs->cfg->block_size - 8,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
};
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
// traverse attrs that need to be written out
lfs_pair_tole32(dir->tail);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_traverse(lfs,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
dir, dir->off, dir->etag, attrs, attrcount, false,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
0, 0, 0, 0, 0,
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
lfs_dir_commit_commit, &(struct lfs_dir_commit_commit){
lfs, &commit});
lfs_pair_fromle32(dir->tail);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
}
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
return err;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
// commit any global diffs if we have any
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (!lfs_gstate_iszero(&lfs->gdelta)) {
err = lfs_dir_getgstate(lfs, dir, &lfs->gdelta);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
return err;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
}
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs_gstate_tole32(&lfs->gdelta);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
err = lfs_dir_commitattr(lfs, &commit,
LFS_MKTAG(LFS_TYPE_MOVESTATE, 0x3ff,
sizeof(lfs->gdelta)), &lfs->gdelta);
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs_gstate_fromle32(&lfs->gdelta);
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
}
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
// finalize commit with the crc
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
err = lfs_dir_commitcrc(lfs, &commit);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
return err;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// successful commit, update dir
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->off = commit.off;
dir->etag = commit.ptag;
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
// note we able to have already handled move here
if (lfs_gstate_hasmovehere(&lfs->gpending, dir->pair)) {
lfs_gstate_xormove(&lfs->gpending, &lfs->gpending, 0x3ff, NULL);
}
// update gstate
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs->gstate = lfs->gpending;
lfs->gdelta = (struct lfs_gstate){0};
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
} else {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
compact:
// fall back to compaction
lfs_cache_drop(lfs, &lfs->pcache);
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
int err = lfs_dir_compact(lfs, dir, attrs, attrcount,
dir, 0, dir->count);
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
if (err) {
return err;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// update any directories that are affected
Fixed testing issues introduced by expanding superblocks This was mostly tweaking test cases to be accommodating for variable sized superblock-lists. Though there were a few bugs that needed fixing: - Changed compact to use source dir for move since the original dir could have changed as a result of an expand. - Created copy of current directory so we don't overwrite ourselves during an internal commit update. Also made sure all of the test suites provide reproducable results when ran independently (the entry tests were behaving differently based on which tests were ran before). (Some where legitimate test failures)
2018-08-07 19:10:03 +00:00
lfs_mdir_t copy = *dir;
// two passes, once for things that aren't us, and one
// for things that are
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) {
Fixed testing issues introduced by expanding superblocks This was mostly tweaking test cases to be accommodating for variable sized superblock-lists. Though there were a few bugs that needed fixing: - Changed compact to use source dir for move since the original dir could have changed as a result of an expand. - Created copy of current directory so we don't overwrite ourselves during an internal commit update. Also made sure all of the test suites provide reproducable results when ran independently (the entry tests were behaving differently based on which tests were ran before). (Some where legitimate test failures)
2018-08-07 19:10:03 +00:00
if (lfs_pair_cmp(d->m.pair, copy.pair) == 0) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
d->m = *dir;
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
if (d->id == lfs_tag_id(deletetag)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
d->m.pair[0] = 0xffffffff;
d->m.pair[1] = 0xffffffff;
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
} else if (d->id > lfs_tag_id(deletetag)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
d->id -= 1;
if (d->type == LFS_TYPE_DIR) {
((lfs_dir_t*)d)->pos -= 1;
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
} else if (&d->m != dir && d->id >= lfs_tag_id(createtag)) {
d->id += 1;
if (d->type == LFS_TYPE_DIR) {
((lfs_dir_t*)d)->pos += 1;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
while (d->id >= d->m.count && d->m.split) {
// we split and id is on tail now
d->id -= d->m.count;
int err = lfs_dir_fetch(lfs, &d->m, d->m.tail);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
return err;
}
}
}
}
return 0;
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level directory operations ///
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
int lfs_mkdir(lfs_t *lfs, const char *path) {
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
return err;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
uint16_t id;
err = lfs_dir_find(lfs, &cwd, &path, &id);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (!(err == LFS_ERR_NOENT && id != 0x3ff)) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return (err < 0) ? err : LFS_ERR_EXIST;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
// check that name fits
lfs_size_t nlen = strlen(path);
Changed name of upper-limits from blah_size to blah_max This standardizes the naming between the LFS_BLAH_MAX macros and the blah_max configuration in the lfs_config structure.
2018-08-05 01:10:08 +00:00
if (nlen > lfs->name_max) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return LFS_ERR_NAMETOOLONG;
}
// build up new directory
lfs_alloc_ack(lfs);
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = lfs_dir_alloc(lfs, &dir);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
// find end of list
lfs_mdir_t pred = cwd;
while (pred.split) {
err = lfs_dir_fetch(lfs, &pred, pred.tail);
if (err) {
return err;
}
}
// setup dir
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_tole32(pred.tail);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &dir, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), pred.tail}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(pred.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
// current block end of list?
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (cwd.split) {
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
// update tails, this creates a desync
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, +1);
lfs_pair_tole32(dir.pair);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &pred, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), dir.pair}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(dir.pair);
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
if (err) {
return err;
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -1);
Fixed mkdir when inserting into a non-end block This was an oversight on my part when adding strict ordering to directories. Unfortunately now we can't take advantage of the atomic creation of tail+dir entries. Now we need to first create the tail, then create the actually directory entry. If we lose power, the orphan is cleaned up like orphans created during remove. Note that we still take advantage of the atomic tail+dir entries if we are an end block. This is actually because this corner case is complicated to _not_ do atomically, needing to update the directory we just committed to.
2018-10-05 23:22:33 +00:00
}
// now insert into our parent block
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_tole32(dir.pair);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &cwd, LFS_MKATTRS(
Enabled -Wextra This only required adding NULLs where commit statements were not fully initialized. Unfortunately we still need -Wno-missing-field-initializers because of a bug in GCC that persists on Travis. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60784 Found by apmorton
2019-02-12 06:11:01 +00:00
{LFS_MKTAG(LFS_TYPE_CREATE, id, 0), NULL},
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
{LFS_MKTAG(LFS_TYPE_DIR, id, nlen), path},
{LFS_MKTAG(LFS_TYPE_DIRSTRUCT, id, 8), dir.pair},
{!cwd.split
? LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8)
: LFS_MKTAG(LFS_FROM_NOOP, 0, 0), dir.pair}));
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_fromle32(dir.pair);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_open(lfs_t *lfs, lfs_dir_t *dir, const char *path) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &dir->m, &path, NULL);
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag < 0) {
return tag;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) != LFS_TYPE_DIR) {
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
return LFS_ERR_NOTDIR;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_block_t pair[2];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_id(tag) == 0x3ff) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// handle root dir separately
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
pair[0] = lfs->root[0];
pair[1] = lfs->root[1];
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else {
// get dir pair from parent
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &dir->m, LFS_MKTAG(0x700, 0x3ff, 0),
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tag_id(tag), 8), pair);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (res < 0) {
return res;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_fromle32(pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// fetch first pair
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// setup entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->head[0] = dir->m.pair[0];
dir->head[1] = dir->m.pair[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->id = 0;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->pos = 0;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// add to list of mdirs
dir->type = LFS_TYPE_DIR;
dir->next = (lfs_dir_t*)lfs->mlist;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->mlist = (struct lfs_mlist*)dir;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// remove from list of mdirs
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (struct lfs_mlist **p = &lfs->mlist; *p; p = &(*p)->next) {
if (*p == (struct lfs_mlist*)dir) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
*p = (*p)->next;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
break;
}
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_read(lfs_t *lfs, lfs_dir_t *dir, struct lfs_info *info) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
memset(info, 0, sizeof(*info));
// special offset for '.' and '..'
if (dir->pos == 0) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, ".");
dir->pos += 1;
return 1;
} else if (dir->pos == 1) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, "..");
dir->pos += 1;
return 1;
}
while (true) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (dir->id == dir->m.count) {
if (!dir->m.split) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return false;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->m.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
dir->id = 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_getinfo(lfs, &dir->m, dir->id, info);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err && err != LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
dir->id += 1;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err != LFS_ERR_NOENT) {
break;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
dir->pos += 1;
return true;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_seek(lfs_t *lfs, lfs_dir_t *dir, lfs_off_t off) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// simply walk from head dir
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_rewind(lfs, dir);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// first two for ./..
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->pos = lfs_min(2, off);
off -= dir->pos;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
while (off != 0) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->id = lfs_min(dir->m.count, off);
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->pos += dir->id;
off -= dir->id;
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (dir->id == dir->m.count) {
if (!dir->m.split) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
return LFS_ERR_INVAL;
}
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = lfs_dir_fetch(lfs, &dir->m, dir->m.tail);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (err) {
return err;
}
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_soff_t lfs_dir_tell(lfs_t *lfs, lfs_dir_t *dir) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return dir->pos;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_rewind(lfs_t *lfs, lfs_dir_t *dir) {
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
// reload the head dir
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->head);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
return err;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->m.pair[0] = dir->head[0];
dir->m.pair[1] = dir->head[1];
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->id = 0;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->pos = 0;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// File index list operations ///
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static int lfs_ctz_index(lfs_t *lfs, lfs_off_t *off) {
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
lfs_off_t size = *off;
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
lfs_off_t b = lfs->cfg->block_size - 2*4;
lfs_off_t i = size / b;
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
if (i == 0) {
return 0;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
i = (size - 4*(lfs_popc(i-1)+2)) / b;
*off = size - b*i - 4*lfs_popc(i);
return i;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static int lfs_ctz_find(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_block_t head, lfs_size_t size,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t pos, lfs_block_t *block, lfs_off_t *off) {
if (size == 0) {
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
*off = 0;
return 0;
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_off_t current = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
lfs_off_t target = lfs_ctz_index(lfs, &pos);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
while (current > target) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t skip = lfs_min(
lfs_npw2(current-target+1) - 1,
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
lfs_ctz(current));
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int err = lfs_bd_read(lfs,
pcache, rcache, sizeof(head),
head, 4*skip, &head, sizeof(head));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
current -= 1 << skip;
}
*block = head;
*off = pos;
return 0;
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static int lfs_ctz_extend(lfs_t *lfs,
lfs_cache_t *pcache, lfs_cache_t *rcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
lfs_block_t *block, lfs_off_t *off) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
// go ahead and grab a block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(nblock >= 2 && nblock <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
err = lfs_bd_erase(lfs, nblock);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (size == 0) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 0;
return 0;
}
size -= 1;
lfs_off_t index = lfs_ctz_index(lfs, &size);
size += 1;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
// just copy out the last block if it is incomplete
if (size != lfs->cfg->block_size) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t data;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_bd_read(lfs,
NULL, rcache, size-i,
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
head, i, &data, 1);
if (err) {
return err;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_bd_prog(lfs,
pcache, rcache, true,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, i, &data, 1);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = size;
return 0;
}
// append block
index += 1;
lfs_size_t skips = lfs_ctz(index) + 1;
for (lfs_off_t i = 0; i < skips; i++) {
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_tole32(head);
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_bd_prog(lfs, pcache, rcache, true,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (i != skips-1) {
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_bd_read(lfs,
NULL, rcache, sizeof(head),
head, 4*i, &head, sizeof(head));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 4*skips;
return 0;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
relocate:
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
LFS_DEBUG("Bad block at %"PRIu32, nblock);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// just clear cache and try a new block
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, pcache);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
static int lfs_ctz_traverse(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int (*cb)(void*, lfs_block_t), void *data) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (size == 0) {
return 0;
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_off_t index = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
while (true) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int err = cb(data, head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
if (index == 0) {
return 0;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
lfs_block_t heads[2];
int count = 2 - (index & 1);
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
err = lfs_bd_read(lfs,
pcache, rcache, count*sizeof(head),
head, 0, &heads, count*sizeof(head));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
heads[0] = lfs_fromle32(heads[0]);
heads[1] = lfs_fromle32(heads[1]);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
for (int i = 0; i < count-1; i++) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
err = cb(data, heads[i]);
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
if (err) {
return err;
}
}
head = heads[count-1];
index -= count;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level file operations ///
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
int lfs_file_opencfg(lfs_t *lfs, lfs_file_t *file,
const char *path, int flags,
const struct lfs_file_config *cfg) {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
// deorphan if we haven't yet, needed at most once after poweron
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if ((flags & 3) != LFS_O_RDONLY) {
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// setup simple file details
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
int err;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
file->cfg = cfg;
file->flags = flags;
file->pos = 0;
file->cache.buffer = NULL;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
// allocate entry for file if it doesn't exist
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &file->m, &path, &file->id);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (tag < 0 && !(tag == LFS_ERR_NOENT && file->id != 0x3ff)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = tag;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// get id, add to list of mdirs to catch update changes
file->type = LFS_TYPE_REG;
file->next = (lfs_file_t*)lfs->mlist;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs->mlist = (struct lfs_mlist*)file;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag == LFS_ERR_NOENT) {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (!(flags & LFS_O_CREAT)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOENT;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
// check that name fits
lfs_size_t nlen = strlen(path);
Changed name of upper-limits from blah_size to blah_max This standardizes the naming between the LFS_BLAH_MAX macros and the blah_max configuration in the lfs_config structure.
2018-08-05 01:10:08 +00:00
if (nlen > lfs->name_max) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NAMETOOLONG;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
// get next slot and create entry to remember name
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &file->m, LFS_MKATTRS(
Enabled -Wextra This only required adding NULLs where commit statements were not fully initialized. Unfortunately we still need -Wno-missing-field-initializers because of a bug in GCC that persists on Travis. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60784 Found by apmorton
2019-02-12 06:11:01 +00:00
{LFS_MKTAG(LFS_TYPE_CREATE, file->id, 0), NULL},
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
{LFS_MKTAG(LFS_TYPE_REG, file->id, nlen), path},
Enabled -Wextra This only required adding NULLs where commit statements were not fully initialized. Unfortunately we still need -Wno-missing-field-initializers because of a bug in GCC that persists on Travis. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60784 Found by apmorton
2019-02-12 06:11:01 +00:00
{LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0), NULL}));
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NAMETOOLONG;
goto cleanup;
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
tag = LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, 0);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
} else if (flags & LFS_O_EXCL) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_EXIST;
goto cleanup;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(tag) != LFS_TYPE_REG) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_ISDIR;
goto cleanup;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
} else if (flags & LFS_O_TRUNC) {
// truncate if requested
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
tag = LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0);
file->flags |= LFS_F_DIRTY;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
// try to load what's on disk, if it's inlined we'll fix it later
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
tag = lfs_dir_get(lfs, &file->m, LFS_MKTAG(0x700, 0x3ff, 0),
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, file->id, 8), &file->ctz);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag < 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = tag;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_ctz_fromle32(&file->ctz);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
// fetch attrs
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
for (unsigned i = 0; i < file->cfg->attr_count; i++) {
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if ((file->flags & 3) != LFS_O_WRONLY) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &file->m,
LFS_MKTAG(0x7ff, 0x3ff, 0),
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
LFS_MKTAG(LFS_TYPE_USERATTR + file->cfg->attrs[i].type,
file->id, file->cfg->attrs[i].size),
file->cfg->attrs[i].buffer);
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if (res < 0 && res != LFS_ERR_NOENT) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = res;
goto cleanup;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
}
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if ((file->flags & 3) != LFS_O_RDONLY) {
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
if (file->cfg->attrs[i].size > lfs->attr_max) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOSPC;
goto cleanup;
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
}
file->flags |= LFS_F_DIRTY;
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
// allocate buffer if needed
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if (file->cfg->buffer) {
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
file->cache.buffer = file->cfg->buffer;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOMEM;
goto cleanup;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
}
Fixed information leaks through reused caches As a shortcut, littlefs never bother to zero any of the buffers is used. It didn't need to because it would always write out the entirety of the data it needed. Unfortunately, this, combined with the extra padding used to align buffers to the nearest prog size, would lead to uninitialized data getting written out to disk. This means unrelated file data could be written to different parts of storage, or worse, information leaked from the malloc calls could be written out to disk unnecessarily. found by rojer
2018-07-06 16:14:30 +00:00
// zero to avoid information leak
lfs_cache_zero(lfs, &file->cache);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_INLINESTRUCT) {
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
// load inline files
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head = 0xfffffffe;
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
file->ctz.size = lfs_tag_size(tag);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->flags |= LFS_F_INLINE;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->cache.block = file->ctz.head;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->cache.off = 0;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.size = lfs->cfg->cache_size;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
// don't always read (may be new/trunc file)
if (file->ctz.size > 0) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &file->m,
LFS_MKTAG(0x700, 0x3ff, 0),
Added better handling of large program sizes (> 1024) The issue here is how commits handle padding to the nearest program size. This is done by exploiting the size field of the LFS_TYPE_CRC tag that completes the commit. Unfortunately, during developement, the size field shrank in size to make room for more type information, limiting the size field to 1024. Normally this isn't a problem, as very rarely do program sizes exceed 1024 bytes. However, using a simulated block device, user earlephilhower found that exceeding 1024 caused littlefs to crash. To make this corner case behave in a more user friendly manner, I've modified this situtation to treat >1024 program sizes as small commits that don't match the prog size. As a part of this, littlefs also needed to understand that non-matching commits indicate an "unerased" dir block, which would be needed for portability (something which notably lacks testing). This raises the question of if the tag size field size needs to be reconsidered, but to change that at this point would need a new major version. found by earlephilhower
2019-04-09 21:06:43 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, file->id,
lfs_min(file->cache.size, 0x3fe)),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->cache.buffer);
if (res < 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = res;
goto cleanup;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
return 0;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
cleanup:
// clean up lingering resources
file->flags |= LFS_F_ERRED;
lfs_file_close(lfs, file);
return err;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
int lfs_file_open(lfs_t *lfs, lfs_file_t *file,
const char *path, int flags) {
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
static const struct lfs_file_config defaults = {0};
return lfs_file_opencfg(lfs, file, path, flags, &defaults);
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_close(lfs_t *lfs, lfs_file_t *file) {
int err = lfs_file_sync(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// remove from list of mdirs
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (struct lfs_mlist **p = &lfs->mlist; *p; p = &(*p)->next) {
if (*p == (struct lfs_mlist*)file) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
*p = (*p)->next;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
break;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// clean up memory
Fixed minor memory leak - Fixed memory leak - Change lfs_globals_zero to use memset as this made leak checking more effective - Checked for leaks with valgrind
2018-08-11 18:45:13 +00:00
if (!file->cfg->buffer) {
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
lfs_free(file->cache.buffer);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
return err;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_relocate(lfs_t *lfs, lfs_file_t *file) {
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
while (true) {
// just relocate what exists into new block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
Adopted redundant cache read in lfs_file_relocate Previously had some custom logic that could be reduced
2017-06-25 22:23:40 +00:00
if (err) {
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
err = lfs_bd_erase(lfs, nblock);
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
// either read from dirty cache or disk
for (lfs_off_t i = 0; i < file->off; i++) {
uint8_t data;
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
if (file->flags & LFS_F_INLINE) {
err = lfs_dir_getread(lfs, &file->m,
// note we evict inline files before they can be dirty
NULL, &file->cache, file->off-i,
LFS_MKTAG(0xfff, 0x1ff, 0),
LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0),
i, &data, 1);
if (err) {
return err;
}
} else {
err = lfs_bd_read(lfs,
&file->cache, &lfs->rcache, file->off-i,
file->block, i, &data, 1);
if (err) {
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
err = lfs_bd_prog(lfs,
&lfs->pcache, &lfs->rcache, true,
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
nblock, i, &data, 1);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
}
// copy over new state of file
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
memcpy(file->cache.buffer, lfs->pcache.buffer, lfs->cfg->cache_size);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
file->cache.block = lfs->pcache.block;
file->cache.off = lfs->pcache.off;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.size = lfs->pcache.size;
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_zero(lfs, &lfs->pcache);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
file->block = nblock;
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
file->flags &= ~LFS_F_INLINE;
file->flags |= LFS_F_WRITING;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
return 0;
relocate:
Fixed inline files when inline_max == cache_size The initial implementation of inline files was thrown together fairly quicky, however it has worked well so far and there hasn't been much reason to change it. One shortcut was to trick file writes into thinking they are writing to imaginary blocks. This works well and reuses most of the file code paths, as long as we don't flush the imaginary block out to disk. Initially we did this by limiting inline_max to cache_max-1, ensuring that the cache never fills up and gets flushed. This was a rather dirty hack, the better solution, implemented here, is to handle the representation of an "imaginary" block correctly all the way down into the cache layer. So now for files specifically, the value -1 represents a null pointer, and the value -2 represents an "imaginary" block. This may become a problem if the number of blocks approaches the max, however this -2 value is never written to disk and can be changed in the future without breaking compatibility.
2018-08-13 19:08:30 +00:00
LFS_DEBUG("Bad block at %"PRIu32, nblock);
// just clear cache and try a new block
lfs_cache_drop(lfs, &lfs->pcache);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file) {
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
if (file->flags & LFS_F_READING) {
if (!(file->flags & LFS_F_INLINE)) {
lfs_cache_drop(lfs, &file->cache);
}
file->flags &= ~LFS_F_READING;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (file->flags & LFS_F_WRITING) {
lfs_off_t pos = file->pos;
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (!(file->flags & LFS_F_INLINE)) {
// copy over anything after current branch
lfs_file_t orig = {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
.ctz.head = file->ctz.head,
.ctz.size = file->ctz.size,
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
.flags = LFS_O_RDONLY,
.pos = file->pos,
.cache = lfs->rcache,
};
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, &lfs->rcache);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
while (file->pos < file->ctz.size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// copy over a byte at a time, leave it up to caching
// to make this efficient
uint8_t data;
lfs_ssize_t res = lfs_file_read(lfs, &orig, &data, 1);
if (res < 0) {
return res;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
res = lfs_file_write(lfs, file, &data, 1);
if (res < 0) {
return res;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// keep our reference to the rcache in sync
if (lfs->rcache.block != 0xffffffff) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_drop(lfs, &orig.cache);
lfs_cache_drop(lfs, &lfs->rcache);
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// write out what we have
while (true) {
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
int err = lfs_bd_flush(lfs, &file->cache, &lfs->rcache, true);
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
break;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
relocate:
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
LFS_DEBUG("Bad block at %"PRIu32, file->block);
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
err = lfs_file_relocate(lfs, file);
if (err) {
return err;
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
Fixed accidental truncate after seek on inline files The cause was mistakenly setting file->ctz.size directly instead of file->pos, which file->ctz.size gets overwritten with later in lfs_file_flush. Also added better seek test cases specifically for inline files. This should also catch most of the inline corner cases related to lfs_file_size/lfs_file_tell. Found by ebinans
2019-05-28 18:55:03 +00:00
file->pos = lfs_max(file->pos, file->ctz.size);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// actual file updates
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head = file->block;
file->ctz.size = file->pos;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_WRITING;
file->flags |= LFS_F_DIRTY;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = pos;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) {
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
while (true) {
int err = lfs_file_flush(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
Fixed issue with handling block device errors in lfs_file_sync lfs_file_sync was not correctly setting the LFS_F_ERRED flag. Fortunately this is a relatively easy fix. LFS_F_ERRED prevents further issues from occuring when cleaning up resources with lfs_file_close. found by TheLoneWolfling
2019-04-09 22:41:26 +00:00
file->flags |= LFS_F_ERRED;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
if ((file->flags & LFS_F_DIRTY) &&
!(file->flags & LFS_F_ERRED) &&
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
!lfs_pair_isnull(file->m.pair)) {
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
// update dir entry
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
uint16_t type;
const void *buffer;
lfs_size_t size;
Fixed issue with big-endian CTZ lists intertwined in commit logic Found while testing big-endian support. Basically, if littlefs is really really unlucky, the block allocator could kick in while committing a file's CTZ reference. If this happens, the block allocator will need to traverse all CTZ skip-lists in memory, including the skip-list we're committing. This means we can't convert the CTZ's endianness in place, and need to make a copy on big-endian systems. We rely on dead-code elimination from the compiler to make the conditional behaviour for big-endian vs little-endian system a noop determined by the lfs_tole32 intrinsic.
2018-08-08 14:37:43 +00:00
struct lfs_ctz ctz;
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
if (file->flags & LFS_F_INLINE) {
// inline the whole file
type = LFS_TYPE_INLINESTRUCT;
buffer = file->cache.buffer;
size = file->ctz.size;
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
} else {
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
// update the ctz reference
type = LFS_TYPE_CTZSTRUCT;
Fixed issue with big-endian CTZ lists intertwined in commit logic Found while testing big-endian support. Basically, if littlefs is really really unlucky, the block allocator could kick in while committing a file's CTZ reference. If this happens, the block allocator will need to traverse all CTZ skip-lists in memory, including the skip-list we're committing. This means we can't convert the CTZ's endianness in place, and need to make a copy on big-endian systems. We rely on dead-code elimination from the compiler to make the conditional behaviour for big-endian vs little-endian system a noop determined by the lfs_tole32 intrinsic.
2018-08-08 14:37:43 +00:00
// copy ctz so alloc will work during a relocate
ctz = file->ctz;
lfs_ctz_tole32(&ctz);
buffer = &ctz;
size = sizeof(ctz);
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
}
// commit file data and attributes
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &file->m, LFS_MKATTRS(
{LFS_MKTAG(type, file->id, size), buffer},
{LFS_MKTAG(LFS_FROM_USERATTRS, file->id,
file->cfg->attr_count), file->cfg->attrs}));
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
if (err) {
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
if (err == LFS_ERR_NOSPC && (file->flags & LFS_F_INLINE)) {
goto relocate;
}
Fixed issue with handling block device errors in lfs_file_sync lfs_file_sync was not correctly setting the LFS_F_ERRED flag. Fortunately this is a relatively easy fix. LFS_F_ERRED prevents further issues from occuring when cleaning up resources with lfs_file_close. found by TheLoneWolfling
2019-04-09 22:41:26 +00:00
file->flags |= LFS_F_ERRED;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
file->flags &= ~LFS_F_DIRTY;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
relocate:
// inline file doesn't fit anymore
file->off = file->pos;
err = lfs_file_relocate(lfs, file);
if (err) {
Fixed issue with handling block device errors in lfs_file_sync lfs_file_sync was not correctly setting the LFS_F_ERRED flag. Fortunately this is a relatively easy fix. LFS_F_ERRED prevents further issues from occuring when cleaning up resources with lfs_file_close. found by TheLoneWolfling
2019-04-09 22:41:26 +00:00
file->flags |= LFS_F_ERRED;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
return err;
}
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
void *buffer, lfs_size_t size) {
uint8_t *data = buffer;
lfs_size_t nsize = size;
if ((file->flags & 3) == LFS_O_WRONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_WRITING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// flush out any writes
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (file->pos >= file->ctz.size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
// eof if past end
return 0;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
size = lfs_min(size, file->ctz.size - file->pos);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
nsize = size;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
while (nsize > 0) {
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_READING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_find(lfs, NULL, &file->cache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->pos, &file->block, &file->off);
if (err) {
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags |= LFS_F_READING;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
// read as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
if (file->flags & LFS_F_INLINE) {
int err = lfs_dir_getread(lfs, &file->m,
NULL, &file->cache, lfs->cfg->block_size,
LFS_MKTAG(0xfff, 0x1ff, 0),
LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0),
file->off, data, diff);
if (err) {
return err;
}
} else {
int err = lfs_bd_read(lfs,
NULL, &file->cache, lfs->cfg->block_size,
file->block, file->off, data, diff);
if (err) {
return err;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
data += diff;
nsize -= diff;
}
return size;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
lfs_size_t nsize = size;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_READING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// drop any reads
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if ((file->flags & LFS_O_APPEND) && file->pos < file->ctz.size) {
file->pos = file->ctz.size;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (file->pos + size > lfs->file_max) {
// Larger than file limit?
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
return LFS_ERR_FBIG;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > file->ctz.size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
// fill with zeros
lfs_off_t pos = file->pos;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->pos = file->ctz.size;
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
while (file->pos < pos) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if ((file->flags & LFS_F_INLINE) &&
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
lfs_max(file->pos+nsize, file->ctz.size) >
Fixed limit of inline files based on LFS_ATTR_MAX The maximum limit of inline files and attributes are unrelated, but were not at a point in littlefs v2 development. This should be checking against the bit-field limit in the littlefs tag. Found by lsilvaalmeida
2019-05-22 21:18:41 +00:00
lfs_min(0x3fe, lfs_min(
Fixed excessive waste from overly large inline files Before this, there were some safety limits, but there was no real default limit to the size of inline files other than the amount of RAM available. On PCs, this meant that inline files were free to fill up directory blocks to a little under the block size. However this is very wasteful in terms of storage space. Because of splitting limits to keep the compact runtime reasonable, each byte of an inline files uses 4x the amount. Fortunately we can find an optimal inline limit: Inline file waste for n bytes = 3n CTZ file waste for n bytes = B - n Where B = block size Solving for n = B/4 So the optimal inline limit is B/4. However, this assumes a perfect inline file and no metadata. We can decrease this to B/8 to give a bit more breathing room for directory+file metadata.
2019-01-14 23:53:41 +00:00
lfs->cfg->cache_size, lfs->cfg->block_size/8))) {
Bumped versions, cleaned up some TODOs and missing comments
2018-04-03 13:29:28 +00:00
// inline file doesn't fit anymore
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
file->off = file->pos;
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
lfs_alloc_ack(lfs);
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
int err = lfs_file_relocate(lfs, file);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
}
while (nsize > 0) {
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_WRITING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > 0) {
// find out which block we're extending from
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_find(lfs, NULL, &file->cache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->pos-1, &file->block, &file->off);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
}
// mark cache as dirty since we may have read data into it
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_cache_zero(lfs, &file->cache);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
// extend file with new blocks
lfs_alloc_ack(lfs);
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_extend(lfs, &file->cache, &lfs->rcache,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->block, file->pos,
&file->block, &file->off);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
file->flags |= LFS_F_WRITING;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// program as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Revisited caching rules to optimize bus transactions The littlefs driver has always had this really weird quirk: larger cache sizes can significantly harm performance. This has probably been one of the most surprising pieces of configuraing and optimizing littlefs. The reason is that littlefs's caches are kinda dumb (this is somewhat intentional, as dumb caches take up much less code space than smart caches). When littlefs needs to read data, it will load the entire cache line. This means that even when we only need a small 4 byte piece of data, we may need to read a full 512 byte cache. And since microcontrollers may be reading from storage over relatively slow bus protocols, the time to send data over the bus may dominate other operations. Now that we have separate configuration options for "cache_size" and "read_size", we can start making littlefs's caches a bit smarter. They aren't going to be perfect, because code size is still a priority, but there are some small improvements we can do: 1. Program caches write to prog_size aligned units, but eagerly cache as much as possible. There's no downside to using the full cache in program operations. 2. Add a hint parameter to cached reads. This internal API allows callers to tell the cache how much data they expect to need. This avoids excess bus traffic, and now we can even bypass the cache if the caller provides enough of a buffer. We can still fall back to reading full cache-lines in the cases where we don't know how much data we need by providing the block size as the hint. We do this for directory fetches and for file reads. This has immediate improvements for both metadata-log traversal and CTZ skip-list traversal, since these both only need to read 4-byte pointers and can always bypass the cache, allowing reuse elsewhere.
2018-08-20 19:47:52 +00:00
int err = lfs_bd_prog(lfs, &file->cache, &lfs->rcache, true,
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
file->block, file->off, data, diff);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
break;
relocate:
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
err = lfs_file_relocate(lfs, file);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
data += diff;
nsize -= diff;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_alloc_ack(lfs);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags &= ~LFS_F_ERRED;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
return size;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t off, int whence) {
// write out everything beforehand, may be noop if rdonly
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
// find new pos
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
lfs_off_t npos = file->pos;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
if (whence == LFS_SEEK_SET) {
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
npos = off;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
} else if (whence == LFS_SEEK_CUR) {
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
npos = file->pos + off;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
} else if (whence == LFS_SEEK_END) {
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
npos = file->ctz.size + off;
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
}
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
if (npos > lfs->file_max) {
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
// file position out of range
return LFS_ERR_INVAL;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added 2GiB file size limit and EFBIG reporting On disk, littlefs uses 32-bit integers to track file size. This sets a theoretical limit of 4GiB for files. However, the API passes file sizes around as signed numbers, with negative values representing error codes. This means that not all of the APIs will work with file sizes > 2GiB. Because of related complications over in FUSE land, I've added the LFS_FILE_MAX constant and proper error reporting if file writes/seeks exceed the 2GiB limit. In v2 this will join the other constants that get stored in the superblock to help portability. Since littlefs is targeting microcontrollers, it's likely this will be a sufficient solution. Note that it's still possible to enable partial-support for 4GiB files by defining LFS_FILE_MAX during compilation. This will work for most of the APIs, except lfs_file_seek, lfs_file_tell, and lfs_file_size. We can also consider improving support for 4GiB files, by making seek a bit more complicated and adding a lfs_file_stat function. I'll leave this for a future improvement if there's interest. Found by cgrozemuller
2018-10-08 19:12:20 +00:00
// update pos
file->pos = npos;
return npos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size) {
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Fix: length more than LFS_FILE_MAX should return error To make lfs_file_truncate inline with ftruncate function, when -ve or size more than maximum file size is passed to function it should return invalid parameter error. In LFS case LFS_ERR_INVAL. Signed-off-by: Ajay Bhargav <contact@rickeyworld.info>
2019-02-17 11:39:58 +00:00
if (size > LFS_FILE_MAX) {
return LFS_ERR_INVAL;
}
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
lfs_off_t oldsize = lfs_file_size(lfs, file);
if (size < oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
// need to flush since directly changing metadata
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
// lookup new head in ctz skip list
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
err = lfs_ctz_find(lfs, NULL, &file->cache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
Fixed issue with writes following a truncate The problem was not setting the file state correctly after the truncate. To truncate < size, we end up using the cache to traverse the ctz skip-list far away from where our file->pos is. We can leave the last block in the cache in case we're going to append to the file, but if we do this we need to set up file->block+file->off to tell use where we are in the file, and set the LFS_F_READING flag to indicate that our cache contains read data. Note this is different than the LFS_F_DIRTY, which we need also. The purpose of the flags are as follows: - LFS_F_DIRTY - file ctz skip-list branch is out of sync with filesystem, need to update metadata - LFS_F_READING - file cache is in use for reading, need to drop cache - LFS_F_WRITING - file cache is in use for writing, need to write out cache to disk The difference between flags is subtle but important because read/prog caches are handled differently. Prog caches have asserts in place to catch programs without erases (the infamous pcache->block == 0xffffffff assert). Though maybe the names deserve an update... Found by ebinans
2019-05-22 19:24:05 +00:00
size, &file->block, &file->off);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (err) {
return err;
}
Fixed issue with writes following a truncate The problem was not setting the file state correctly after the truncate. To truncate < size, we end up using the cache to traverse the ctz skip-list far away from where our file->pos is. We can leave the last block in the cache in case we're going to append to the file, but if we do this we need to set up file->block+file->off to tell use where we are in the file, and set the LFS_F_READING flag to indicate that our cache contains read data. Note this is different than the LFS_F_DIRTY, which we need also. The purpose of the flags are as follows: - LFS_F_DIRTY - file ctz skip-list branch is out of sync with filesystem, need to update metadata - LFS_F_READING - file cache is in use for reading, need to drop cache - LFS_F_WRITING - file cache is in use for writing, need to write out cache to disk The difference between flags is subtle but important because read/prog caches are handled differently. Prog caches have asserts in place to catch programs without erases (the infamous pcache->block == 0xffffffff assert). Though maybe the names deserve an update... Found by ebinans
2019-05-22 19:24:05 +00:00
file->ctz.head = file->block;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.size = size;
Fixed issue with writes following a truncate The problem was not setting the file state correctly after the truncate. To truncate < size, we end up using the cache to traverse the ctz skip-list far away from where our file->pos is. We can leave the last block in the cache in case we're going to append to the file, but if we do this we need to set up file->block+file->off to tell use where we are in the file, and set the LFS_F_READING flag to indicate that our cache contains read data. Note this is different than the LFS_F_DIRTY, which we need also. The purpose of the flags are as follows: - LFS_F_DIRTY - file ctz skip-list branch is out of sync with filesystem, need to update metadata - LFS_F_READING - file cache is in use for reading, need to drop cache - LFS_F_WRITING - file cache is in use for writing, need to write out cache to disk The difference between flags is subtle but important because read/prog caches are handled differently. Prog caches have asserts in place to catch programs without erases (the infamous pcache->block == 0xffffffff assert). Though maybe the names deserve an update... Found by ebinans
2019-05-22 19:24:05 +00:00
file->flags |= LFS_F_DIRTY | LFS_F_READING;
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
} else if (size > oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
lfs_off_t pos = file->pos;
// flush+seek if not already at end
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
if (file->pos != oldsize) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
int err = lfs_file_seek(lfs, file, 0, LFS_SEEK_END);
Fixed error check when truncating files to larger size
2018-02-04 19:48:44 +00:00
if (err < 0) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
return err;
}
}
// fill with zeros
while (file->pos < size) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
// restore pos
int err = lfs_file_seek(lfs, file, pos, LFS_SEEK_SET);
if (err < 0) {
return err;
}
}
return 0;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_tell(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
int lfs_file_rewind(lfs_t *lfs, lfs_file_t *file) {
lfs_soff_t res = lfs_file_seek(lfs, file, 0, LFS_SEEK_SET);
if (res < 0) {
return res;
}
return 0;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (file->flags & LFS_F_WRITING) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return lfs_max(file->pos, file->ctz.size);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
} else {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return file->ctz.size;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Commentary typo fix
2018-01-30 19:07:37 +00:00
/// General fs operations ///
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
int lfs_stat(lfs_t *lfs, const char *path, struct lfs_info *info) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &cwd, &path, NULL);
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag < 0) {
return tag;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
return lfs_dir_getinfo(lfs, &cwd, lfs_tag_id(tag), info);
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int lfs_remove(lfs_t *lfs, const char *path) {
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &cwd, &path, NULL);
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
if (tag < 0 || lfs_tag_id(tag) == 0x3ff) {
return (tag < 0) ? tag : LFS_ERR_INVAL;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs_mdir_t dir;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_DIR) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// must be empty before removal
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_block_t pair[2];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &cwd, LFS_MKTAG(0x700, 0x3ff, 0),
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tag_id(tag), 8), pair);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (res < 0) {
return res;
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_fromle32(pair);
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = lfs_dir_fetch(lfs, &dir, pair);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
if (dir.count > 0 || dir.split) {
return LFS_ERR_NOTEMPTY;
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// mark fs as orphaned
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, +1);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// delete the entry
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &cwd, LFS_MKATTRS(
Enabled -Wextra This only required adding NULLs where commit statements were not fully initialized. Unfortunately we still need -Wno-missing-field-initializers because of a bug in GCC that persists on Travis. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60784 Found by apmorton
2019-02-12 06:11:01 +00:00
{LFS_MKTAG(LFS_TYPE_DELETE, lfs_tag_id(tag), 0), NULL}));
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_DIR) {
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// fix orphan
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -1);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = lfs_fs_pred(lfs, dir.pair, &cwd);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err) {
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
err = lfs_dir_drop(lfs, &cwd, &dir);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return 0;
}
int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) {
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// find old entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t oldcwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t oldtag = lfs_dir_find(lfs, &oldcwd, &oldpath, NULL);
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
if (oldtag < 0 || lfs_tag_id(oldtag) == 0x3ff) {
return (oldtag < 0) ? oldtag : LFS_ERR_INVAL;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// find new entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t newcwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
uint16_t newid;
lfs_stag_t prevtag = lfs_dir_find(lfs, &newcwd, &newpath, &newid);
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
if ((prevtag < 0 || lfs_tag_id(prevtag) == 0x3ff) &&
!(prevtag == LFS_ERR_NOENT && newid != 0x3ff)) {
return (prevtag < 0) ? prevtag : LFS_ERR_INVAL;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs_mdir_t prevdir;
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
if (prevtag == LFS_ERR_NOENT) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// check that name fits
lfs_size_t nlen = strlen(newpath);
Changed name of upper-limits from blah_size to blah_max This standardizes the naming between the LFS_BLAH_MAX macros and the blah_max configuration in the lfs_config structure.
2018-08-05 01:10:08 +00:00
if (nlen > lfs->name_max) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return LFS_ERR_NAMETOOLONG;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(prevtag) != lfs_tag_type3(oldtag)) {
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
return LFS_ERR_ISDIR;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
} else if (lfs_tag_type3(prevtag) == LFS_TYPE_DIR) {
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// must be empty before removal
lfs_block_t prevpair[2];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &newcwd, LFS_MKTAG(0x700, 0x3ff, 0),
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, newid, 8), prevpair);
if (res < 0) {
return res;
}
lfs_pair_fromle32(prevpair);
// must be empty before removal
err = lfs_dir_fetch(lfs, &prevdir, prevpair);
if (err) {
return err;
}
if (prevdir.count > 0 || prevdir.split) {
return LFS_ERR_NOTEMPTY;
}
// mark fs as orphaned
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, +1);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// create move to fix later
Switched to traversal-based compact logic This simplifies some of the interactions between reading and writing inside the commit logic. Unfortunately this change didn't decrease code size as was initially hoped, but it does offer a nice runtime improvement for the common case and should improve debugability. Before, the compact logic required three iterations: 1. iterate through all the ids in a directory 2. scan attrs bound to each id in the directory 3. lookup attrs in the in-progress commit The code for this, while terse and complicated, did have some nice side effect. The directory lookup logic could be reused for looking up in the in-progress commit, and iterating through each id allows us to know exactly how many ids we can fit during a compact. Giving us a O(n^3) compact and O(n^3) split. However, this was complicated by a few things. First, this compact logic doesn't handle deleted attrs. To work around this, I added a marker for the last commit (or first based on your perspective) which would indicate if a delete should be copied over. This worked but was a bit hacky and meant deletes weren't cleaned up on the first compact. Second, we can't actually figure out our compacted size until we compact. This worked ok except for the fact that splits will always have a failed compact. This means we waste an erase which could very expensive. It is possible to work around this by keeping our work, but with only a single prog cache this was very tricky and also somewhat hacky. Third, the interactions between reading and writing to the same block were tricky and error-prone. They should mostly be working now, but seeing this requirement go away does not make me sad. The new compact logic fixes these issues by moving the complexity into a general-purpose lfs_dir_traverse function which has much fewer side effects on the system. We can even use it for dry-runs to precompute our estimated size. How does it work? 1. iterate through all attr in the directory 2. for each attr, scan the rest of the directory to figure out the attr's history, this will change the attr based on dir modifications and may even exit early if the attr was deleted. The end result is a traversal function that gives us the resulting state of each attr in only O(n^2). To make this complete, we allow a bounded recursion into mcu-side move attrs, although this ends up being O(n^3) unlike moves in the original solution (however moves are less common. This gives us a nice traversal function we can use for compacts and moves, handles deletes, and is overall simpler to reason about. Two minor hiccups: 1. We need to handle create attrs specially, since this algorithm doesn't care or id order, which can cause problems since attr insertion are order sensitive. We can fix this by simply looking up each create (since there is only one per file) in order at the beginning of our traversal. This is oddly complimentary to the move logic, which also handles create attrs separately. 2. We no longer know exactly how many ids we can write to a dir during splits. However, since we can do a dry-run traversal, we can use that to simply binary search for the mid-point. This gives us a O(n^2) compact and O(n^2 log n) split, which is a nice minor improvement (remember n is bounded by block size).
2018-12-27 02:27:34 +00:00
uint16_t newoldtagid = lfs_tag_id(oldtag);
if (lfs_pair_cmp(oldcwd.pair, newcwd.pair) == 0 &&
prevtag == LFS_ERR_NOENT && newid <= newoldtagid) {
// there is a small chance we are being renamed in the same directory
// to an id less than our old id, the global update to handle this
// is a bit messy
newoldtagid += 1;
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_prepmove(lfs, newoldtagid, oldcwd.pair);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// move over all attributes
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &newcwd, LFS_MKATTRS(
{prevtag != LFS_ERR_NOENT
? LFS_MKTAG(LFS_TYPE_DELETE, newid, 0)
Enabled -Wextra This only required adding NULLs where commit statements were not fully initialized. Unfortunately we still need -Wno-missing-field-initializers because of a bug in GCC that persists on Travis. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60784 Found by apmorton
2019-02-12 06:11:01 +00:00
: LFS_MKTAG(LFS_FROM_NOOP, 0, 0), NULL},
{LFS_MKTAG(LFS_TYPE_CREATE, newid, 0), NULL},
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
{LFS_MKTAG(lfs_tag_type3(oldtag), newid, strlen(newpath)),
newpath},
{LFS_MKTAG(LFS_FROM_MOVE, newid, lfs_tag_id(oldtag)), &oldcwd}));
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// let commit clean up after move (if we're different! otherwise move
// logic already fixed it for us)
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
if (lfs_pair_cmp(oldcwd.pair, newcwd.pair) != 0) {
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &oldcwd, NULL, 0);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (prevtag != LFS_ERR_NOENT && lfs_tag_type3(prevtag) == LFS_TYPE_DIR) {
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// fix orphan
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -1);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = lfs_fs_pred(lfs, prevdir.pair, &newcwd);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err) {
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
err = lfs_dir_drop(lfs, &newcwd, &prevdir);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
return 0;
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
lfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path,
uint8_t type, void *buffer, lfs_size_t size) {
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &cwd, &path, NULL);
if (tag < 0) {
return tag;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
uint16_t id = lfs_tag_id(tag);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (id == 0x3ff) {
Dropped lfs_fs_getattr for the more implicit lfs_getattr("/") This was a pretty simple oversight on my part. Conceptually, there's no difference between lfs_fs_getattr and lfs_getattr("/"). Any operations on directories can be applied "globally" by referring to the root directory. Implementation wise, this actually fixes the "corner case" of storing attributes on the root directory, which is broken since the root directory doesn't have a related entry. Instead we need to use the root superblock for this purpose. Fewer functions means less code to document and maintain, so this is a nice benefit. Now we just have a single lfs_getattr/setattr/removeattr set of functions along with the ability to access attributes atomically in lfs_file_opencfg.
2018-08-12 04:05:52 +00:00
// special case for root
id = 0;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
tag = lfs_dir_get(lfs, &cwd, LFS_MKTAG(0x7ff, 0x3ff, 0),
LFS_MKTAG(LFS_TYPE_USERATTR + type,
id, lfs_min(size, lfs->attr_max)),
Dropped lfs_fs_getattr for the more implicit lfs_getattr("/") This was a pretty simple oversight on my part. Conceptually, there's no difference between lfs_fs_getattr and lfs_getattr("/"). Any operations on directories can be applied "globally" by referring to the root directory. Implementation wise, this actually fixes the "corner case" of storing attributes on the root directory, which is broken since the root directory doesn't have a related entry. Instead we need to use the root superblock for this purpose. Fewer functions means less code to document and maintain, so this is a nice benefit. Now we just have a single lfs_getattr/setattr/removeattr set of functions along with the ability to access attributes atomically in lfs_file_opencfg.
2018-08-12 04:05:52 +00:00
buffer);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
if (tag < 0) {
if (tag == LFS_ERR_NOENT) {
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
return LFS_ERR_NOATTR;
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return tag;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
return lfs_tag_size(tag);
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
static int lfs_commitattr(lfs_t *lfs, const char *path,
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
uint8_t type, const void *buffer, lfs_size_t size) {
lfs_mdir_t cwd;
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_stag_t tag = lfs_dir_find(lfs, &cwd, &path, NULL);
if (tag < 0) {
return tag;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
uint16_t id = lfs_tag_id(tag);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (id == 0x3ff) {
Dropped lfs_fs_getattr for the more implicit lfs_getattr("/") This was a pretty simple oversight on my part. Conceptually, there's no difference between lfs_fs_getattr and lfs_getattr("/"). Any operations on directories can be applied "globally" by referring to the root directory. Implementation wise, this actually fixes the "corner case" of storing attributes on the root directory, which is broken since the root directory doesn't have a related entry. Instead we need to use the root superblock for this purpose. Fewer functions means less code to document and maintain, so this is a nice benefit. Now we just have a single lfs_getattr/setattr/removeattr set of functions along with the ability to access attributes atomically in lfs_file_opencfg.
2018-08-12 04:05:52 +00:00
// special case for root
id = 0;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
}
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
return lfs_dir_commit(lfs, &cwd, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_USERATTR + type, id, size), buffer}));
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
int lfs_setattr(lfs_t *lfs, const char *path,
uint8_t type, const void *buffer, lfs_size_t size) {
if (size > lfs->attr_max) {
return LFS_ERR_NOSPC;
}
return lfs_commitattr(lfs, path, type, buffer, size);
}
int lfs_removeattr(lfs_t *lfs, const char *path, uint8_t type) {
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return lfs_commitattr(lfs, path, type, NULL, 0x3ff);
Added deletion to custom attributes This follows from enabling tag deletion, however does require some consideration with the APIs. Now we can remove custom attributes, as well as determine if an attribute exists or not.
2018-09-09 23:48:18 +00:00
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
/// Filesystem operations ///
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
lfs->cfg = cfg;
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
int err = 0;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
// check that block size is a multiple of cache size is a multiple
// of prog and read sizes
LFS_ASSERT(lfs->cfg->cache_size % lfs->cfg->read_size == 0);
LFS_ASSERT(lfs->cfg->cache_size % lfs->cfg->prog_size == 0);
LFS_ASSERT(lfs->cfg->block_size % lfs->cfg->cache_size == 0);
// check that the block size is large enough to fit ctz pointers
LFS_ASSERT(4*lfs_npw2(0xffffffff / (lfs->cfg->block_size-2*4))
<= lfs->cfg->block_size);
Fixed several small issues - Fixed uninitialized values found by valgrind. - Fixed uninitialized value in lfs_dir_fetchmatch when handling revision counts. - Fixed mess left by lfs_dir_find when attempting to find the root directory in lfs_rename and lfs_remove. - Fixed corner case with definitions of lfs->cfg->block_cycles. - Added test cases around different forms of the root directory. I think all of these were found by TheLoneWolfling, so props!
2019-02-12 06:01:28 +00:00
// we don't support some corner cases
LFS_ASSERT(lfs->cfg->block_cycles < 0xffffffff);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup read cache
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->read_buffer) {
lfs->rcache.buffer = lfs->cfg->read_buffer;
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->rcache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->rcache.buffer) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = LFS_ERR_NOMEM;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
goto cleanup;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup program cache
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->prog_buffer) {
lfs->pcache.buffer = lfs->cfg->prog_buffer;
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->pcache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->pcache.buffer) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = LFS_ERR_NOMEM;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
goto cleanup;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Fixed information leaks through reused caches As a shortcut, littlefs never bother to zero any of the buffers is used. It didn't need to because it would always write out the entirety of the data it needed. Unfortunately, this, combined with the extra padding used to align buffers to the nearest prog size, would lead to uninitialized data getting written out to disk. This means unrelated file data could be written to different parts of storage, or worse, information leaked from the malloc calls could be written out to disk unnecessarily. found by rojer
2018-07-06 16:14:30 +00:00
// zero to avoid information leaks
lfs_cache_zero(lfs, &lfs->rcache);
lfs_cache_zero(lfs, &lfs->pcache);
Changed required alignment of lookahead_size to 64 bits This is to prepare for future compatibility with other implementations of the allocator's lookahead that are under consideration. The most promising design so far is a sort of segments-list data structure that stores pointer+size pairs, requiring 64-bits of alignment. Changing this now takes advantage of the major version to avoid a compatibility break in the future. If we end up not changing the allocator or don't need 64-bit alignment we can easily drop this requirement without breaking anyone's code.
2018-10-22 22:58:47 +00:00
// setup lookahead, must be multiple of 64-bits
Changed lookahead configuration unit to bytes instead of bits The fact that the lookahead buffer uses bits instead of bytes is an internal detail. Poking this through to the user API has caused a decent amount of confusion. Most buffers are provided as bytes and the inconsistency here can be surprising. The use of bytes instead of bits also makes us forward compatible in the case that we want to change the lookahead internal representation (hint segment list). Additionally, we change the configuration name to lookahead_size. This matches other configurations, such as cache_size and read_size, while also notifying the user that something important changed at compile time (by breaking).
2018-10-03 01:18:30 +00:00
LFS_ASSERT(lfs->cfg->lookahead_size > 0);
Added clarification on buffer alignment. In v2, the lookahead_buffer was changed from requiring 4-byte alignment to requiring 8-byte alignment. This was not documented as well as it could be, and as FabianInostroza noted, this also implies that lfs_malloc must provide 8-byte alignment. To protect against this, I've also added an assert on the alignment of both the lookahead_size and lookahead_buffer. found by FabianInostroza and amitv87
2019-04-09 23:07:44 +00:00
LFS_ASSERT(lfs->cfg->lookahead_size % 8 == 0 &&
(uintptr_t)lfs->cfg->lookahead_buffer % 8 == 0);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
if (lfs->cfg->lookahead_buffer) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer = lfs->cfg->lookahead_buffer;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
} else {
Changed lookahead configuration unit to bytes instead of bits The fact that the lookahead buffer uses bits instead of bytes is an internal detail. Poking this through to the user API has caused a decent amount of confusion. Most buffers are provided as bytes and the inconsistency here can be surprising. The use of bytes instead of bits also makes us forward compatible in the case that we want to change the lookahead internal representation (hint segment list). Additionally, we change the configuration name to lookahead_size. This matches other configurations, such as cache_size and read_size, while also notifying the user that something important changed at compile time (by breaking).
2018-10-03 01:18:30 +00:00
lfs->free.buffer = lfs_malloc(lfs->cfg->lookahead_size);
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!lfs->free.buffer) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
err = LFS_ERR_NOMEM;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
goto cleanup;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that the size limits are sane
Changed attr_max to be specific to custom attributes While technically, both system and user attributes share the same disk limitations, that's not what attr_max represents when considered from the user's perspective. To the user, attr_max applies only to custom attributes. This means attr_max should not impact other configurable limitations, such as inline files, and the ordering should be reconsidered with what the user finds most important.
2018-10-02 23:28:37 +00:00
LFS_ASSERT(lfs->cfg->name_max <= LFS_NAME_MAX);
lfs->name_max = lfs->cfg->name_max;
if (!lfs->name_max) {
lfs->name_max = LFS_NAME_MAX;
}
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
LFS_ASSERT(lfs->cfg->file_max <= LFS_FILE_MAX);
lfs->file_max = lfs->cfg->file_max;
if (!lfs->file_max) {
lfs->file_max = LFS_FILE_MAX;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Changed name of upper-limits from blah_size to blah_max This standardizes the naming between the LFS_BLAH_MAX macros and the blah_max configuration in the lfs_config structure.
2018-08-05 01:10:08 +00:00
LFS_ASSERT(lfs->cfg->attr_max <= LFS_ATTR_MAX);
lfs->attr_max = lfs->cfg->attr_max;
if (!lfs->attr_max) {
lfs->attr_max = LFS_ATTR_MAX;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// setup default state
lfs->root[0] = 0xffffffff;
lfs->root[1] = 0xffffffff;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
lfs->mlist = NULL;
Added allocation randomization for dynamic wear-leveling This implements the second step of full dynamic wear-leveling, block allocation randomization. This is the key part the uniformly distributes wear across the filesystem, even through reboots. The entropy actually comes from the filesystem itself, by xoring together all of the CRCs in the metadata-pairs on the filesystem. While this sounds like a ridiculous operation, it's easy to do when we already scan the metadata-pairs at mount time. This gives us a random number we can use for block allocation. Unfortunately it's not a great general purpose random generator as the output only changes every filesystem write. Fortunately that's exactly when we need our allocator. --- Additionally, the randomization created a mess for the testing framework. Fortunately, this method of randomization is deterministic. A very useful property for reproducing bugs.
2018-08-09 14:06:17 +00:00
lfs->seed = 0;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs->gstate = (struct lfs_gstate){0};
lfs->gpending = (struct lfs_gstate){0};
lfs->gdelta = (struct lfs_gstate){0};
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
#ifdef LFS_MIGRATE
lfs->lfs1 = NULL;
#endif
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
cleanup:
lfs_deinit(lfs);
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
return err;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
static int lfs_deinit(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// free allocated memory
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->cfg->read_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->rcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
if (!lfs->cfg->prog_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->pcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
if (!lfs->cfg->lookahead_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->free.buffer);
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
int lfs_format(lfs_t *lfs, const struct lfs_config *cfg) {
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
int err = 0;
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
// create free lookahead
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead_size);
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
lfs->free.off = 0;
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
lfs->free.size = lfs_min(8*lfs->cfg->lookahead_size,
lfs->cfg->block_count);
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
lfs->free.i = 0;
lfs_alloc_ack(lfs);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
// create root dir
lfs_mdir_t root;
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
err = lfs_dir_alloc(lfs, &root);
if (err) {
goto cleanup;
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
// write one superblock
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
lfs_superblock_t superblock = {
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
.version = LFS_DISK_VERSION,
.block_size = lfs->cfg->block_size,
.block_count = lfs->cfg->block_count,
.name_max = lfs->name_max,
.file_max = lfs->file_max,
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
.attr_max = lfs->attr_max,
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
};
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
lfs_superblock_tole32(&superblock);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &root, LFS_MKATTRS(
Enabled -Wextra This only required adding NULLs where commit statements were not fully initialized. Unfortunately we still need -Wno-missing-field-initializers because of a bug in GCC that persists on Travis. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60784 Found by apmorton
2019-02-12 06:11:01 +00:00
{LFS_MKTAG(LFS_TYPE_CREATE, 0, 0), NULL},
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
{LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, 8), "littlefs"},
{LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)),
&superblock}));
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
if (err) {
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
goto cleanup;
}
// sanity check that fetch works
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
2018-10-21 02:02:25 +00:00
err = lfs_dir_fetch(lfs, &root, (const lfs_block_t[2]){0, 1});
Added -Wjump-misses-init and fixed uninitialized warnings
2018-09-26 15:11:40 +00:00
if (err) {
goto cleanup;
}
Added redundant compaction to lfs_format/lfs_migrate This ensures that both blocks in the superblock pair are written with the superblock info. While this does use an additional erase cycle, it prevents older versions of littlefs from accidentally being picked up in the case that the disk is mounted on a system that doesn't support the newer version. This does bring back the risk of picking up old littlefs versions on a disk that has been formatted with a filesystem that doesn't use block 2 (such as FAT), but this risk already exists, and moving between versions of littlefs is more likely with the recent v1 -> v2 update. Suggested by rojer
2019-05-28 23:16:51 +00:00
// force compaction to prevent accidentally mounting any
// older version of littlefs that may live on disk
root.erased = false;
err = lfs_dir_commit(lfs, &root, NULL, 0);
if (err) {
goto cleanup;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
cleanup:
lfs_deinit(lfs);
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// scan directory blocks for superblock and any global updates
lfs_mdir_t dir = {.tail = {0, 1}};
while (!lfs_pair_isnull(dir.tail)) {
// fetch next block in tail list
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t tag = lfs_dir_fetchmatch(lfs, &dir, dir.tail,
LFS_MKTAG(0x7ff, 0x3ff, 0),
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, 8),
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
NULL,
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
lfs_dir_find_match, &(struct lfs_dir_find_match){
lfs, "littlefs", 8});
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
if (tag < 0) {
err = tag;
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
goto cleanup;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// has superblock?
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
if (tag && !lfs_tag_isdelete(tag)) {
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// update root
lfs->root[0] = dir.pair[0];
lfs->root[1] = dir.pair[1];
// grab superblock
lfs_superblock_t superblock;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
tag = lfs_dir_get(lfs, &dir, LFS_MKTAG(0x7ff, 0x3ff, 0),
Fixed issue with creating files named "littlefs" A rather humorous issue, we accidentally ended up mixing our file namespace with our superblocks. This meant if we created a file named "littlefs" it would reference the superblock and all sorts of things would break. Fixing this also highlighted another issue, the fact that the superblock always needs to come before any file entries in the directory. I didn't account for this in the initial B-tree design, but we need a higher ordering for superblocks + children + files than just name. To fix this I added ordering information in the 2 bits currently unused in the tag type. Though note that the size of these fields are flexible. 9-bit type field: [--- 9 ---] [1|- 3 -|- 2 -|- 3 -] ^ ^ ^ ^- type-specific info | | \------- ordering info | \------------- subtype \----------------- user bit
2018-10-04 21:25:19 +00:00
LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)),
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
&superblock);
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
if (tag < 0) {
err = tag;
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
goto cleanup;
}
lfs_superblock_fromle32(&superblock);
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// check version
uint16_t major_version = (0xffff & (superblock.version >> 16));
uint16_t minor_version = (0xffff & (superblock.version >> 0));
if ((major_version != LFS_DISK_VERSION_MAJOR ||
minor_version > LFS_DISK_VERSION_MINOR)) {
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
LFS_ERROR("Invalid version %"PRIu16".%"PRIu16,
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
major_version, minor_version);
err = LFS_ERR_INVAL;
goto cleanup;
}
Added root entry and expanding superblocks Expanding superblocks has been on my wishlist for a while. The basic idea is that instead of maintaining a fixed offset blocks {0, 1} to the the root directory (1 pointer), we maintain a dynamically sized linked-list of superblocks that point to the actual root. If the number of writes to the root exceeds some value, we increase the size of the superblock linked-list. This can leverage existing metadata-pair operations. The revision count for metadata-pairs provides some knowledge on how much wear we've put on the superblock, and the threaded linked-list can also be reused for this purpose. This means superblock expansion is both optional and cheap to implement. Expanding superblocks helps both extremely small and extremely large filesystem (extreme being relative of course). On the small end, we can actually collapse the superblock into the root directory and drop the hard requirement of 4-blocks for the superblock. On the large end, our superblock will now last longer than the rest of the filesystem. Each time we expand, the number of cycles until the superblock dies is increased by a power. Before we were stuck with this layout: level cycles limit layout 1 E^2 390 MiB s0 -> root Now we expand every time a fixed offset is exceeded: level cycles limit layout 0 E 4 KiB s0+root 1 E^2 390 MiB s0 -> root 2 E^3 37 TiB s0 -> s1 -> root 3 E^4 3.6 EiB s0 -> s1 -> s2 -> root ... Where the cycles are the number of cycles before death, and the limit is the worst-case size a filesystem where early superblock death becomes a concern (all writes to root using this formula: E^|s| = E*B, E = erase cycles = 100000, B = block count, assuming 4096 byte blocks). Note we can also store copies of the superblock entry on the expanded superblocks. This may help filesystem recover tools in the future.
2018-08-06 18:30:51 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
// check superblock configuration
if (superblock.name_max) {
if (superblock.name_max > lfs->name_max) {
LFS_ERROR("Unsupported name_max (%"PRIu32" > %"PRIu32")",
superblock.name_max, lfs->name_max);
err = LFS_ERR_INVAL;
goto cleanup;
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
lfs->name_max = superblock.name_max;
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
if (superblock.file_max) {
if (superblock.file_max > lfs->file_max) {
LFS_ERROR("Unsupported file_max (%"PRIu32" > %"PRIu32")",
superblock.file_max, lfs->file_max);
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
err = LFS_ERR_INVAL;
goto cleanup;
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
Added support for RAM-independent reading of inline files One of the new features in LittleFS is "inline files", which is the inlining of small files in the parent directory. Inline files have a big limitation in that they no longer have a dedicated scratch area to write out data before commit-time. This is fine as long as inline files are small enough to fit in RAM. However, this dependency on RAM creates an uncomfortable situation for portability, with larger devices able to create larger files than smaller devices. This problem is especially important on embedded systems, where RAM is at a premium. Recently, I realized this RAM requirement is necessary for _writing_ inline files, but not for _reading_ inline files. By allowing fetches of specific slices of inline files it's possible to read inline files without the RAM to back it. However however, this creates a conflict with COW semantics. Normally, when a file is open twice, it is referenced by a COW data structure that can be updated independently. Inlines files that fit in RAM also allows independent updates, but the moment an inline file can't fit in RAM, any updates to that directory block could corrupt open files referencing the inline file. The fact that this behaviour is only inconsistent for inline files created on a different device with more RAM creates a potential nightmare for user experience. Fortunately, there is a workaround for this. When we are commiting to a directory, any open files needs to live in a COW structure or in RAM. While we could move large inline files to COW structures at open time, this would break the separation of read/write operations and could lead to write errors at read time (ie ENOSPC). But since this is only an issue for commits, we can defer the move to a COW structure to any commits to that directory. This means when committing to a directory we need to find any _open_ large inline files and evict them from the directory, leaving the file with a new COW structure even if it was opened read only. While complicated, the end result is inline files that can use the MAX RAM that is available, but can be read with MIN RAM, even with multiple write operations happening to the underlying directory block. This prevents users from needing to learn the idiosyncrasies of inline files to use the filesystem portably.
2019-01-13 17:08:42 +00:00
lfs->file_max = superblock.file_max;
Combined superblock scan and fetch of xored-globals during mount Conceptually these are two separate operations. However, they are both only needed during mount, both require iteration over the linked-list of metadata-pairs, and both are independent from each other. Combining these into one gives us a nice code savings. Additionally, this greatly simplifies the lookup of the root directory. Initially we used a flag to indicate which superblock was root, since we didn't want to fetch more pairs than we needed to. But since we're going to fetch all metadata-pairs anyways, we can just use the last superblock we find as the indicator of our root directory.
2018-09-12 06:50:21 +00:00
}
Changed attr_max to be specific to custom attributes While technically, both system and user attributes share the same disk limitations, that's not what attr_max represents when considered from the user's perspective. To the user, attr_max applies only to custom attributes. This means attr_max should not impact other configurable limitations, such as inline files, and the ordering should be reconsidered with what the user finds most important.
2018-10-02 23:28:37 +00:00
if (superblock.attr_max) {
if (superblock.attr_max > lfs->attr_max) {
LFS_ERROR("Unsupported attr_max (%"PRIu32" > %"PRIu32")",
superblock.attr_max, lfs->attr_max);
err = LFS_ERR_INVAL;
goto cleanup;
}
lfs->attr_max = superblock.attr_max;
}
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// has gstate?
err = lfs_dir_getgstate(lfs, &dir, &lfs->gpending);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
return err;
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
}
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
}
Fixed issue with creating files named "littlefs" A rather humorous issue, we accidentally ended up mixing our file namespace with our superblocks. This meant if we created a file named "littlefs" it would reference the superblock and all sorts of things would break. Fixing this also highlighted another issue, the fact that the superblock always needs to come before any file entries in the directory. I didn't account for this in the initial B-tree design, but we need a higher ordering for superblocks + children + files than just name. To fix this I added ordering information in the 2 bits currently unused in the tag type. Though note that the size of these fields are flexible. 9-bit type field: [--- 9 ---] [1|- 3 -|- 2 -|- 3 -] ^ ^ ^ ^- type-specific info | | \------- ordering info | \------------- subtype \----------------- user bit
2018-10-04 21:25:19 +00:00
// found superblock?
if (lfs_pair_isnull(lfs->root)) {
err = LFS_ERR_INVAL;
goto cleanup;
}
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
// update littlefs with gstate
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs->gpending.tag += !lfs_tag_isvalid(lfs->gpending.tag);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs->gstate = lfs->gpending;
if (lfs_gstate_hasmove(&lfs->gstate)) {
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
LFS_DEBUG("Found move %"PRIu32" %"PRIu32" %"PRIu16,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs->gstate.pair[0],
lfs->gstate.pair[1],
lfs_tag_id(lfs->gstate.tag));
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Added allocation randomization for dynamic wear-leveling This implements the second step of full dynamic wear-leveling, block allocation randomization. This is the key part the uniformly distributes wear across the filesystem, even through reboots. The entropy actually comes from the filesystem itself, by xoring together all of the CRCs in the metadata-pairs on the filesystem. While this sounds like a ridiculous operation, it's easy to do when we already scan the metadata-pairs at mount time. This gives us a random number we can use for block allocation. Unfortunately it's not a great general purpose random generator as the output only changes every filesystem write. Fortunately that's exactly when we need our allocator. --- Additionally, the randomization created a mess for the testing framework. Fortunately, this method of randomization is deterministic. A very useful property for reproducing bugs.
2018-08-09 14:06:17 +00:00
// setup free lookahead
lfs->free.off = lfs->seed % lfs->cfg->block_size;
lfs->free.size = 0;
lfs->free.i = 0;
lfs_alloc_ack(lfs);
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return 0;
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
cleanup:
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
lfs_unmount(lfs);
Fix memory leaks in lfs_mount and lfs_format Squashed: - Change lfs_deinit() return to void to simplify error handling - Move lfs_deinit() before lfs_init() - Fix memory leaks in lfs_init() - Fix memory leaks in lfs_format() - Fix memory leaks in lfs_mount()
2018-07-18 14:50:00 +00:00
return err;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
int lfs_unmount(lfs_t *lfs) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return lfs_deinit(lfs);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
/// Filesystem filesystem operations ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_fs_traverse(lfs_t *lfs,
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int (*cb)(void *data, lfs_block_t block), void *data) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// iterate over metadata pairs
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir = {.tail = {0, 1}};
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
#ifdef LFS_MIGRATE
// also consider v1 blocks during migration
if (lfs->lfs1) {
int err = lfs1_traverse(lfs, cb, data);
if (err) {
return err;
}
dir.tail[0] = lfs->root[0];
dir.tail[1] = lfs->root[1];
}
#endif
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
while (!lfs_pair_isnull(dir.tail)) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
for (int i = 0; i < 2; i++) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int err = cb(data, dir.tail[i]);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
// iterate through ids in directory
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
for (uint16_t id = 0; id < dir.count; id++) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
struct lfs_ctz ctz;
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t tag = lfs_dir_get(lfs, &dir, LFS_MKTAG(0x700, 0x3ff, 0),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
LFS_MKTAG(LFS_TYPE_STRUCT, id, sizeof(ctz)), &ctz);
if (tag < 0) {
if (tag == LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
continue;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return tag;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_ctz_fromle32(&ctz);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (lfs_tag_type3(tag) == LFS_TYPE_CTZSTRUCT) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
err = lfs_ctz_traverse(lfs, NULL, &lfs->rcache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
ctz.head, ctz.size, cb, data);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
}
// iterate over any open files
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
for (lfs_file_t *f = (lfs_file_t*)lfs->mlist; f; f = f->next) {
if (f->type != LFS_TYPE_REG) {
continue;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if ((f->flags & LFS_F_DIRTY) && !(f->flags & LFS_F_INLINE)) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_traverse(lfs, &f->cache, &lfs->rcache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
f->ctz.head, f->ctz.size, cb, data);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
if ((f->flags & LFS_F_WRITING) && !(f->flags & LFS_F_INLINE)) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
int err = lfs_ctz_traverse(lfs, &f->cache, &lfs->rcache,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
f->block, f->pos, cb, data);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
return 0;
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_pred(lfs_t *lfs,
const lfs_block_t pair[2], lfs_mdir_t *pdir) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// iterate over all directory directory entries
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
pdir->tail[0] = 0;
pdir->tail[1] = 1;
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
while (!lfs_pair_isnull(pdir->tail)) {
if (lfs_pair_cmp(pdir->tail, pair) == 0) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return 0;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, pdir, pdir->tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return LFS_ERR_NOENT;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
struct lfs_fs_parent_match {
lfs_t *lfs;
const lfs_block_t pair[2];
};
static int lfs_fs_parent_match(void *data,
lfs_tag_t tag, const void *buffer) {
struct lfs_fs_parent_match *find = data;
lfs_t *lfs = find->lfs;
const struct lfs_diskoff *disk = buffer;
(void)tag;
lfs_block_t child[2];
int err = lfs_bd_read(lfs,
&lfs->pcache, &lfs->rcache, lfs->cfg->block_size,
disk->block, disk->off, &child, sizeof(child));
if (err) {
return err;
}
lfs_pair_fromle32(child);
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return (lfs_pair_cmp(child, find->pair) == 0) ? LFS_CMP_EQ : LFS_CMP_LT;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
}
static lfs_stag_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t pair[2],
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
lfs_mdir_t *parent) {
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
// use fetchmatch with callback to find pairs
parent->tail[0] = 0;
parent->tail[1] = 1;
while (!lfs_pair_isnull(parent->tail)) {
lfs_stag_t tag = lfs_dir_fetchmatch(lfs, parent, parent->tail,
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
LFS_MKTAG(0x7ff, 0, 0x3ff),
LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 0, 8),
NULL,
Switched to strongly ordered directories Instead of storing files in an arbitrary order, we now store files in ascending lexicographical order by filename. Although a big change, this actually has little impact on how littlefs works internally. We need to support file insertion, and compare file names to find our position. But since we already need to scan the entire directory block, this adds relatively little overhead. What this does allow, is the potential to add B-tree support in the future in a backwards compatible manner. How could you add B-trees to littlefs? 1. Add an optional "child" tag with a pointer that allows you to skip to a position in the metadata-pair list that composes the directory 2. When splitting a metadata-pair (sound familiar?), we either insert a second child tag in our parent, or we create a new root containing the child tags. 3. Each layer needs a bit stored in the tail-pointer to indicate if we're going to the next layer. This can be created trivially when we create a new root. 4. During lookup we keep two pointers containing the bounds of our search. We may need to iterate through multiple metadata-pairs in our linked-list, but this gives us a O(log n) lookup cost in a balanced tree. 5. During deletion we also delete any children pointers. Note that children pointers must come before the actual file entry. This gives us a B-tree implementation that is compatible with the current directory layout (assuming the files are ordered). This means that B-trees could be supported by a host PC and ignored on a small device. And during power-loss, we never end up with a broken filesystem, just a less-than-optimal tree. Note that we don't handle removes, so it's possible for a tree to become unbalanced. But worst case that's the same as the current linked-list implementation. All we need to do now is keep directories ordered. If we decide to drop B-tree support in the future or the B-tree implementation turns out inherently flawed, we can just drop the ordered requirement without breaking compatibility and recover the code cost.
2018-10-04 19:49:34 +00:00
lfs_fs_parent_match, &(struct lfs_fs_parent_match){
lfs, {pair[0], pair[1]}});
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
if (tag && tag != LFS_ERR_NOENT) {
Added root entry and expanding superblocks Expanding superblocks has been on my wishlist for a while. The basic idea is that instead of maintaining a fixed offset blocks {0, 1} to the the root directory (1 pointer), we maintain a dynamically sized linked-list of superblocks that point to the actual root. If the number of writes to the root exceeds some value, we increase the size of the superblock linked-list. This can leverage existing metadata-pair operations. The revision count for metadata-pairs provides some knowledge on how much wear we've put on the superblock, and the threaded linked-list can also be reused for this purpose. This means superblock expansion is both optional and cheap to implement. Expanding superblocks helps both extremely small and extremely large filesystem (extreme being relative of course). On the small end, we can actually collapse the superblock into the root directory and drop the hard requirement of 4-blocks for the superblock. On the large end, our superblock will now last longer than the rest of the filesystem. Each time we expand, the number of cycles until the superblock dies is increased by a power. Before we were stuck with this layout: level cycles limit layout 1 E^2 390 MiB s0 -> root Now we expand every time a fixed offset is exceeded: level cycles limit layout 0 E 4 KiB s0+root 1 E^2 390 MiB s0 -> root 2 E^3 37 TiB s0 -> s1 -> root 3 E^4 3.6 EiB s0 -> s1 -> s2 -> root ... Where the cycles are the number of cycles before death, and the limit is the worst-case size a filesystem where early superblock death becomes a concern (all writes to root using this formula: E^|s| = E*B, E = erase cycles = 100000, B = block count, assuming 4096 byte blocks). Note we can also store copies of the superblock entry on the expanded superblocks. This may help filesystem recover tools in the future.
2018-08-06 18:30:51 +00:00
return tag;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Modified results from find-like functions to use tags Tags offer all of the necessary info from the find functions (which makes sense, this is the structure that stores the info on disk). Passing around a single tag instead of separate id and type fields simplifies the internal functions while leverages the tag's compactness.
2018-07-11 12:18:30 +00:00
return LFS_ERR_NOENT;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_relocate(lfs_t *lfs,
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
const lfs_block_t oldpair[2], lfs_block_t newpair[2]) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// update internal root
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
if (lfs_pair_cmp(oldpair, lfs->root) == 0) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
LFS_DEBUG("Relocating root %"PRIu32" %"PRIu32,
newpair[0], newpair[1]);
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
lfs->root[0] = newpair[0];
lfs->root[1] = newpair[1];
}
// update internally tracked dirs
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
for (struct lfs_mlist *d = lfs->mlist; d; d = d->next) {
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
if (lfs_pair_cmp(oldpair, d->m.pair) == 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
d->m.pair[0] = newpair[0];
d->m.pair[1] = newpair[1];
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// find parent
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t parent;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_stag_t tag = lfs_fs_parent(lfs, oldpair, &parent);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag < 0 && tag != LFS_ERR_NOENT) {
return tag;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag != LFS_ERR_NOENT) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// update disk, this creates a desync
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, +1);
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_tole32(newpair);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
int err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS({tag, newpair}));
Standardized naming for internal functions - lfs_pairblah -> lfs_pair_blah - lfs_ctzblah -> lfs_ctz_blah - lfs_tagblah -> lfs_tag_blah - lfs_globalblah -> lfs_global_blah - lfs_commitblah -> lfs_commit_blah
2018-08-05 04:57:43 +00:00
lfs_pair_fromle32(newpair);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// next step, clean up orphans
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -1);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
// find pred
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_pred(lfs, oldpair, &parent);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// if we can't find dir, it must be new
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err != LFS_ERR_NOENT) {
Collapsed recursive deorphans into a single pass Because a block can go bad at any time, if we're unlucky, we may end up generating multiple orphans in a single metadata write. This is exacerbated by the early eviction in dynamic wear-leveling. We can't track _all_ orphans, because that would require unbounded storage and significantly complicate things, but there are a handful of intentional orphans we do track because they are easy to resolve without the O(n^2) deorphan scan. These are anytime we intentionally remove a metadata-pair. Initially we cleaned up orphans as they occur with whatever knowledge we do have, and just accepted the extra O(n^2) deorphan scans in the unlucky case. However we can do a bit better by being lazy and leaving deorphaning up to the next metadata write. This needs to work with the known orphans while still setting the orphan flag on disk correctly. To accomplish this we replace the internal flag with a small counter. Note, this means that our internal representation of orphans differs from what's on disk. This is annoying but not the end of the world.
2018-08-13 14:03:13 +00:00
// replace bad pair, either we clean up desync, or no desync occured
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_tole32(newpair);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &parent, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_TAIL + parent.split, 0x3ff, 8), newpair}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(newpair);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (err) {
return err;
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return 0;
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
static void lfs_fs_preporphans(lfs_t *lfs, int8_t orphans) {
lfs->gpending.tag += orphans;
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs_gstate_xororphans(&lfs->gdelta, &lfs->gpending,
lfs_gstate_hasorphans(&lfs->gpending));
lfs_gstate_xororphans(&lfs->gpending, &lfs->gpending,
lfs_gstate_hasorphans(&lfs->gpending));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
static void lfs_fs_prepmove(lfs_t *lfs,
uint16_t id, const lfs_block_t pair[2]) {
Fixed issue with global state updates being lost during relocates Caught during power resilience testing, this was a bug that only occurs when we need to compact in the middle of a move commit and we find that the destination block is bad, forcing a relocate. This series of events would cause littlefs to clear the "gpending" state in preparation for fixing the move atomically, but this fix never gets written out because of the relocate. The fix here is to separate the update to the "gdelta" and "gpending" state, marking "gdelta" in preparation for the move, but waiting to update "gpending" until after our commit completes. This keeps our disk state in sync without prematurely dropping moves.
2019-01-22 22:09:51 +00:00
lfs_gstate_xormove(&lfs->gdelta, &lfs->gpending, id, pair);
lfs_gstate_xormove(&lfs->gpending, &lfs->gpending, id, pair);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
}
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
static int lfs_fs_demove(lfs_t *lfs) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (!lfs_gstate_hasmove(&lfs->gstate)) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
return 0;
}
// Fix bad moves
Fixed several small issues found during wider testing - Fixed cache tarnishing issue where flush did not clean up read caches - Removed extra alloc acks which would prevent file relocations from resolving on an exhausted filesystem - Removed unsigned comparison < 0 from changed in file seek - Fixed bug in lfs_dir_getslice with using gtag's size - Removed warnings around PRIu32 used 16-bit types in debug info
2019-01-22 22:21:16 +00:00
LFS_DEBUG("Fixing move %"PRIu32" %"PRIu32" %"PRIu16,
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs->gstate.pair[0],
lfs->gstate.pair[1],
lfs_tag_id(lfs->gstate.tag));
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
// fetch and delete the moved entry
lfs_mdir_t movedir;
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
int err = lfs_dir_fetch(lfs, &movedir, lfs->gstate.pair);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
return err;
}
// rely on cancel logic inside commit
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &movedir, NULL, 0);
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
if (err) {
return err;
}
return 0;
}
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
static int lfs_fs_deorphan(lfs_t *lfs) {
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
if (!lfs_gstate_hasorphans(&lfs->gstate)) {
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
return 0;
}
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// Fix any orphans
lfs_mdir_t pdir = {.split = true};
lfs_mdir_t dir = {.tail = {0, 1}};
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// iterate over all directory directory entries
while (!lfs_pair_isnull(dir.tail)) {
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// check head blocks for orphans
if (!pdir.split) {
// check if we have a parent
lfs_mdir_t parent;
Revisited some generic concepts, callbacks, and some reorganization - Callbacks for get/match, this does have a code cost, but allows more code reuse, which almost balances out the code cost, but also reduces maintenance and increased flexibility. Also callbacks may be able to be gc-ed in some cases. - Consistent struct vs _t usage, _t for external-facing struct that shouldn't be messed with outside the library. structs for external and internal structs where anyone with access is allowed to modify. - Reorganized several high-level function groups - Inlined structures that didn't need separate definitions in header
2018-09-11 03:07:59 +00:00
lfs_stag_t tag = lfs_fs_parent(lfs, pdir.tail, &parent);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (tag < 0 && tag != LFS_ERR_NOENT) {
return tag;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (tag == LFS_ERR_NOENT) {
// we are an orphan
LFS_DEBUG("Fixing orphan %"PRIu32" %"PRIu32,
pdir.tail[0], pdir.tail[1]);
Removed xored-globals from the mdir struct The xored-globals have a very large footprint. In the worst case, the xored-globals are stored on each metadata-pair, twice in memory. They must be very small, but are also very useful, so at risk of growing in the future (hint global free-list?). Initially we also stored a copy in each mdir structure, since this avoided extra disk access to look up the globals when we need to modify the global state on a metadata-pair. But we can easily just fetch the globals when needed. This is more costly in terms of runtime, but reduces RAM impact of globals, which was previously needed for each open dir and file.
2018-09-12 06:34:03 +00:00
err = lfs_dir_drop(lfs, &pdir, &dir);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
break;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
lfs_block_t pair[2];
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
lfs_stag_t res = lfs_dir_get(lfs, &parent,
LFS_MKTAG(0x7ff, 0x3ff, 0), tag, pair);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (res < 0) {
return res;
}
lfs_pair_fromle32(pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (!lfs_pair_sync(pair, pdir.tail)) {
// we have desynced
LFS_DEBUG("Fixing half-orphan %"PRIu32" %"PRIu32,
pair[0], pair[1]);
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_tole32(pair);
Changed custom attribute descriptors to used arrays While linked-lists do have some minor benefits, arrays are more idiomatic in C and may provide a more intuitive API. Initially the linked-list approach was more beneficial than it is now, since it allowed custom attributes to be chained to internal linked lists of attributes. However, this was dropped because exposing the internal attribute list in this way created a rather messy user interface that required strictly encoding the attributes with the on-disk tag format. Minor downside, users can no longer introduce custom attributes in different layers (think OS vs app). Minor upside, the code size and stack usage was reduced a bit. Fortunately, this API can always be changed in the future without breaking anything (except maybe API compatibility).
2019-01-08 14:52:03 +00:00
err = lfs_dir_commit(lfs, &pdir, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 8), pair}));
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_pair_fromle32(pair);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
break;
}
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
memcpy(&pdir, &dir, sizeof(pdir));
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
// mark orphans as fixed
Modified global state format to work with new tag format The main difference here is a change from encoding "hasorphans" and "hasmove" bits in the tag itself. This worked with the old format, but in the new format the space these bits take up must be consistent for each tag type. The tradeoff is that the new tag format allows for up to 256 different global states which may be useful in the future (for example, a global free list). The new format encodes this info in the data blob, using an additional word of storage. This word is actually formatted the same as though it was a tag, which simplified internal handling and may allow other tag types in the future. Format for global state: [---- 96 bits ----] [1|- 11 -|- 10 -|- 10 -|--- 64 ---] ^ ^ ^ ^ ^- move dir pair | | | \-------------------------- unused, must be 0s | | \--------------------------------- move id | \---------------------------------------- type, 0xfff for move \--------------------------------------------- has orphans This also included another iteration over globals (renamed to gstate) with some simplifications to how globals are handled.
2019-01-04 23:23:36 +00:00
lfs_fs_preporphans(lfs, -lfs_gstate_getorphans(&lfs->gstate));
lfs->gstate = lfs->gpending;
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
return 0;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
static int lfs_fs_forceconsistency(lfs_t *lfs) {
int err = lfs_fs_demove(lfs);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Revisited xored-globals and related logic Added separate bit for "hasmove", which means we don't need to check the move id, and allows us to add more sync-related global states in the future, as long as they never happen simultaneously (such as orphans and moves). Also refactored some of the logic and removed the union in the global structure, which didn't really add anything of value.
2018-09-15 03:02:39 +00:00
err = lfs_fs_deorphan(lfs);
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added building blocks for dynamic wear-leveling Initially, littlefs relied entirely on bad-block detection for wear-leveling. Conceptually, at the end of a devices lifespan, all blocks would be worn evenly, even if they weren't worn out at the same time. However, this doesn't work for all devices, rather than causing corruption during writes, wear reduces a devices "sticking power", causing bits to flip over time. This means for many devices, true wear-leveling (dynamic or static) is required. Fortunately, way back at the beginning, littlefs was designed to do full dynamic wear-leveling, only dropping it when making the retrospectively short-sighted realization that bad-block detection is theoretically sufficient. We can enable dynamic wear-leveling with only a few tweaks to littlefs. These can be implemented without breaking backwards compatibility. 1. Evict metadata-pairs after a certain number of writes. Eviction in this case is identical to a relocation to recover from a bad block. We move our data and stick the old block back into our pool of blocks. For knowing when to evict, we already have a revision count for each metadata-pair which gives us enough information. We add the configuration option block_cycles and evict when our revision count is a multiple of this value. 2. Now all blocks participate in COW behaviour. However we don't store the state of our allocator, so every boot cycle we reuse the first blocks on storage. This is very bad on a microcontroller, where we may reboot often. We need a way to spread our usage across the disk. To pull this off, we can simply randomize which block we start our allocator at. But we need a random number generator that is different on each boot. Fortunately we have a great source of entropy, our filesystem. So we seed our block allocator with a simple hash of the CRCs on our metadata-pairs. This can be done for free since we already need to scan the metadata-pairs during mount. What we end up with is a uniform distribution of wear on storage. The wear is not perfect, if a block is used for metadata it gets more wear, and the randomization may not be exact. But we can never actually get perfect wear-leveling, since we're already resigned to dynamic wear-leveling at the file level. With the addition of metadata logging, we end up with a really interesting two-stage wear-leveling algorithm. At the low-level, metadata is statically wear-leveled. At the high-level, blocks are dynamically wear-leveled. --- This specific commit implements the first step, eviction of metadata pairs. Entertwining this into the already complicated compact logic was a bit annoying, however we can combine the logic for superblock expansion with the logic for metadata-pair eviction.
2018-08-08 21:34:56 +00:00
return 0;
}
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
static int lfs_fs_size_count(void *p, lfs_block_t block) {
Merge remote-tracking branch 'origin/master' into v2-alpha
2018-08-05 01:33:09 +00:00
(void)block;
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
lfs_size_t *size = p;
*size += 1;
return 0;
}
lfs_ssize_t lfs_fs_size(lfs_t *lfs) {
lfs_size_t size = 0;
int err = lfs_fs_traverse(lfs, lfs_fs_size_count, &size);
if (err) {
return err;
}
Cleaned up tag encoding, now with clear chunk field Before, the tag format's type field was limited to 9-bits. This sounds like a lot, but this field needed to encode up to 256 user-specified types. This limited the flexibility of the encoded types. As time went on, more bits in the type field were repurposed for various things, leaving a rather fragile type field. Here we make the jump to full 11-bit type fields. This comes at the cost of a smaller length field, however the use of the length field was always going to come with a RAM limitation. Rather than putting pressure on RAM for inline files, the new type field lets us encode a chunk number, splitting up inline files into multiple updatable units. This actually pushes the theoretical inline max from 8KiB to 256KiB! (Note that we only allow a single 1KiB chunk for now, chunky inline files is just a theoretical future improvement). Here is the new 32-bit tag format, note that there are multiple levels of types which break down into more info: [---- 32 ----] [1|-- 11 --|-- 10 --|-- 10 --] ^. ^ . ^ ^- entry length |. | . \------------ file id chunk info |. \-----.------------------ type info (type3) \.-----------.------------------ valid bit [-3-|-- 8 --] ^ ^- chunk info \------- type info (type1) Additionally, I've split the CREATE tag into separate SPLICE and NAME tags. This simplified the new compact logic a bit. For now, littlefs still follows the rule that a NAME tag precedes any other tags related to a file, but this can change in the future.
2018-12-29 13:53:12 +00:00
return size;
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
}
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
#ifdef LFS_MIGRATE
////// Migration from littelfs v1 below this //////
/// Version info ///
// Software library version
// Major (top-nibble), incremented on backwards incompatible changes
// Minor (bottom-nibble), incremented on feature additions
#define LFS1_VERSION 0x00010007
#define LFS1_VERSION_MAJOR (0xffff & (LFS1_VERSION >> 16))
#define LFS1_VERSION_MINOR (0xffff & (LFS1_VERSION >> 0))
// Version of On-disk data structures
// Major (top-nibble), incremented on backwards incompatible changes
// Minor (bottom-nibble), incremented on feature additions
#define LFS1_DISK_VERSION 0x00010001
#define LFS1_DISK_VERSION_MAJOR (0xffff & (LFS1_DISK_VERSION >> 16))
#define LFS1_DISK_VERSION_MINOR (0xffff & (LFS1_DISK_VERSION >> 0))
/// v1 Definitions ///
// File types
enum lfs1_type {
LFS1_TYPE_REG = 0x11,
LFS1_TYPE_DIR = 0x22,
LFS1_TYPE_SUPERBLOCK = 0x2e,
};
typedef struct lfs1 {
lfs_block_t root[2];
} lfs1_t;
typedef struct lfs1_entry {
lfs_off_t off;
struct lfs1_disk_entry {
uint8_t type;
uint8_t elen;
uint8_t alen;
uint8_t nlen;
union {
struct {
lfs_block_t head;
lfs_size_t size;
} file;
lfs_block_t dir[2];
} u;
} d;
} lfs1_entry_t;
typedef struct lfs1_dir {
struct lfs1_dir *next;
lfs_block_t pair[2];
lfs_off_t off;
lfs_block_t head[2];
lfs_off_t pos;
struct lfs1_disk_dir {
uint32_t rev;
lfs_size_t size;
lfs_block_t tail[2];
} d;
} lfs1_dir_t;
typedef struct lfs1_superblock {
lfs_off_t off;
struct lfs1_disk_superblock {
uint8_t type;
uint8_t elen;
uint8_t alen;
uint8_t nlen;
lfs_block_t root[2];
uint32_t block_size;
uint32_t block_count;
uint32_t version;
char magic[8];
} d;
} lfs1_superblock_t;
/// Low-level wrappers v1->v2 ///
Make lfs1_crc static so it doesn't conflict with prefixed LFS1 code When LFS1 code is present and LFS_MIGRATE is enabled
2019-05-16 16:51:22 +00:00
static void lfs1_crc(uint32_t *crc, const void *buffer, size_t size) {
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
*crc = lfs_crc(*crc, buffer, size);
}
static int lfs1_bd_read(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, void *buffer, lfs_size_t size) {
// if we ever do more than writes to alternating pairs,
// this may need to consider pcache
return lfs_bd_read(lfs, &lfs->pcache, &lfs->rcache, size,
block, off, buffer, size);
}
static int lfs1_bd_crc(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
int err = lfs1_bd_read(lfs, block, off+i, &c, 1);
if (err) {
return err;
}
lfs1_crc(crc, &c, 1);
}
return 0;
}
/// Endian swapping functions ///
static void lfs1_dir_fromle32(struct lfs1_disk_dir *d) {
d->rev = lfs_fromle32(d->rev);
d->size = lfs_fromle32(d->size);
d->tail[0] = lfs_fromle32(d->tail[0]);
d->tail[1] = lfs_fromle32(d->tail[1]);
}
static void lfs1_dir_tole32(struct lfs1_disk_dir *d) {
d->rev = lfs_tole32(d->rev);
d->size = lfs_tole32(d->size);
d->tail[0] = lfs_tole32(d->tail[0]);
d->tail[1] = lfs_tole32(d->tail[1]);
}
static void lfs1_entry_fromle32(struct lfs1_disk_entry *d) {
d->u.dir[0] = lfs_fromle32(d->u.dir[0]);
d->u.dir[1] = lfs_fromle32(d->u.dir[1]);
}
static void lfs1_entry_tole32(struct lfs1_disk_entry *d) {
d->u.dir[0] = lfs_tole32(d->u.dir[0]);
d->u.dir[1] = lfs_tole32(d->u.dir[1]);
}
static void lfs1_superblock_fromle32(struct lfs1_disk_superblock *d) {
d->root[0] = lfs_fromle32(d->root[0]);
d->root[1] = lfs_fromle32(d->root[1]);
d->block_size = lfs_fromle32(d->block_size);
d->block_count = lfs_fromle32(d->block_count);
d->version = lfs_fromle32(d->version);
}
///// Metadata pair and directory operations ///
static inline lfs_size_t lfs1_entry_size(const lfs1_entry_t *entry) {
return 4 + entry->d.elen + entry->d.alen + entry->d.nlen;
}
static int lfs1_dir_fetch(lfs_t *lfs,
lfs1_dir_t *dir, const lfs_block_t pair[2]) {
// copy out pair, otherwise may be aliasing dir
const lfs_block_t tpair[2] = {pair[0], pair[1]};
bool valid = false;
// check both blocks for the most recent revision
for (int i = 0; i < 2; i++) {
struct lfs1_disk_dir test;
int err = lfs1_bd_read(lfs, tpair[i], 0, &test, sizeof(test));
lfs1_dir_fromle32(&test);
if (err) {
if (err == LFS_ERR_CORRUPT) {
continue;
}
return err;
}
if (valid && lfs_scmp(test.rev, dir->d.rev) < 0) {
continue;
}
if ((0x7fffffff & test.size) < sizeof(test)+4 ||
(0x7fffffff & test.size) > lfs->cfg->block_size) {
continue;
}
uint32_t crc = 0xffffffff;
lfs1_dir_tole32(&test);
lfs1_crc(&crc, &test, sizeof(test));
lfs1_dir_fromle32(&test);
err = lfs1_bd_crc(lfs, tpair[i], sizeof(test),
(0x7fffffff & test.size) - sizeof(test), &crc);
if (err) {
if (err == LFS_ERR_CORRUPT) {
continue;
}
return err;
}
if (crc != 0) {
continue;
}
valid = true;
// setup dir in case it's valid
dir->pair[0] = tpair[(i+0) % 2];
dir->pair[1] = tpair[(i+1) % 2];
dir->off = sizeof(dir->d);
dir->d = test;
}
if (!valid) {
LFS_ERROR("Corrupted dir pair at %" PRIu32 " %" PRIu32 ,
tpair[0], tpair[1]);
return LFS_ERR_CORRUPT;
}
return 0;
}
static int lfs1_dir_next(lfs_t *lfs, lfs1_dir_t *dir, lfs1_entry_t *entry) {
while (dir->off + sizeof(entry->d) > (0x7fffffff & dir->d.size)-4) {
if (!(0x80000000 & dir->d.size)) {
entry->off = dir->off;
return LFS_ERR_NOENT;
}
int err = lfs1_dir_fetch(lfs, dir, dir->d.tail);
if (err) {
return err;
}
dir->off = sizeof(dir->d);
dir->pos += sizeof(dir->d) + 4;
}
int err = lfs1_bd_read(lfs, dir->pair[0], dir->off,
&entry->d, sizeof(entry->d));
lfs1_entry_fromle32(&entry->d);
if (err) {
return err;
}
entry->off = dir->off;
dir->off += lfs1_entry_size(entry);
dir->pos += lfs1_entry_size(entry);
return 0;
}
/// littlefs v1 specific operations ///
int lfs1_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data) {
if (lfs_pair_isnull(lfs->lfs1->root)) {
return 0;
}
// iterate over metadata pairs
lfs1_dir_t dir;
lfs1_entry_t entry;
lfs_block_t cwd[2] = {0, 1};
while (true) {
for (int i = 0; i < 2; i++) {
int err = cb(data, cwd[i]);
if (err) {
return err;
}
}
int err = lfs1_dir_fetch(lfs, &dir, cwd);
if (err) {
return err;
}
// iterate over contents
while (dir.off + sizeof(entry.d) <= (0x7fffffff & dir.d.size)-4) {
err = lfs1_bd_read(lfs, dir.pair[0], dir.off,
&entry.d, sizeof(entry.d));
lfs1_entry_fromle32(&entry.d);
if (err) {
return err;
}
dir.off += lfs1_entry_size(&entry);
if ((0x70 & entry.d.type) == (0x70 & LFS1_TYPE_REG)) {
err = lfs_ctz_traverse(lfs, NULL, &lfs->rcache,
entry.d.u.file.head, entry.d.u.file.size, cb, data);
if (err) {
return err;
}
}
}
// we also need to check if we contain a threaded v2 directory
lfs_mdir_t dir2 = {.split=true, .tail={cwd[0], cwd[1]}};
while (dir2.split) {
err = lfs_dir_fetch(lfs, &dir2, dir2.tail);
if (err) {
break;
}
for (int i = 0; i < 2; i++) {
err = cb(data, dir2.pair[i]);
if (err) {
return err;
}
}
}
cwd[0] = dir.d.tail[0];
cwd[1] = dir.d.tail[1];
if (lfs_pair_isnull(cwd)) {
break;
}
}
return 0;
}
static int lfs1_moved(lfs_t *lfs, const void *e) {
if (lfs_pair_isnull(lfs->lfs1->root)) {
return 0;
}
// skip superblock
lfs1_dir_t cwd;
int err = lfs1_dir_fetch(lfs, &cwd, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
// iterate over all directory directory entries
lfs1_entry_t entry;
while (!lfs_pair_isnull(cwd.d.tail)) {
err = lfs1_dir_fetch(lfs, &cwd, cwd.d.tail);
if (err) {
return err;
}
while (true) {
err = lfs1_dir_next(lfs, &cwd, &entry);
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err == LFS_ERR_NOENT) {
break;
}
if (!(0x80 & entry.d.type) &&
memcmp(&entry.d.u, e, sizeof(entry.d.u)) == 0) {
return true;
}
}
}
return false;
}
/// Filesystem operations ///
static int lfs1_mount(lfs_t *lfs, struct lfs1 *lfs1,
const struct lfs_config *cfg) {
int err = 0;
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
err = lfs_init(lfs, cfg);
if (err) {
return err;
}
lfs->lfs1 = lfs1;
lfs->lfs1->root[0] = 0xffffffff;
lfs->lfs1->root[1] = 0xffffffff;
// setup free lookahead
lfs->free.off = 0;
lfs->free.size = 0;
lfs->free.i = 0;
lfs_alloc_ack(lfs);
// load superblock
lfs1_dir_t dir;
lfs1_superblock_t superblock;
err = lfs1_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
if (err && err != LFS_ERR_CORRUPT) {
goto cleanup;
}
if (!err) {
err = lfs1_bd_read(lfs, dir.pair[0], sizeof(dir.d),
&superblock.d, sizeof(superblock.d));
lfs1_superblock_fromle32(&superblock.d);
if (err) {
goto cleanup;
}
lfs->lfs1->root[0] = superblock.d.root[0];
lfs->lfs1->root[1] = superblock.d.root[1];
}
if (err || memcmp(superblock.d.magic, "littlefs", 8) != 0) {
LFS_ERROR("Invalid superblock at %d %d", 0, 1);
err = LFS_ERR_CORRUPT;
goto cleanup;
}
uint16_t major_version = (0xffff & (superblock.d.version >> 16));
uint16_t minor_version = (0xffff & (superblock.d.version >> 0));
if ((major_version != LFS1_DISK_VERSION_MAJOR ||
minor_version > LFS1_DISK_VERSION_MINOR)) {
LFS_ERROR("Invalid version %d.%d", major_version, minor_version);
err = LFS_ERR_INVAL;
goto cleanup;
}
return 0;
}
cleanup:
lfs_deinit(lfs);
return err;
}
static int lfs1_unmount(lfs_t *lfs) {
return lfs_deinit(lfs);
}
/// v1 migration ///
int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg) {
struct lfs1 lfs1;
int err = lfs1_mount(lfs, &lfs1, cfg);
if (err) {
return err;
}
Cleaned up a couple of warnings - Shifting signed 32-bit value by 31 bits is undefined behaviour This was an interesting one as on initial inspection, `uint8_t & 1` looks like it will result in an unsigned variable. However, due to uint8_t being "smaller" than int, this actually results in a signed int, causing an undefined shift operation. - Identical inner 'if' condition is always true (outer condition is 'true' and inner condition is 'true'). This was caused by the use of `if (true) {` to avoid "goto bypasses variable initialization" warnings. Using just `{` instead seems to avoid this problem. found by keck-in-space and armandas
2019-04-09 23:37:53 +00:00
{
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
// iterate through each directory, copying over entries
// into new directory
lfs1_dir_t dir1;
lfs_mdir_t dir2;
dir1.d.tail[0] = lfs->lfs1->root[0];
dir1.d.tail[1] = lfs->lfs1->root[1];
while (!lfs_pair_isnull(dir1.d.tail)) {
// iterate old dir
err = lfs1_dir_fetch(lfs, &dir1, dir1.d.tail);
if (err) {
goto cleanup;
}
// create new dir and bind as temporary pretend root
err = lfs_dir_alloc(lfs, &dir2);
if (err) {
goto cleanup;
}
dir2.rev = dir1.d.rev;
Added v1->v2 migration into CI Also fixed issue where migration would not handle large dirs due to v1 iteration changing the pair of the directory.
2019-04-02 03:12:08 +00:00
dir1.head[0] = dir1.pair[0];
dir1.head[1] = dir1.pair[1];
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
lfs->root[0] = dir2.pair[0];
lfs->root[1] = dir2.pair[1];
err = lfs_dir_commit(lfs, &dir2, NULL, 0);
if (err) {
goto cleanup;
}
while (true) {
lfs1_entry_t entry1;
err = lfs1_dir_next(lfs, &dir1, &entry1);
if (err && err != LFS_ERR_NOENT) {
goto cleanup;
}
if (err == LFS_ERR_NOENT) {
break;
}
// check that entry has not been moved
if (entry1.d.type & 0x80) {
int moved = lfs1_moved(lfs, &entry1.d.u);
if (moved < 0) {
err = moved;
goto cleanup;
}
if (moved) {
continue;
}
entry1.d.type &= ~0x80;
}
// also fetch name
char name[LFS_NAME_MAX+1];
memset(name, 0, sizeof(name));
err = lfs1_bd_read(lfs, dir1.pair[0],
entry1.off + 4+entry1.d.elen+entry1.d.alen,
name, entry1.d.nlen);
if (err) {
goto cleanup;
}
bool isdir = (entry1.d.type == LFS1_TYPE_DIR);
// create entry in new dir
err = lfs_dir_fetch(lfs, &dir2, lfs->root);
if (err) {
goto cleanup;
}
uint16_t id;
err = lfs_dir_find(lfs, &dir2, &(const char*){name}, &id);
if (!(err == LFS_ERR_NOENT && id != 0x3ff)) {
err = (err < 0) ? err : LFS_ERR_EXIST;
goto cleanup;
}
lfs1_entry_tole32(&entry1.d);
err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_CREATE, id, 0), NULL},
{LFS_MKTAG(
isdir ? LFS_TYPE_DIR : LFS_TYPE_REG,
id, entry1.d.nlen), name},
{LFS_MKTAG(
isdir ? LFS_TYPE_DIRSTRUCT : LFS_TYPE_CTZSTRUCT,
id, sizeof(&entry1.d.u)), &entry1.d.u}));
lfs1_entry_fromle32(&entry1.d);
if (err) {
goto cleanup;
}
}
if (!lfs_pair_isnull(dir1.d.tail)) {
// find last block and update tail to thread into fs
err = lfs_dir_fetch(lfs, &dir2, lfs->root);
if (err) {
goto cleanup;
}
while (dir2.split) {
err = lfs_dir_fetch(lfs, &dir2, dir2.tail);
if (err) {
goto cleanup;
}
}
lfs_pair_tole32(dir2.pair);
err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_SOFTTAIL, 0x3ff, 0),
dir1.d.tail}));
lfs_pair_fromle32(dir2.pair);
if (err) {
goto cleanup;
}
}
// Copy over first block to thread into fs. Unfortunately
// if this fails there is not much we can do.
Added v1->v2 migration into CI Also fixed issue where migration would not handle large dirs due to v1 iteration changing the pair of the directory.
2019-04-02 03:12:08 +00:00
LFS_DEBUG("Migrating %"PRIu32" %"PRIu32" -> %"PRIu32" %"PRIu32,
lfs->root[0], lfs->root[1], dir1.head[0], dir1.head[1]);
err = lfs_bd_erase(lfs, dir1.head[1]);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
if (err) {
goto cleanup;
}
err = lfs_dir_fetch(lfs, &dir2, lfs->root);
if (err) {
goto cleanup;
}
for (lfs_off_t i = 0; i < dir2.off; i++) {
uint8_t dat;
err = lfs_bd_read(lfs,
NULL, &lfs->rcache, dir2.off,
dir2.pair[0], i, &dat, 1);
if (err) {
goto cleanup;
}
err = lfs_bd_prog(lfs,
&lfs->pcache, &lfs->rcache, true,
Added v1->v2 migration into CI Also fixed issue where migration would not handle large dirs due to v1 iteration changing the pair of the directory.
2019-04-02 03:12:08 +00:00
dir1.head[1], i, &dat, 1);
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
if (err) {
goto cleanup;
}
}
Fixed missing cache flush in lfs_migrate The data written to the prog cache would make littlefs internally consistent, but because this was never written to disk, the filesystem would become unmountable. Unfortunately, this wasn't found during testing because caches automatically flush if data is written up to a program boundary (maybe this was a mistake?). Found by rojer
2019-05-28 16:35:22 +00:00
err = lfs_bd_flush(lfs, &lfs->pcache, &lfs->rcache, true);
if (err) {
goto cleanup;
}
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
}
// Create new superblock. This marks a successful migration!
err = lfs1_dir_fetch(lfs, &dir1, (const lfs_block_t[2]){0, 1});
if (err) {
goto cleanup;
}
dir2.pair[0] = dir1.pair[0];
dir2.pair[1] = dir1.pair[1];
dir2.rev = dir1.d.rev;
dir2.off = sizeof(dir2.rev);
dir2.etag = 0xffffffff;
dir2.count = 0;
dir2.tail[0] = lfs->lfs1->root[0];
dir2.tail[1] = lfs->lfs1->root[1];
dir2.erased = false;
dir2.split = true;
lfs_superblock_t superblock = {
.version = LFS_DISK_VERSION,
.block_size = lfs->cfg->block_size,
.block_count = lfs->cfg->block_count,
.name_max = lfs->name_max,
.file_max = lfs->file_max,
.attr_max = lfs->attr_max,
};
lfs_superblock_tole32(&superblock);
err = lfs_dir_commit(lfs, &dir2, LFS_MKATTRS(
{LFS_MKTAG(LFS_TYPE_CREATE, 0, 0), NULL},
{LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, 8), "littlefs"},
{LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, sizeof(superblock)),
&superblock}));
if (err) {
goto cleanup;
}
// sanity check that fetch works
err = lfs_dir_fetch(lfs, &dir2, (const lfs_block_t[2]){0, 1});
if (err) {
goto cleanup;
}
Added redundant compaction to lfs_format/lfs_migrate This ensures that both blocks in the superblock pair are written with the superblock info. While this does use an additional erase cycle, it prevents older versions of littlefs from accidentally being picked up in the case that the disk is mounted on a system that doesn't support the newer version. This does bring back the risk of picking up old littlefs versions on a disk that has been formatted with a filesystem that doesn't use block 2 (such as FAT), but this risk already exists, and moving between versions of littlefs is more likely with the recent v1 -> v2 update. Suggested by rojer
2019-05-28 23:16:51 +00:00
// force compaction to prevent accidentally mounting v1
dir2.erased = false;
err = lfs_dir_commit(lfs, &dir2, NULL, 0);
if (err) {
goto cleanup;
}
Added migration from littlefs v1 This is the help the introduction of littlefs v2, which is disk incompatible with littlefs v1. While v2 can't mount v1, what we can do is provide an optional migration, which can convert v1 into v2 partially in-place. At worse, we only need to carry over the readonly operations on v1, which are much less complicated than the write operations, so the extra code cost may be as low as 25% of the v1 code size. Also, because v2 contains only metadata changes, it's possible to avoid copying file data during the update. Enabling the migration requires two steps 1. Defining LFS_MIGRATE 2. Call lfs_migrate (only available with the above macro) Each macro multiplies the number of configurations needed to be tested, so I've been avoiding macro controlled features since there's still work to be done around testing the single configuration that's already available. However, here the cost would be too high if we included migration code in the standard build. We can't use the lfs_migrate function for link time gc because of a dependency between the allocator and v1 data structures. So how does lfs_migrate work? It turned out to be a bit complicated, but the answer is a multistep process that relies on mounting v1 readonly and building the metadata skeleton needed by v2. 1. For each directory, create a v2 directory 2. Copy over v1 entries into v2 directory, including the soft-tail entry 3. Move head block of v2 directory into the unused metadata block in v1 directory. This results in both a v1 and v2 directory sharing the same metadata pair. 4. Finally, create a new superblock in the unused metadata block of the v1 superblock. Just like with normal metadata updates, the completion of the write to the second metadata block marks a succesful migration that can be mounted with littlefs v2. And all of this can occur atomically, enabling complete fallback if power is lost of an error occurs. Note there are several limitations with this solution. 1. While migration doesn't duplicate file data, it does temporarily duplicate all metadata. This can cause a device to run out of space if storage is tight and the filesystem as many files. If the device was created with >~2x the expected storage, it should be fine. 2. The current implementation is not able to recover if the metadata pairs develop bad blocks. It may be possilbe to workaround this, but it creates the problem that directories may change location during the migration. The other solutions I've looked at are complicated and require superlinear runtime. Currently I don't think it's worth fixing this limitation. 3. Enabling the migration requires additional code size. Currently this looks like it's roughly 11% at least on x86. And, if any failure does occur, no harm is done to the original v1 filesystem on disk.
2019-02-23 03:34:03 +00:00
}
cleanup:
lfs1_unmount(lfs);
return err;
}
#endif
Reference in New Issue Copy Permalink