openharmony/third_party_littlefs
1
0
Fork 0
mirror of https://gitee.com/openharmony/third_party_littlefs synced 2024-11-27 09:01:27 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
213530c376
third_party_littlefs/lfs.c

3308 lines
94 KiB
C
Raw Normal View History

Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
/*
* The little filesystem
*
Updated copyright Due to employee contract Per ARM license remains under Apache 2.0
2017-10-13 01:27:33 +00:00
* Copyright (c) 2017 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
*/
#include "lfs.h"
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
#include "lfs_util.h"
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// Caching block device operations ///
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
static int lfs_cache_read(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache, bool store,
lfs_block_t block, lfs_off_t off,
void *buffer, lfs_size_t size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (pcache && block == pcache->block &&
off >= pcache->off &&
off < pcache->off + pcache->size) {
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
// is already in pcache?
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_size_t diff = lfs_min(size, pcache->size - (off-pcache->off));
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
memcpy(data, &pcache->buffer[off-pcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (block == rcache->block &&
off >= rcache->off &&
off < rcache->off + rcache->size) {
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
// is already in rcache?
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_size_t diff = lfs_min(size, rcache->size - (off-rcache->off));
if (pcache && block == pcache->block) {
diff = lfs_min(diff, pcache->off - off);
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
memcpy(data, &rcache->buffer[off-rcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (!store && off % lfs->cfg->read_size == 0 &&
size >= lfs->cfg->read_size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
// bypass cache?
lfs_size_t diff = size - (size % lfs->cfg->read_size);
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
int err = lfs->cfg->read(lfs->cfg, block, off, data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
data += diff;
off += diff;
size -= diff;
continue;
}
// load to cache, first condition can no longer fail
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_size_t size = store ? lfs->cfg->cache_size : lfs->cfg->prog_size;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
rcache->block = block;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
rcache->off = lfs_aligndown(off, size);
rcache->size = size;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
int err = lfs->cfg->read(lfs->cfg, rcache->block,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
rcache->off, rcache->buffer, size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
static int lfs_cache_cmp(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache,
lfs_block_t block, lfs_off_t off,
const void *buffer, lfs_size_t size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
const uint8_t *data = buffer;
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
int err = lfs_cache_read(lfs, pcache, rcache, true,
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
block, off+i, &c, 1);
if (err) {
return err;
}
if (c != data[i]) {
return false;
}
}
return true;
}
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
static int lfs_cache_crc(lfs_t *lfs,
const lfs_cache_t *pcache, lfs_cache_t *rcache,
lfs_block_t block, lfs_off_t off, lfs_size_t size, uint32_t *crc) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
int err = lfs_cache_read(lfs, pcache, rcache, true,
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
block, off+i, &c, 1);
if (err) {
return err;
}
lfs_crc(crc, &c, 1);
}
return 0;
}
static int lfs_cache_flush(lfs_t *lfs,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_cache_t *pcache, lfs_cache_t *rcache, bool validate) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (pcache->block != 0xffffffff) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(pcache->block < lfs->cfg->block_count);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs_size_t diff = lfs_alignup(pcache->size, lfs->cfg->prog_size);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs->cfg->prog(lfs->cfg, pcache->block,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
pcache->off, pcache->buffer, diff);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (validate) {
// check data on disk
rcache->block = 0xffffffff;
int res = lfs_cache_cmp(lfs, NULL, rcache, pcache->block,
pcache->off, pcache->buffer, diff);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_CORRUPT;
}
}
pcache->block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
return 0;
}
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
static int lfs_cache_prog(lfs_t *lfs,
lfs_cache_t *pcache, lfs_cache_t *rcache, bool validate,
lfs_block_t block, lfs_off_t off,
const void *buffer, lfs_size_t size) {
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
const uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
LFS_ASSERT(off + size <= lfs->cfg->block_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
if (block == pcache->block &&
off >= pcache->off &&
off < pcache->off + lfs->cfg->cache_size) {
// already fits in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->cfg->cache_size - (off-pcache->off));
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
memcpy(&pcache->buffer[off-pcache->off], data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
pcache->size = off - pcache->off;
if (pcache->size == lfs->cfg->cache_size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// eagerly flush out pcache if we fill up
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
int err = lfs_cache_flush(lfs, pcache, rcache, validate);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// pcache must have been flushed, either by programming and
// entire block or manually flushing the pcache
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(pcache->block == 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// prepare pcache, first condition can no longer fail
pcache->block = block;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
pcache->off = lfs_aligndown(off, lfs->cfg->prog_size);
pcache->size = 0;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// General lfs block device operations ///
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_read(lfs_t *lfs, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, void *buffer, lfs_size_t size) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
return lfs_cache_read(lfs, &lfs->pcache, &lfs->rcache, true,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
block, off, buffer, size);
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_prog(lfs_t *lfs, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, const void *buffer, lfs_size_t size) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
return lfs_cache_prog(lfs, &lfs->pcache, &lfs->rcache, false,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
block, off, buffer, size);
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_cmp(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, const void *buffer, lfs_size_t size) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
return lfs_cache_cmp(lfs, NULL, &lfs->rcache, block, off, buffer, size);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
static int lfs_bd_crc(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
return lfs_cache_crc(lfs, NULL, &lfs->rcache, block, off, size, crc);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
static int lfs_bd_erase(lfs_t *lfs, lfs_block_t block) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->erase(lfs->cfg, block);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_sync(lfs_t *lfs) {
Fixed dirty rcache during directory commit An interesting side-effect of adding internal checks to the littlefs for block errors, is that the littlefs starts to cover up its own flaws. Probably out of embarrassment. In this case, the relocation logic for directories left the littlefs rcache dirty with invalid data. The littlefs detected the error, treated it as a corrupted write, and just moved the "corrupted" block to a new block, which as a side-effect flushes the rcache. Since committing a dir will end up flushing the rcache to check for errors anyways, we can just drop the rcache in lfs_bd_sync.
2017-06-25 21:21:14 +00:00
lfs->rcache.block = 0xffffffff;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
int err = lfs_cache_flush(lfs, &lfs->pcache, &lfs->rcache, false);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->sync(lfs->cfg);
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
/// Internal operations predeclared here ///
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_pred(lfs_t *lfs, const lfs_block_t dir[2],
lfs_mdir_t *pdir);
static int32_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t dir[2],
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
lfs_mdir_t *parent);
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_relocate(lfs_t *lfs,
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
const lfs_block_t oldpair[2], lfs_block_t newpair[2]);
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_forceconsistency(lfs_t *lfs);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
/// Block allocator ///
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
static int lfs_alloc_lookahead(void *p, lfs_block_t block) {
lfs_t *lfs = (lfs_t*)p;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs_block_t off = ((block - lfs->free.off)
Fixed issue with negative modulo with unaligned lookaheads When the lookahead buffer wraps around in an unaligned filesystem, it's possible for blocks at the beginning of the disk to have a negative distance from the lookahead, but still reside in the lookahead buffer. Switching to signed modulo doesn't quite work due to how negative modulo is implemented in C, so the simple solution is to shift the region to be positive.
2017-09-17 19:52:25 +00:00
+ lfs->cfg->block_count) % lfs->cfg->block_count;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
if (off < lfs->free.size) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer[off / 32] |= 1U << (off % 32);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
return 0;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) {
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
while (true) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
while (lfs->free.i != lfs->free.size) {
lfs_block_t off = lfs->free.i;
lfs->free.i += 1;
lfs->free.ack -= 1;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!(lfs->free.buffer[off / 32] & (1U << (off % 32)))) {
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// found a free block
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
*block = (lfs->free.off + off) % lfs->cfg->block_count;
Fixed issue with lookahead trusting old lookahead blocks One of the big simplifications in littlefs's implementation is the complete lack of tracking free blocks, allowing operations to simply drop blocks that are no longer in use. However, this means the lookahead buffer can easily contain outdated blocks that were previously deleted. This is usually fine, as littlefs will rescan the storage if it can't find a free block in the lookahead buffer, but after changes that caused littlefs to more conservatively respect the alloc acks (e611cf5), any scanned blocks after an ack would be incorrectly trusted. The fix is to eagerly scan ahead in the lookahead when we allocate so that alloc acks are better able to discredit old lookahead blocks. Since usually alloc acks are tightly coupled to allocations of one or two blocks, this allows littlefs to properly rescan every set of allocations. This may still be a concern if there is a long series of worn out blocks, but in the worst case littlefs will conservatively avoid using blocks it's not sure about. Found by davidefer
2018-04-06 22:00:29 +00:00
// eagerly find next off so an alloc ack can
// discredit old lookahead blocks
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
while (lfs->free.i != lfs->free.size &&
(lfs->free.buffer[lfs->free.i / 32]
& (1U << (lfs->free.i % 32)))) {
lfs->free.i += 1;
lfs->free.ack -= 1;
Fixed issue with lookahead trusting old lookahead blocks One of the big simplifications in littlefs's implementation is the complete lack of tracking free blocks, allowing operations to simply drop blocks that are no longer in use. However, this means the lookahead buffer can easily contain outdated blocks that were previously deleted. This is usually fine, as littlefs will rescan the storage if it can't find a free block in the lookahead buffer, but after changes that caused littlefs to more conservatively respect the alloc acks (e611cf5), any scanned blocks after an ack would be incorrectly trusted. The fix is to eagerly scan ahead in the lookahead when we allocate so that alloc acks are better able to discredit old lookahead blocks. Since usually alloc acks are tightly coupled to allocations of one or two blocks, this allows littlefs to properly rescan every set of allocations. This may still be a concern if there is a long series of worn out blocks, but in the worst case littlefs will conservatively avoid using blocks it's not sure about. Found by davidefer
2018-04-06 22:00:29 +00:00
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
// check if we have looked at all blocks since last ack
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
if (lfs->free.ack == 0) {
LFS_WARN("No more free space %d", lfs->free.i + lfs->free.off);
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
return LFS_ERR_NOSPC;
}
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.off = (lfs->free.off + lfs->free.size)
% lfs->cfg->block_count;
lfs->free.size = lfs_min(lfs->cfg->lookahead, lfs->free.ack);
lfs->free.i = 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// find mask of free blocks from tree
Fixed issue with aggressively rounding down lookahead configuration The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
2017-11-10 01:10:08 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead/8);
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int err = lfs_fs_traverse(lfs, lfs_alloc_lookahead, lfs);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
if (err) {
return err;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static void lfs_alloc_ack(lfs_t *lfs) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.ack = lfs->cfg->block_count;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Metadata pair and directory operations ///
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static inline void lfs_pairswap(lfs_block_t pair[2]) {
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
lfs_block_t t = pair[0];
pair[0] = pair[1];
pair[1] = t;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
static inline bool lfs_pairisnull(const lfs_block_t pair[2]) {
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
return pair[0] == 0xffffffff || pair[1] == 0xffffffff;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static inline int lfs_paircmp(
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
return !(paira[0] == pairb[0] || paira[1] == pairb[1] ||
paira[0] == pairb[1] || paira[1] == pairb[0]);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static inline bool lfs_pairsync(
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
return (paira[0] == pairb[0] && paira[1] == pairb[1]) ||
(paira[0] == pairb[1] && paira[1] == pairb[0]);
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
static inline void lfs_pairfromle32(lfs_block_t *pair) {
pair[0] = lfs_fromle32(pair[0]);
pair[1] = lfs_fromle32(pair[1]);
}
static inline void lfs_pairtole32(lfs_block_t *pair) {
pair[0] = lfs_tole32(pair[0]);
pair[1] = lfs_tole32(pair[1]);
}
static void lfs_ctzfromle32(struct lfs_ctz *ctz) {
ctz->head = lfs_fromle32(ctz->head);
ctz->size = lfs_fromle32(ctz->size);
}
static void lfs_ctztole32(struct lfs_ctz *ctz) {
ctz->head = lfs_tole32(ctz->head);
ctz->size = lfs_tole32(ctz->size);
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
/// Entry tag operations ///
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
#define LFS_MKTAG(type, id, size) \
(((uint32_t)(type) << 22) | ((uint32_t)(id) << 12) | (uint32_t)(size))
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
#define LFS_MKATTR(type, id, buffer, size, next) \
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
&(const lfs_mattr_t){LFS_MKTAG(type, id, size), (buffer), (next)}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
static inline bool lfs_tagisvalid(uint32_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return !(tag & 0x80000000);
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
static inline bool lfs_tagisuser(uint32_t tag) {
Restructured types to use a more flexible bit encoding Recall that the 32-bit tag structure contains a 9-bit type. The type structure then decomposes into a bit more information: [--- 9 ---] [1|- 4 -|- 4 -] ^ ^ ^- specific type | \------- subtype \----------- user bit The main change is an observation from moving type info to the name tag from the struct tag. Since we don't need the type info in the struct tag, we can significantly simplify the type structure.
2018-07-09 19:13:31 +00:00
return (tag & 0x40000000);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
static inline uint16_t lfs_tagtype(uint32_t tag) {
Restructured types to use a more flexible bit encoding Recall that the 32-bit tag structure contains a 9-bit type. The type structure then decomposes into a bit more information: [--- 9 ---] [1|- 4 -|- 4 -] ^ ^ ^- specific type | \------- subtype \----------- user bit The main change is an observation from moving type info to the name tag from the struct tag. Since we don't need the type info in the struct tag, we can significantly simplify the type structure.
2018-07-09 19:13:31 +00:00
return (tag & 0x7fc00000) >> 22;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
static inline uint16_t lfs_tagsubtype(uint32_t tag) {
Restructured types to use a more flexible bit encoding Recall that the 32-bit tag structure contains a 9-bit type. The type structure then decomposes into a bit more information: [--- 9 ---] [1|- 4 -|- 4 -] ^ ^ ^- specific type | \------- subtype \----------- user bit The main change is an observation from moving type info to the name tag from the struct tag. Since we don't need the type info in the struct tag, we can significantly simplify the type structure.
2018-07-09 19:13:31 +00:00
return (tag & 0x7c000000) >> 22;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
static inline uint16_t lfs_tagid(uint32_t tag) {
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
return (tag & 0x003ff000) >> 12;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
static inline lfs_size_t lfs_tagsize(uint32_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return tag & 0x00000fff;
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// operations on set of globals
static inline void lfs_globalxor(lfs_global_t *a, const lfs_global_t *b) {
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
for (int i = 0; i < sizeof(lfs_global_t)/4; i++) {
a->u32[i] ^= b->u32[i];
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
}
static inline bool lfs_globaliszero(const lfs_global_t *a) {
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
for (int i = 0; i < sizeof(lfs_global_t)/4; i++) {
if (a->u32[i] != 0) {
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
return false;
}
}
return true;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
static inline void lfs_globalzero(lfs_global_t *a) {
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs_globalxor(a, a);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
static inline void lfs_globalfromle32(lfs_global_t *a) {
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs_pairfromle32(a->s.movepair);
a->s.moveid = lfs_fromle16(a->s.moveid);
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
}
static inline void lfs_globaltole32(lfs_global_t *a) {
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs_pairtole32(a->s.movepair);
a->s.moveid = lfs_tole16(a->s.moveid);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
}
static inline void lfs_globalmove(lfs_t *lfs,
const lfs_block_t pair[2], uint16_t id) {
lfs_global_t diff;
lfs_globalzero(&diff);
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
diff.s.movepair[0] ^= lfs->globals.s.movepair[0] ^ pair[0];
diff.s.movepair[1] ^= lfs->globals.s.movepair[1] ^ pair[1];
diff.s.moveid ^= lfs->globals.s.moveid ^ id;
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_globalfromle32(&lfs->locals);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&lfs->locals, &diff);
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_globaltole32(&lfs->locals);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&lfs->globals, &diff);
}
static inline void lfs_globaldeorphaned(lfs_t *lfs, bool deorphaned) {
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs->locals.s.deorphaned ^= lfs->globals.s.deorphaned ^ deorphaned;
lfs->globals.s.deorphaned ^= lfs->globals.s.deorphaned ^ deorphaned;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
// commit logic
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
struct lfs_commit {
lfs_block_t block;
lfs_off_t off;
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
uint32_t ptag;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint32_t crc;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
lfs_off_t begin;
lfs_off_t end;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
};
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
struct lfs_diskoff {
lfs_block_t block;
lfs_off_t off;
};
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int32_t lfs_commitget(lfs_t *lfs, lfs_block_t block, lfs_off_t off,
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
uint32_t tag, uint32_t getmask, uint32_t gettag, int32_t getdiff,
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
void *buffer, bool stopatcommit) {
// iterate over dir block backwards (for faster lookups)
Fixed test bugs around handling corruption The main thing to consider was how lfs_dir_fetchwith reacts to corruption it finds and to make sure falling back to old values works correctly. Some of the tricky bits involved making sure we could fall back to both old commits and old metadata blocks while still handling things like synthetic moves correctly.
2018-07-30 19:23:51 +00:00
while (off >= 2*sizeof(tag)+lfs_tagsize(tag)) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
off -= sizeof(tag)+lfs_tagsize(tag);
if (lfs_tagtype(tag) == LFS_TYPE_CRC && stopatcommit) {
break;
} else if (lfs_tagtype(tag) == LFS_TYPE_DELETE) {
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
if (lfs_tagid(tag) <= lfs_tagid(gettag + getdiff)) {
getdiff += LFS_MKTAG(0, 1, 0);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
} else if ((tag & getmask) == ((gettag + getdiff) & getmask)) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (buffer) {
lfs_size_t diff = lfs_min(
lfs_tagsize(gettag), lfs_tagsize(tag));
int err = lfs_bd_read(lfs, block,
off+sizeof(tag), buffer, diff);
if (err) {
return err;
}
memset((uint8_t*)buffer + diff, 0,
lfs_tagsize(gettag) - diff);
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
return tag - getdiff;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
uint32_t ntag;
int err = lfs_bd_read(lfs, block, off, &ntag, sizeof(ntag));
if (err) {
return err;
}
tag ^= lfs_fromle32(ntag);
}
return LFS_ERR_NOENT;
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
static int lfs_commitattrs(lfs_t *lfs, struct lfs_commit *commit,
uint16_t id, const struct lfs_attr *attrs);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_commitmove(lfs_t *lfs, struct lfs_commit *commit,
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
uint16_t fromid, uint16_t toid,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
const lfs_mdir_t *dir, const lfs_mattr_t *attrs);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_commitattr(lfs_t *lfs, struct lfs_commit *commit,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
uint32_t tag, const void *buffer) {
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
if (lfs_tagtype(tag) == LFS_FROM_ATTRS) {
// special case for custom attributes
return lfs_commitattrs(lfs, commit,
lfs_tagid(tag), buffer);
} else if (lfs_tagtype(tag) == LFS_FROM_MOVE) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// special case for moves
return lfs_commitmove(lfs, commit,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_tagsize(tag), lfs_tagid(tag),
buffer, NULL);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// check if we fit
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_size_t size = lfs_tagsize(tag);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (commit->off + sizeof(tag)+size > commit->end) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return LFS_ERR_NOSPC;
}
// write out tag
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
uint32_t ntag = lfs_tole32((tag & 0x7fffffff) ^ commit->ptag);
lfs_crc(&commit->crc, &ntag, sizeof(ntag));
int err = lfs_bd_prog(lfs, commit->block, commit->off,
&ntag, sizeof(ntag));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
commit->off += sizeof(ntag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (!(tag & 0x80000000)) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// from memory
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_crc(&commit->crc, buffer, size);
err = lfs_bd_prog(lfs, commit->block, commit->off, buffer, size);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
} else {
// from disk
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
const struct lfs_diskoff *disk = buffer;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
for (lfs_off_t i = 0; i < size; i++) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// rely on caching to make this efficient
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint8_t dat;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
int err = lfs_bd_read(lfs, disk->block, disk->off+i, &dat, 1);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
lfs_crc(&commit->crc, &dat, 1);
err = lfs_bd_prog(lfs, commit->block, commit->off+i, &dat, 1);
if (err) {
return err;
}
}
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
commit->off += size;
commit->ptag = tag & 0x7fffffff;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return 0;
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
static int lfs_commitattrs(lfs_t *lfs, struct lfs_commit *commit,
uint16_t id, const struct lfs_attr *attrs) {
for (const struct lfs_attr *a = attrs; a; a = a->next) {
int err = lfs_commitattr(lfs, commit,
LFS_MKTAG(0x100 | a->type, id, a->size), a->buffer);
if (err) {
return err;
}
}
return 0;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_commitmove(lfs_t *lfs, struct lfs_commit *commit,
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
uint16_t fromid, uint16_t toid,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
const lfs_mdir_t *dir, const lfs_mattr_t *attrs) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// iterate through list and commits, only committing unique entries
lfs_off_t off = dir->off;
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
uint32_t ntag = dir->etag;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
while (attrs || off > sizeof(uint32_t)) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
struct lfs_diskoff disk;
uint32_t tag;
const void *buffer;
if (attrs) {
tag = attrs->tag;
buffer = attrs->buffer;
attrs = attrs->next;
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
} else {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
LFS_ASSERT(off > sizeof(ntag)+lfs_tagsize(ntag));
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
off -= sizeof(ntag)+lfs_tagsize(ntag);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
tag = ntag;
buffer = &disk;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
disk.block = dir->pair[0];
disk.off = off + sizeof(tag);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_bd_read(lfs, dir->pair[0], off, &ntag, sizeof(ntag));
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
if (err) {
return err;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
ntag = lfs_fromle32(ntag);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
ntag ^= tag;
tag |= 0x80000000;
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (lfs_tagtype(tag) == LFS_TYPE_DELETE && lfs_tagid(tag) <= fromid) {
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
// something was deleted, we need to move around it
fromid += 1;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
} else if (lfs_tagid(tag) != fromid) {
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
// ignore non-matching ids
} else {
// check if type has already been committed
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int32_t res = lfs_commitget(lfs, commit->block,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
commit->off, commit->ptag,
lfs_tagisuser(tag) ? 0x7ffff000 : 0x7c3ff000,
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
LFS_MKTAG(lfs_tagtype(tag), toid, 0),
0, NULL, true);
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (res < 0 && res != LFS_ERR_NOENT) {
return res;
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (res == LFS_ERR_NOENT) {
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
// update id and commit, as we are currently unique
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_commitattr(lfs, commit,
(tag & 0xffc00fff) | LFS_MKTAG(0, toid, 0),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
buffer);
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
if (err) {
return err;
}
}
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
}
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
return 0;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_commitglobals(lfs_t *lfs, struct lfs_commit *commit,
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_global_t *locals) {
if (lfs_globaliszero(&lfs->locals)) {
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
return 0;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(locals, &lfs->locals);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_commitattr(lfs, commit,
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
LFS_MKTAG(LFS_TYPE_GLOBALS + locals->s.deorphaned,
0x3ff, sizeof(lfs_global_t)), locals);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(locals, &lfs->locals);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
return err;
}
static int lfs_commitcrc(lfs_t *lfs, struct lfs_commit *commit) {
// align to program units
lfs_off_t off = lfs_alignup(commit->off + 2*sizeof(uint32_t),
lfs->cfg->prog_size);
// read erased state from next program unit
uint32_t tag;
int err = lfs_bd_read(lfs, commit->block, off, &tag, sizeof(tag));
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
if (err) {
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// build crc tag
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
tag = lfs_fromle32(tag);
tag = (0x80000000 & ~tag) |
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
LFS_MKTAG(LFS_TYPE_CRC, 0x3ff,
off - (commit->off+sizeof(uint32_t)));
// write out crc
uint32_t footer[2];
footer[0] = lfs_tole32(tag ^ commit->ptag);
lfs_crc(&commit->crc, &footer[0], sizeof(footer[0]));
footer[1] = lfs_tole32(commit->crc);
err = lfs_bd_prog(lfs, commit->block, commit->off, footer, sizeof(footer));
if (err) {
return err;
}
commit->off += sizeof(tag)+lfs_tagsize(tag);
commit->ptag = tag;
// flush buffers
err = lfs_bd_sync(lfs);
if (err) {
return err;
}
// successful commit, check checksum to make sure
uint32_t crc = 0xffffffff;
err = lfs_bd_crc(lfs, commit->block, commit->begin,
commit->off-lfs_tagsize(tag)-commit->begin, &crc);
if (err) {
return err;
}
if (crc != commit->crc) {
return LFS_ERR_CORRUPT;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
return 0;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// internal dir operations
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
static int lfs_dir_alloc(lfs_t *lfs, lfs_mdir_t *dir) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// allocate pair of dir blocks (backwards, so we write to block 1 first)
for (int i = 0; i < 2; i++) {
int err = lfs_alloc(lfs, &dir->pair[(i+1)%2]);
if (err) {
return err;
}
}
// rather than clobbering one of the blocks we just pretend
// the revision may be valid
int err = lfs_bd_read(lfs, dir->pair[0], 0, &dir->rev, 4);
dir->rev = lfs_fromle32(dir->rev);
if (err) {
return err;
}
// set defaults
dir->off = sizeof(dir->rev);
dir->etag = 0;
dir->count = 0;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
dir->tail[0] = 0xffffffff;
dir->tail[1] = 0xffffffff;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->erased = false;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
dir->split = false;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalzero(&dir->locals);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// don't write out yet, let caller take care of that
return 0;
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
static int32_t lfs_dir_find(lfs_t *lfs,
lfs_mdir_t *dir, const lfs_block_t pair[2],
uint32_t findmask, uint32_t findtag,
const void *findbuffer) {
dir->pair[0] = pair[0];
dir->pair[1] = pair[1];
int32_t foundtag = LFS_ERR_NOENT;
// find the block with the most recent revision
uint32_t rev[2];
for (int i = 0; i < 2; i++) {
int err = lfs_bd_read(lfs, dir->pair[i], 0, &rev[i], sizeof(rev[i]));
if (err) {
return err;
}
rev[i] = lfs_fromle32(rev[i]);
}
if (lfs_scmp(rev[1], rev[0]) > 0) {
lfs_pairswap(dir->pair);
lfs_pairswap(rev);
}
// load blocks and check crc
for (int i = 0; i < 2; i++) {
lfs_off_t off = sizeof(dir->rev);
uint32_t ptag = 0;
uint32_t crc = 0xffffffff;
dir->rev = lfs_tole32(rev[0]);
lfs_crc(&crc, &dir->rev, sizeof(dir->rev));
dir->rev = lfs_fromle32(dir->rev);
dir->off = 0;
uint32_t tempfoundtag = foundtag;
uint16_t tempcount = 0;
lfs_block_t temptail[2] = {0xffffffff, 0xffffffff};
bool tempsplit = false;
lfs_global_t templocals;
lfs_globalzero(&templocals);
while (true) {
// extract next tag
uint32_t tag;
int err = lfs_bd_read(lfs, dir->pair[0],
off, &tag, sizeof(tag));
if (err) {
return err;
}
lfs_crc(&crc, &tag, sizeof(tag));
tag = lfs_fromle32(tag) ^ ptag;
// next commit not yet programmed
if (lfs_tagtype(ptag) == LFS_TYPE_CRC && !lfs_tagisvalid(tag)) {
dir->erased = true;
break;
}
// check we're in valid range
if (off + sizeof(tag)+lfs_tagsize(tag) > lfs->cfg->block_size) {
dir->erased = false;
break;
}
if (lfs_tagtype(tag) == LFS_TYPE_CRC) {
// check the crc attr
uint32_t dcrc;
int err = lfs_bd_read(lfs, dir->pair[0],
off+sizeof(tag), &dcrc, sizeof(dcrc));
if (err) {
return err;
}
dcrc = lfs_fromle32(dcrc);
if (crc != dcrc) {
dir->erased = false;
break;
}
foundtag = tempfoundtag;
dir->off = off + sizeof(tag)+lfs_tagsize(tag);
dir->etag = tag;
dir->count = tempcount;
dir->tail[0] = temptail[0];
dir->tail[1] = temptail[1];
dir->split = tempsplit;
dir->locals = templocals;
crc = 0xffffffff;
} else {
err = lfs_bd_crc(lfs, dir->pair[0],
off+sizeof(tag), lfs_tagsize(tag), &crc);
if (err) {
return err;
}
if (lfs_tagid(tag) < 0x3ff && lfs_tagid(tag) >= tempcount) {
tempcount = lfs_tagid(tag)+1;
}
if (lfs_tagsubtype(tag) == LFS_TYPE_TAIL) {
tempsplit = (lfs_tagtype(tag) & 1);
err = lfs_bd_read(lfs, dir->pair[0], off+sizeof(tag),
temptail, sizeof(temptail));
if (err) {
return err;
}
lfs_pairfromle32(temptail);
} else if (lfs_tagsubtype(tag) == LFS_TYPE_GLOBALS) {
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
templocals.s.deorphaned = (lfs_tagtype(tag) & 1);
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = lfs_bd_read(lfs, dir->pair[0], off+sizeof(tag),
&templocals, sizeof(templocals));
if (err) {
return err;
}
} else if (lfs_tagsubtype(tag) == LFS_TYPE_DELETE) {
LFS_ASSERT(tempcount > 0);
tempcount -= 1;
if (lfs_tagid(tag) == lfs_tagid(tempfoundtag)) {
tempfoundtag = LFS_ERR_NOENT;
} else if (lfs_tagisvalid(tempfoundtag) &&
lfs_tagid(tag) < lfs_tagid(tempfoundtag)) {
tempfoundtag -= LFS_MKTAG(0, 1, 0);
}
} else if ((tag & findmask) == (findtag & findmask)) {
int res = lfs_bd_cmp(lfs, dir->pair[0], off+sizeof(tag),
findbuffer, lfs_tagsize(tag));
if (res < 0) {
return res;
}
if (res) {
// found a match
tempfoundtag = tag;
}
}
}
ptag = tag;
off += sizeof(tag)+lfs_tagsize(tag);
}
// consider what we have good enough
if (dir->off > 0) {
// synthetic move
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
if (lfs_paircmp(dir->pair, lfs->globals.s.movepair) == 0) {
if (lfs->globals.s.moveid == lfs_tagid(foundtag)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
foundtag = LFS_ERR_NOENT;
} else if (lfs_tagisvalid(foundtag) &&
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs->globals.s.moveid < lfs_tagid(foundtag)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
foundtag -= LFS_MKTAG(0, 1, 0);
}
}
return foundtag;
}
// failed, try the other crc?
lfs_pairswap(dir->pair);
lfs_pairswap(rev);
}
LFS_ERROR("Corrupted dir pair at %d %d", dir->pair[0], dir->pair[1]);
return LFS_ERR_CORRUPT;
}
static int lfs_dir_fetch(lfs_t *lfs,
lfs_mdir_t *dir, const lfs_block_t pair[2]) {
int32_t res = lfs_dir_find(lfs, dir, pair, 0xffffffff, 0xffffffff, NULL);
if (res < 0 && res != LFS_ERR_NOENT) {
return res;
}
return 0;
}
static int32_t lfs_dir_get(lfs_t *lfs, lfs_mdir_t *dir,
uint32_t getmask, uint32_t gettag, void *buffer) {
int32_t getdiff = 0;
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
if (lfs_paircmp(dir->pair, lfs->globals.s.movepair) == 0 &&
lfs_tagid(gettag) <= lfs->globals.s.moveid) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// synthetic moves
getdiff = LFS_MKTAG(0, 1, 0);
}
return lfs_commitget(lfs, dir->pair[0], dir->off, dir->etag,
getmask, gettag, getdiff, buffer, false);
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_dir_compact(lfs_t *lfs,
lfs_mdir_t *dir, const lfs_mattr_t *attrs,
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *source, uint16_t begin, uint16_t end) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// save some state in case block is bad
const lfs_block_t oldpair[2] = {dir->pair[1], dir->pair[0]};
bool relocated = false;
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
// There's nothing special about our global delta, so feed it back
// into the global global delta
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&lfs->locals, &dir->locals);
lfs_globalzero(&dir->locals);
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// increment revision count
dir->rev += 1;
while (true) {
// last complete id
int16_t ack = -1;
dir->count = end - begin;
if (true) {
// erase block to write to
int err = lfs_bd_erase(lfs, dir->pair[1]);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// write out header
uint32_t crc = 0xffffffff;
uint32_t rev = lfs_tole32(dir->rev);
lfs_crc(&crc, &rev, sizeof(rev));
err = lfs_bd_prog(lfs, dir->pair[1], 0, &rev, sizeof(rev));
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// setup compaction
struct lfs_commit commit = {
.block = dir->pair[1],
.off = sizeof(dir->rev),
.crc = crc,
.ptag = 0,
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// space is complicated, we need room for tail, crc, globals,
// and we cap at half a block to give room for metadata updates
.begin = 0,
.end = lfs_min(
lfs_alignup(lfs->cfg->block_size/2, lfs->cfg->prog_size),
lfs->cfg->block_size - 34),
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
};
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// commit with a move
for (uint16_t id = begin; id < end; id++) {
err = lfs_commitmove(lfs, &commit,
id, id - begin, source, attrs);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
if (err == LFS_ERR_NOSPC) {
goto split;
} else if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
ack = id;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// reopen reserved space at the end
commit.end = lfs->cfg->block_size - 8;
if (!relocated) {
err = lfs_commitglobals(lfs, &commit, &dir->locals);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err == LFS_ERR_CORRUPT) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
goto relocate;
}
return err;
}
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
if (!lfs_pairisnull(dir->tail)) {
Removed old move logic, now passing move tests The introduction of xored-globals required quite a bit of work to integrate. But now that that is working, we can strip out the old move logic. It's worth noting that the xored-globals integration with commits is relatively complex and subtle.
2018-07-09 16:47:04 +00:00
// commit tail, which may be new after last size check
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairtole32(dir->tail);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
err = lfs_commitattr(lfs, &commit,
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
LFS_MKTAG(LFS_TYPE_TAIL + dir->split,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
0x3ff, sizeof(dir->tail)), dir->tail);
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(dir->tail);
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
err = lfs_commitcrc(lfs, &commit);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// successful compaction, swap dir pair to indicate most recent
lfs_pairswap(dir->pair);
dir->off = commit.off;
dir->etag = commit.ptag;
dir->erased = true;
}
break;
split:
// commit no longer fits, need to split dir,
// drop caches and create tail
lfs->pcache.block = 0xffffffff;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
if (ack == -1) {
// If we can't fit in this block, we won't fit in next block
return LFS_ERR_NOSPC;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t tail;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
int err = lfs_dir_alloc(lfs, &tail);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
return err;
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
tail.split = dir->split;
tail.tail[0] = dir->tail[0];
tail.tail[1] = dir->tail[1];
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
err = lfs_dir_compact(lfs, &tail, attrs, dir, ack+1, end);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
return err;
}
end = ack+1;
dir->tail[0] = tail.pair[0];
dir->tail[1] = tail.pair[1];
dir->split = true;
continue;
relocate:
//commit was corrupted
LFS_DEBUG("Bad block at %d", dir->pair[1]);
// drop caches and prepare to relocate block
relocated = true;
lfs->pcache.block = 0xffffffff;
// can't relocate superblock, filesystem is now frozen
if (lfs_paircmp(oldpair, (const lfs_block_t[2]){0, 1}) == 0) {
LFS_WARN("Superblock %d has become unwritable", oldpair[1]);
return LFS_ERR_CORRUPT;
}
// relocate half of pair
err = lfs_alloc(lfs, &dir->pair[1]);
if (err) {
return err;
}
continue;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (!relocated) {
// successful commit, update globals
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&dir->locals, &lfs->locals);
lfs_globalzero(&lfs->locals);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
} else {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// update references if we relocated
LFS_DEBUG("Relocating %d %d to %d %d",
oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]);
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_relocate(lfs, oldpair, dir->pair);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
return err;
}
}
return 0;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir,
const lfs_mattr_t *attrs) {
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_mattr_t cancelattr;
lfs_global_t canceldiff;
lfs_globalzero(&canceldiff);
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
if (lfs_paircmp(dir->pair, lfs->globals.s.movepair) == 0) {
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// Wait, we have the move? Just cancel this out here
// We need to, or else the move can become outdated
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
canceldiff.s.movepair[0] ^= lfs->globals.s.movepair[0] ^ 0xffffffff;
canceldiff.s.movepair[1] ^= lfs->globals.s.movepair[1] ^ 0xffffffff;
canceldiff.s.moveid ^= lfs->globals.s.moveid ^ 0x3ff;
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_globalfromle32(&lfs->locals);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&lfs->locals, &canceldiff);
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_globaltole32(&lfs->locals);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
cancelattr.tag = LFS_MKTAG(LFS_TYPE_DELETE, lfs->globals.s.moveid, 0);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
cancelattr.next = attrs;
attrs = &cancelattr;
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
}
// calculate new directory size
uint32_t deletetag = 0xffffffff;
for (const lfs_mattr_t *a = attrs; a; a = a->next) {
if (lfs_tagid(a->tag) < 0x3ff && lfs_tagid(a->tag) >= dir->count) {
dir->count = lfs_tagid(a->tag)+1;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
if (lfs_tagtype(a->tag) == LFS_TYPE_DELETE) {
LFS_ASSERT(dir->count > 0);
dir->count -= 1;
deletetag = a->tag;
if (dir->count == 0) {
// should we actually drop the directory block?
lfs_mdir_t pdir;
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_pred(lfs, dir->pair, &pdir);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err != LFS_ERR_NOENT && pdir.split) {
// steal tail and global state
pdir.split = dir->split;
pdir.tail[0] = dir->tail[0];
pdir.tail[1] = dir->tail[1];
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&lfs->locals, &dir->locals);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
return lfs_dir_commit(lfs, &pdir,
LFS_MKATTR(LFS_TYPE_TAIL + pdir.split, 0x3ff,
pdir.tail, sizeof(pdir.tail),
NULL));
}
}
}
}
if (!dir->erased) {
compact:
// fall back to compaction
lfs->pcache.block = 0xffffffff;
int err = lfs_dir_compact(lfs, dir, attrs, dir, 0, dir->count);
if (err) {
return err;
}
} else {
// try to commit
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
struct lfs_commit commit = {
.block = dir->pair[0],
.off = dir->off,
.crc = 0xffffffff,
.ptag = dir->etag,
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
.begin = dir->off,
.end = lfs->cfg->block_size - 8,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
};
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
for (const lfs_mattr_t *a = attrs; a; a = a->next) {
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
if (lfs_tagtype(a->tag) != LFS_TYPE_DELETE) {
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairtole32(dir->tail);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
int err = lfs_commitattr(lfs, &commit, a->tag, a->buffer);
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(dir->tail);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
return err;
}
}
}
if (lfs_tagisvalid(deletetag)) {
// special case for deletes, since order matters
int err = lfs_commitattr(lfs, &commit, deletetag, NULL);
Restructured get/traverse functions While it makes sense to reuse as many code paths as possible, it turns out that the logic behind the traversal of littlefs's metadata-pairs is so simple that it's actually cheaper to duplicate the traversal code where needed. This means instead of the code path move -> traverse -> movescan -> get -> traverse -> getscan, we can use the relatively flatter code path of move -> get.
2018-07-12 23:11:18 +00:00
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
return err;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
}
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_commitglobals(lfs, &commit, &dir->locals);
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
err = lfs_commitcrc(lfs, &commit);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
return err;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// successful commit, update dir
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->off = commit.off;
dir->etag = commit.ptag;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// successful commit, update globals
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&dir->locals, &lfs->locals);
lfs_globalzero(&lfs->locals);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// update globals that are affected
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&lfs->globals, &canceldiff);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// update any directories that are affected
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
for (lfs_mlist_t *d = lfs->mlist; d; d = d->next) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (lfs_paircmp(d->m.pair, dir->pair) == 0) {
d->m = *dir;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
if (d->id == lfs_tagid(deletetag)) {
d->m.pair[0] = 0xffffffff;
d->m.pair[1] = 0xffffffff;
} else if (d->id > lfs_tagid(deletetag)) {
d->id -= 1;
if (d->type == LFS_TYPE_DIR) {
((lfs_dir_t*)d)->pos -= 1;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
while (d->id >= d->m.count && d->m.split) {
// we split and id is on tail now
d->id -= d->m.count;
int err = lfs_dir_fetch(lfs, &d->m, d->m.tail);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (err) {
return err;
}
}
}
}
return 0;
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
static int32_t lfs_dir_lookup(lfs_t *lfs, lfs_mdir_t *dir, const char **path) {
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
// we reduce path to a single name if we can find it
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
const char *name = *path;
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
*path = NULL;
// default to root dir
int32_t tag = LFS_MKTAG(LFS_TYPE_DIR, 0x3ff, 0);
lfs_block_t pair[2] = {lfs->root[0], lfs->root[1]};
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
while (true) {
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
nextname:
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// skip slashes
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
name += strspn(name, "/");
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
lfs_size_t namelen = strcspn(name, "/");
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// skip '.' and root '..'
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
if ((namelen == 1 && memcmp(name, ".", 1) == 0) ||
(namelen == 2 && memcmp(name, "..", 2) == 0)) {
name += namelen;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
goto nextname;
}
// skip if matched by '..' in name
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
const char *suffix = name + namelen;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
lfs_size_t sufflen;
int depth = 1;
while (true) {
suffix += strspn(suffix, "/");
sufflen = strcspn(suffix, "/");
if (sufflen == 0) {
break;
}
if (sufflen == 2 && memcmp(suffix, "..", 2) == 0) {
depth -= 1;
if (depth == 0) {
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
name = suffix + sufflen;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
goto nextname;
}
} else {
depth += 1;
}
suffix += sufflen;
}
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
// found path
if (name[0] == '\0') {
return tag;
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
// update what we've found if path is only a name
if (strchr(name, '/') == NULL) {
*path = name;
}
// only continue if we hit a directory
if (lfs_tagtype(tag) != LFS_TYPE_DIR) {
return LFS_ERR_NOTDIR;
}
// grab the entry data
if (lfs_tagid(tag) != 0x3ff) {
int32_t res = lfs_dir_get(lfs, dir, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tagid(tag), 8), pair);
if (res < 0) {
return res;
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(pair);
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
}
// find entry matching name
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
while (true) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
tag = lfs_dir_find(lfs, dir, pair, 0x7c000fff,
LFS_MKTAG(LFS_TYPE_NAME, 0, namelen), name);
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag < 0 && tag != LFS_ERR_NOENT) {
return tag;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag != LFS_ERR_NOENT) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// found it
break;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (!dir->split) {
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
// couldn't find it
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return LFS_ERR_NOENT;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
pair[0] = dir->tail[0];
pair[1] = dir->tail[1];
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
// to next name
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
name += namelen;
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
}
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_dir_getinfo(lfs_t *lfs, lfs_mdir_t *dir,
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
uint16_t id, struct lfs_info *info) {
if (id == 0x3ff) {
// special case for root
strcpy(info->name, "/");
info->type = LFS_TYPE_DIR;
return 0;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int32_t tag = lfs_dir_get(lfs, dir, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_NAME, id, lfs->name_size+1), info->name);
if (tag < 0) {
return tag;
}
info->type = lfs_tagtype(tag);
struct lfs_ctz ctz;
tag = lfs_dir_get(lfs, dir, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_STRUCT, id, sizeof(ctz)), &ctz);
if (tag < 0) {
return tag;
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_ctzfromle32(&ctz);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
if (lfs_tagtype(tag) == LFS_TYPE_CTZSTRUCT) {
info->size = ctz.size;
} else if (lfs_tagtype(tag) == LFS_TYPE_INLINESTRUCT) {
info->size = lfs_tagsize(tag);
}
return 0;
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level directory operations ///
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
int lfs_mkdir(lfs_t *lfs, const char *path) {
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
return err;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
int32_t res = lfs_dir_lookup(lfs, &cwd, &path);
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
if (!(res == LFS_ERR_NOENT && path)) {
Pulled in fixes for additional path corner cases Pulled in 015b86b. Merging this now avoids duplicate effort restructuring the path lookup logic.
2018-07-28 17:07:18 +00:00
return (res < 0) ? res : LFS_ERR_EXIST;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
// check that name fits
lfs_size_t nlen = strlen(path);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
// build up new directory
lfs_alloc_ack(lfs);
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = lfs_dir_alloc(lfs, &dir);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
dir.tail[0] = cwd.tail[0];
dir.tail[1] = cwd.tail[1];
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
err = lfs_dir_commit(lfs, &dir, NULL);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
// get next slot and commit
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
uint16_t id = cwd.count;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
cwd.tail[0] = dir.pair[0];
cwd.tail[1] = dir.pair[1];
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairtole32(dir.pair);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
err = lfs_dir_commit(lfs, &cwd,
LFS_MKATTR(LFS_TYPE_DIR, id, path, nlen,
LFS_MKATTR(LFS_TYPE_DIRSTRUCT, id, dir.pair, sizeof(dir.pair),
LFS_MKATTR(LFS_TYPE_SOFTTAIL, 0x3ff, cwd.tail, sizeof(cwd.tail),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
NULL))));
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(dir.pair);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_open(lfs_t *lfs, lfs_dir_t *dir, const char *path) {
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
int32_t tag = lfs_dir_lookup(lfs, &dir->m, &path);
if (tag < 0) {
return tag;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (lfs_tagtype(tag) != LFS_TYPE_DIR) {
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
return LFS_ERR_NOTDIR;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_block_t pair[2];
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (lfs_tagid(tag) == 0x3ff) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// handle root dir separately
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
pair[0] = lfs->root[0];
pair[1] = lfs->root[1];
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else {
// get dir pair from parent
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
int32_t res = lfs_dir_get(lfs, &dir->m, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tagid(tag), 8), pair);
if (res < 0) {
return res;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// fetch first pair
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// setup entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->head[0] = dir->m.pair[0];
dir->head[1] = dir->m.pair[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->id = 0;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->pos = 0;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// add to list of mdirs
dir->type = LFS_TYPE_DIR;
dir->next = (lfs_dir_t*)lfs->mlist;
lfs->mlist = (lfs_mlist_t*)dir;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// remove from list of mdirs
for (lfs_mlist_t **p = &lfs->mlist; *p; p = &(*p)->next) {
if (*p == (lfs_mlist_t*)dir) {
*p = (*p)->next;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
break;
}
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_read(lfs_t *lfs, lfs_dir_t *dir, struct lfs_info *info) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
memset(info, 0, sizeof(*info));
// special offset for '.' and '..'
if (dir->pos == 0) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, ".");
dir->pos += 1;
return 1;
} else if (dir->pos == 1) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, "..");
dir->pos += 1;
return 1;
}
while (true) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (dir->id == dir->m.count) {
if (!dir->m.split) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return false;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->m.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
dir->id = 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_getinfo(lfs, &dir->m, dir->id, info);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err && err != LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
dir->id += 1;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err != LFS_ERR_NOENT) {
break;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
dir->pos += 1;
return true;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_seek(lfs_t *lfs, lfs_dir_t *dir, lfs_off_t off) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// simply walk from head dir
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_rewind(lfs, dir);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// first two for ./..
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->pos = lfs_min(2, off);
off -= dir->pos;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
while (off != 0) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->id = lfs_min(dir->m.count, off);
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->pos += dir->id;
off -= dir->id;
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (dir->id == dir->m.count) {
if (!dir->m.split) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
return LFS_ERR_INVAL;
}
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->m.tail);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (err) {
return err;
}
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_soff_t lfs_dir_tell(lfs_t *lfs, lfs_dir_t *dir) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return dir->pos;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_rewind(lfs_t *lfs, lfs_dir_t *dir) {
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
// reload the head dir
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->head);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
return err;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->m.pair[0] = dir->head[0];
dir->m.pair[1] = dir->head[1];
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->id = 0;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->pos = 0;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// File index list operations ///
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_ctzindex(lfs_t *lfs, lfs_off_t *off) {
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
lfs_off_t size = *off;
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
lfs_off_t b = lfs->cfg->block_size - 2*4;
lfs_off_t i = size / b;
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
if (i == 0) {
return 0;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
i = (size - 4*(lfs_popc(i-1)+2)) / b;
*off = size - b*i - 4*lfs_popc(i);
return i;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_ctzfind(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, const lfs_cache_t *pcache,
lfs_block_t head, lfs_size_t size,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t pos, lfs_block_t *block, lfs_off_t *off) {
if (size == 0) {
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
*off = 0;
return 0;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
lfs_off_t current = lfs_ctzindex(lfs, &(lfs_off_t){size-1});
lfs_off_t target = lfs_ctzindex(lfs, &pos);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
while (current > target) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t skip = lfs_min(
lfs_npw2(current-target+1) - 1,
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
lfs_ctz(current));
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
int err = lfs_cache_read(lfs, pcache, rcache, false,
head, 4*skip, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
current -= 1 << skip;
}
*block = head;
*off = pos;
return 0;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_ctzextend(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, lfs_cache_t *pcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
lfs_block_t *block, lfs_off_t *off) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
// go ahead and grab a block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(nblock >= 2 && nblock <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
if (true) {
err = lfs_bd_erase(lfs, nblock);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (size == 0) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 0;
return 0;
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
size -= 1;
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
lfs_off_t index = lfs_ctzindex(lfs, &size);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
size += 1;
// just copy out the last block if it is incomplete
if (size != lfs->cfg->block_size) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t data;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
err = lfs_cache_read(lfs, NULL, rcache, true,
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
head, i, &data, 1);
if (err) {
return err;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
err = lfs_cache_prog(lfs, pcache, rcache, true,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, i, &data, 1);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = size;
return 0;
}
// append block
index += 1;
lfs_size_t skips = lfs_ctz(index) + 1;
for (lfs_off_t i = 0; i < skips; i++) {
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_tole32(head);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
err = lfs_cache_prog(lfs, pcache, rcache, true,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (i != skips-1) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
err = lfs_cache_read(lfs, NULL, rcache, false,
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
head, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 4*skips;
return 0;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
relocate:
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
LFS_DEBUG("Bad block at %d", nblock);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// just clear cache and try a new block
pcache->block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
static int lfs_ctztraverse(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, const lfs_cache_t *pcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int (*cb)(void*, lfs_block_t), void *data) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (size == 0) {
return 0;
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
lfs_off_t index = lfs_ctzindex(lfs, &(lfs_off_t){size-1});
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
while (true) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int err = cb(data, head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
if (index == 0) {
return 0;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
lfs_block_t heads[2];
int count = 2 - (index & 1);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
err = lfs_cache_read(lfs, pcache, rcache, false,
head, 0, &heads, count*4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
heads[0] = lfs_fromle32(heads[0]);
heads[1] = lfs_fromle32(heads[1]);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
for (int i = 0; i < count-1; i++) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
err = cb(data, heads[i]);
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
if (err) {
return err;
}
}
head = heads[count-1];
index -= count;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level file operations ///
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
int lfs_file_opencfg(lfs_t *lfs, lfs_file_t *file,
const char *path, int flags,
const struct lfs_file_config *cfg) {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
// deorphan if we haven't yet, needed at most once after poweron
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if ((flags & 3) != LFS_O_RDONLY) {
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// setup simple file details
int err = 0;
file->cfg = cfg;
file->flags = flags;
file->pos = 0;
file->cache.buffer = NULL;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
// allocate entry for file if it doesn't exist
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
int32_t tag = lfs_dir_lookup(lfs, &file->m, &path);
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
if (tag < 0 && !(tag == LFS_ERR_NOENT && path)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = tag;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// get id, add to list of mdirs to catch update changes
file->id = lfs_tagid(tag);
file->type = LFS_TYPE_REG;
file->next = (lfs_file_t*)lfs->mlist;
lfs->mlist = (lfs_mlist_t*)file;
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (tag == LFS_ERR_NOENT) {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (!(flags & LFS_O_CREAT)) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOENT;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
// check that name fits
lfs_size_t nlen = strlen(path);
if (nlen > lfs->name_size) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NAMETOOLONG;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
// get next slot and create entry to remember name
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
file->id = file->m.count;
int err = lfs_dir_commit(lfs, &file->m,
LFS_MKATTR(LFS_TYPE_REG, file->id, path, nlen,
LFS_MKATTR(LFS_TYPE_INLINESTRUCT, file->id, NULL, 0,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
NULL)));
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NAMETOOLONG;
goto cleanup;
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
}
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
tag = LFS_MKTAG(LFS_TYPE_INLINESTRUCT, 0, 0);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
} else if (flags & LFS_O_EXCL) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_EXIST;
goto cleanup;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
} else if (lfs_tagtype(tag) != LFS_TYPE_REG) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_ISDIR;
goto cleanup;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
} else if (flags & LFS_O_TRUNC) {
// truncate if requested
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
tag = LFS_MKTAG(LFS_TYPE_INLINESTRUCT, file->id, 0);
file->flags |= LFS_F_DIRTY;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
// try to load what's on disk, if it's inlined we'll fix it later
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
tag = lfs_dir_get(lfs, &file->m, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_STRUCT, file->id, 8), &file->ctz);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag < 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = tag;
goto cleanup;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_ctzfromle32(&file->ctz);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
// fetch attrs
for (const struct lfs_attr *a = file->cfg->attrs; a; a = a->next) {
if ((file->flags & 3) != LFS_O_WRONLY) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
int32_t res = lfs_dir_get(lfs, &file->m, 0x7ffff000,
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
LFS_MKTAG(0x100 | a->type, file->id, a->size), a->buffer);
if (res < 0 && res != LFS_ERR_NOENT) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = res;
goto cleanup;
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
}
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if ((file->flags & 3) != LFS_O_RDONLY) {
if (a->size > lfs->attr_size) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOSPC;
goto cleanup;
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
}
file->flags |= LFS_F_DIRTY;
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
// allocate buffer if needed
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
file->cache.block = 0xffffffff;
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if (file->cfg->buffer) {
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
file->cache.buffer = file->cfg->buffer;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = LFS_ERR_NOMEM;
goto cleanup;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (lfs_tagtype(tag) == LFS_TYPE_INLINESTRUCT) {
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
// load inline files
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head = 0xfffffffe;
file->ctz.size = lfs_tagsize(tag);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->flags |= LFS_F_INLINE;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->cache.block = file->ctz.head;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->cache.off = 0;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.size = lfs->cfg->cache_size;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
// don't always read (may be new/trunc file)
if (file->ctz.size > 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
int32_t res = lfs_dir_get(lfs, &file->m, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_STRUCT, file->id, file->ctz.size),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->cache.buffer);
if (res < 0) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = res;
goto cleanup;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
return 0;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
cleanup:
// clean up lingering resources
file->flags |= LFS_F_ERRED;
lfs_file_close(lfs, file);
return err;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
int lfs_file_open(lfs_t *lfs, lfs_file_t *file,
const char *path, int flags) {
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
static const struct lfs_file_config defaults = {0};
return lfs_file_opencfg(lfs, file, path, flags, &defaults);
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_close(lfs_t *lfs, lfs_file_t *file) {
int err = lfs_file_sync(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// remove from list of mdirs
for (lfs_mlist_t **p = &lfs->mlist; *p; p = &(*p)->next) {
if (*p == (lfs_mlist_t*)file) {
*p = (*p)->next;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
break;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// clean up memory
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
if (file->cfg->buffer) {
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
lfs_free(file->cache.buffer);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
return err;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_relocate(lfs_t *lfs, lfs_file_t *file) {
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
while (true) {
// just relocate what exists into new block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
Adopted redundant cache read in lfs_file_relocate Previously had some custom logic that could be reduced
2017-06-25 22:23:40 +00:00
if (err) {
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
err = lfs_bd_erase(lfs, nblock);
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
// either read from dirty cache or disk
for (lfs_off_t i = 0; i < file->off; i++) {
uint8_t data;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
err = lfs_cache_read(lfs, &file->cache, &lfs->rcache, true,
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
file->block, i, &data, 1);
if (err) {
return err;
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
err = lfs_cache_prog(lfs, &lfs->pcache, &lfs->rcache, true,
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
nblock, i, &data, 1);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
}
// copy over new state of file
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
memcpy(file->cache.buffer, lfs->pcache.buffer, lfs->cfg->cache_size);
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
file->cache.block = lfs->pcache.block;
file->cache.off = lfs->pcache.off;
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
file->cache.size = lfs->pcache.size;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
lfs->pcache.block = 0xffffffff;
file->block = nblock;
return 0;
relocate:
continue;
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file) {
if (file->flags & LFS_F_READING) {
file->flags &= ~LFS_F_READING;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (file->flags & LFS_F_WRITING) {
lfs_off_t pos = file->pos;
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (!(file->flags & LFS_F_INLINE)) {
// copy over anything after current branch
lfs_file_t orig = {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
.ctz.head = file->ctz.head,
.ctz.size = file->ctz.size,
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
.flags = LFS_O_RDONLY,
.pos = file->pos,
.cache = lfs->rcache,
};
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
lfs->rcache.block = 0xffffffff;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
while (file->pos < file->ctz.size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// copy over a byte at a time, leave it up to caching
// to make this efficient
uint8_t data;
lfs_ssize_t res = lfs_file_read(lfs, &orig, &data, 1);
if (res < 0) {
return res;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
res = lfs_file_write(lfs, file, &data, 1);
if (res < 0) {
return res;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// keep our reference to the rcache in sync
if (lfs->rcache.block != 0xffffffff) {
orig.cache.block = 0xffffffff;
lfs->rcache.block = 0xffffffff;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// write out what we have
while (true) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
int err = lfs_cache_flush(lfs,
&file->cache, &lfs->rcache, true);
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
break;
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
relocate:
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
LFS_DEBUG("Bad block at %d", file->block);
err = lfs_file_relocate(lfs, file);
if (err) {
return err;
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.size = lfs_max(file->pos, file->ctz.size);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// actual file updates
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head = file->block;
file->ctz.size = file->pos;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_WRITING;
file->flags |= LFS_F_DIRTY;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = pos;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) {
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
while (true) {
int err = lfs_file_flush(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
if ((file->flags & LFS_F_DIRTY) &&
!(file->flags & LFS_F_ERRED) &&
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
!lfs_pairisnull(file->m.pair)) {
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
// update dir entry
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
uint16_t type;
const void *buffer;
lfs_size_t size;
if (file->flags & LFS_F_INLINE) {
// inline the whole file
type = LFS_TYPE_INLINESTRUCT;
buffer = file->cache.buffer;
size = file->ctz.size;
} else {
// update the ctz reference
type = LFS_TYPE_CTZSTRUCT;
buffer = &file->ctz;
size = sizeof(file->ctz);
}
// commit file data and attributes
lfs_ctztole32(&file->ctz);
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
int err = lfs_dir_commit(lfs, &file->m,
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
LFS_MKATTR(type, file->id, buffer, size,
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
LFS_MKATTR(LFS_FROM_ATTRS, file->id, file->cfg->attrs, 0,
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
NULL)));
lfs_ctzfromle32(&file->ctz);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
if (err) {
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
if (err == LFS_ERR_NOSPC && (file->flags & LFS_F_INLINE)) {
goto relocate;
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
return err;
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
file->flags &= ~LFS_F_DIRTY;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Fixed ENOSPC issues with zero-granularity blocks Result of testing on zero-granularity blocks, where the prog size and read size equals the block size. This represents SD cards and other traditional forms of block storage where we don't really get a benefit from the metadata logging. Unfortunately, since updates in both are tested by the same script, we can't really use simple bash commands. Added a more complex script to simulate corruption. Fortunately this should be more robust than the previous solutions. The main fixes were around corner cases where the commit logic fell apart when it didn't have room to complete commits, but these were fixable in the current design.
2018-07-31 02:12:00 +00:00
relocate:
// inline file doesn't fit anymore
file->block = 0xfffffffe;
file->off = file->pos;
lfs_alloc_ack(lfs);
err = lfs_file_relocate(lfs, file);
if (err) {
return err;
}
file->flags &= ~LFS_F_INLINE;
file->flags |= LFS_F_WRITING;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
void *buffer, lfs_size_t size) {
uint8_t *data = buffer;
lfs_size_t nsize = size;
if ((file->flags & 3) == LFS_O_WRONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_WRITING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// flush out any writes
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (file->pos >= file->ctz.size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
// eof if past end
return 0;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
size = lfs_min(size, file->ctz.size - file->pos);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
nsize = size;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
while (nsize > 0) {
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_READING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_ctzfind(lfs, &file->cache, NULL,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->pos, &file->block, &file->off);
if (err) {
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags |= LFS_F_READING;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
// read as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
int err = lfs_cache_read(lfs, NULL, &file->cache, true,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->block, file->off, data, diff);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
data += diff;
nsize -= diff;
}
return size;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
lfs_size_t nsize = size;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_READING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// drop any reads
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if ((file->flags & LFS_O_APPEND) && file->pos < file->ctz.size) {
file->pos = file->ctz.size;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > file->ctz.size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
// fill with zeros
lfs_off_t pos = file->pos;
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->pos = file->ctz.size;
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
while (file->pos < pos) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if ((file->flags & LFS_F_INLINE) &&
Cleaned up config usage in file logic The main change here was to drop the in-place twiddling of custom attributes to match the internal attribute structures. The original thought was that this could allow the compiler to garbage collect more of the custom attribute logic when not used, but since this occurs in the common lfs_file_opencfg function, gc can't really happen. Not twiddling the user's structure is the polite thing to do, opens up the ability to store the lfs_attr structure in ROM, and avoids surprising the user if they attempt to use the structure for their own purposes. This means we can make the lfs_attr structure const and rely on the list in the lfs_file_config structure, similar to how we rely on the global lfs_config structure. Some other tweaks: - Dropped the global file_buffer, replaced entirely by per-file buffers. - Updated LFS_INLINE_MAX and LFS_ATTR_MAX to correct values - Added workaround for compiler bug related to zero initializer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
2018-07-30 20:15:05 +00:00
file->pos + nsize >= lfs->inline_size) {
Bumped versions, cleaned up some TODOs and missing comments
2018-04-03 13:29:28 +00:00
// inline file doesn't fit anymore
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
file->block = 0xfffffffe;
file->off = file->pos;
lfs_alloc_ack(lfs);
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
int err = lfs_file_relocate(lfs, file);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
file->flags &= ~LFS_F_INLINE;
file->flags |= LFS_F_WRITING;
}
while (nsize > 0) {
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_WRITING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > 0) {
// find out which block we're extending from
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_ctzfind(lfs, &file->cache, NULL,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->pos-1, &file->block, &file->off);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
}
// mark cache as dirty since we may have read data into it
file->cache.block = 0xffffffff;
}
// extend file with new blocks
lfs_alloc_ack(lfs);
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_ctzextend(lfs, &lfs->rcache, &file->cache,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->block, file->pos,
&file->block, &file->off);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
file->flags |= LFS_F_WRITING;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// program as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
int err = lfs_cache_prog(lfs, &file->cache, &lfs->rcache, true,
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
file->block, file->off, data, diff);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
break;
relocate:
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
err = lfs_file_relocate(lfs, file);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
data += diff;
nsize -= diff;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_alloc_ack(lfs);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags &= ~LFS_F_ERRED;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
return size;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t off, int whence) {
// write out everything beforehand, may be noop if rdonly
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// update pos
if (whence == LFS_SEEK_SET) {
file->pos = off;
} else if (whence == LFS_SEEK_CUR) {
Fixed positive seek bounds checking This bug was a result of an annoying corner case around intermingling signed and unsigned offsets. The boundary check that prevents seeking a file to a position before the file was preventing valid seeks with positive offsets. This corner case is a bit more complicated than it looks because the offset is signed, while the size of the file is unsigned. Simply casting both to signed or unsigned offsets won't handle large files.
2018-01-03 21:00:04 +00:00
if (off < 0 && (lfs_off_t)-off > file->pos) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return LFS_ERR_INVAL;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = file->pos + off;
} else if (whence == LFS_SEEK_END) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (off < 0 && (lfs_off_t)-off > file->ctz.size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return LFS_ERR_INVAL;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->pos = file->ctz.size + off;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Fixed incorrect return value from lfs_file_seek lfs_file_seek returned the _previous_ file offset on success, where most standards return the _calculated_ offset on success. This just falls into me not noticing a mistake, and shows why it's always helpful to have a second set of eyes on code.
2017-09-27 00:50:39 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size) {
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
lfs_off_t oldsize = lfs_file_size(lfs, file);
if (size < oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
// need to flush since directly changing metadata
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
// lookup new head in ctz skip list
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
err = lfs_ctzfind(lfs, &file->cache, NULL,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.head, file->ctz.size,
size, &file->ctz.head, &(lfs_off_t){0});
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (err) {
return err;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
file->ctz.size = size;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
file->flags |= LFS_F_DIRTY;
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
} else if (size > oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
lfs_off_t pos = file->pos;
// flush+seek if not already at end
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
if (file->pos != oldsize) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
int err = lfs_file_seek(lfs, file, 0, LFS_SEEK_END);
Fixed error check when truncating files to larger size
2018-02-04 19:48:44 +00:00
if (err < 0) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
return err;
}
}
// fill with zeros
while (file->pos < size) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
// restore pos
int err = lfs_file_seek(lfs, file, pos, LFS_SEEK_SET);
if (err < 0) {
return err;
}
}
return 0;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_tell(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
int lfs_file_rewind(lfs_t *lfs, lfs_file_t *file) {
lfs_soff_t res = lfs_file_seek(lfs, file, 0, LFS_SEEK_SET);
if (res < 0) {
return res;
}
return 0;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (file->flags & LFS_F_WRITING) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return lfs_max(file->pos, file->ctz.size);
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
} else {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return file->ctz.size;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Commentary typo fix
2018-01-30 19:07:37 +00:00
/// General fs operations ///
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
int lfs_stat(lfs_t *lfs, const char *path, struct lfs_info *info) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
int32_t tag = lfs_dir_lookup(lfs, &cwd, &path);
if (tag < 0) {
return tag;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
return lfs_dir_getinfo(lfs, &cwd, lfs_tagid(tag), info);
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int lfs_remove(lfs_t *lfs, const char *path) {
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
err = lfs_dir_fetch(lfs, &cwd, lfs->root);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
int32_t tag = lfs_dir_lookup(lfs, &cwd, &path);
if (tag < 0) {
return tag;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs_mdir_t dir;
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (lfs_tagtype(tag) == LFS_TYPE_DIR) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// must be empty before removal
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_block_t pair[2];
int32_t res = lfs_dir_get(lfs, &cwd, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_STRUCT, lfs_tagid(tag), 8), pair);
if (res < 0) {
return res;
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(pair);
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
int err = lfs_dir_fetch(lfs, &dir, pair);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
if (dir.count > 0 || dir.split) {
return LFS_ERR_NOTEMPTY;
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// mark fs as orphaned
lfs_globaldeorphaned(lfs, false);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// delete the entry
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
err = lfs_dir_commit(lfs, &cwd,
LFS_MKATTR(LFS_TYPE_DELETE, lfs_tagid(tag), NULL, 0,
NULL));
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (lfs_tagtype(tag) == LFS_TYPE_DIR) {
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_pred(lfs, dir.pair, &cwd);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err) {
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// fix orphan
lfs_globaldeorphaned(lfs, true);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// steal state
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
cwd.tail[0] = dir.tail[0];
cwd.tail[1] = dir.tail[1];
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&lfs->locals, &dir.locals);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
err = lfs_dir_commit(lfs, &cwd,
LFS_MKATTR(LFS_TYPE_SOFTTAIL, 0x3ff,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
cwd.tail, sizeof(cwd.tail),
NULL));
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return 0;
}
int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) {
// deorphan if we haven't yet, needed at most once after poweron
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_forceconsistency(lfs);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// find old entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t oldcwd;
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
int32_t oldtag = lfs_dir_lookup(lfs, &oldcwd, &oldpath);
if (oldtag < 0) {
return oldtag;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// find new entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t newcwd;
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
int32_t prevtag = lfs_dir_lookup(lfs, &newcwd, &newpath);
if (prevtag < 0 && prevtag != LFS_ERR_NOENT) {
return prevtag;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
uint16_t newid = lfs_tagid(prevtag);
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
//bool prevexists = (prevtag != LFS_ERR_NOENT);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
//bool samepair = (lfs_paircmp(oldcwd.pair, newcwd.pair) == 0);
lfs_mdir_t prevdir;
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (prevtag != LFS_ERR_NOENT) {
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
// check that we have same type
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (lfs_tagtype(prevtag) != lfs_tagtype(oldtag)) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return LFS_ERR_ISDIR;
}
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (lfs_tagtype(prevtag) == LFS_TYPE_DIR) {
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
// must be empty before removal
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
lfs_block_t prevpair[2];
int32_t res = lfs_dir_get(lfs, &newcwd, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_STRUCT, newid, 8), prevpair);
if (res < 0) {
return res;
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(prevpair);
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// must be empty before removal
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
int err = lfs_dir_fetch(lfs, &prevdir, prevpair);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
if (prevdir.count > 0 || prevdir.split) {
return LFS_ERR_NOTEMPTY;
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// mark fs as orphaned
lfs_globaldeorphaned(lfs, false);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
} else {
// check that name fits
lfs_size_t nlen = strlen(newpath);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
// get next id
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
newid = newcwd.count;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// create move to fix later
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalmove(lfs, oldcwd.pair, lfs_tagid(oldtag));
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// move over all attributes
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
err = lfs_dir_commit(lfs, &newcwd,
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
LFS_MKATTR(lfs_tagtype(oldtag), newid, newpath, strlen(newpath),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
LFS_MKATTR(LFS_FROM_MOVE, newid, &oldcwd, lfs_tagid(oldtag),
NULL)));
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// let commit clean up after move (if we're different! otherwise move
// logic already fixed it for us)
if (lfs_paircmp(oldcwd.pair, newcwd.pair) != 0) {
err = lfs_dir_commit(lfs, &oldcwd, NULL);
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
if (prevtag != LFS_ERR_NOENT && lfs_tagtype(prevtag) == LFS_TYPE_DIR) {
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_pred(lfs, prevdir.pair, &newcwd);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err) {
return err;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// fix orphan
lfs_globaldeorphaned(lfs, true);
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
// steal state
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
newcwd.tail[0] = prevdir.tail[0];
newcwd.tail[1] = prevdir.tail[1];
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_globalxor(&lfs->locals, &prevdir.locals);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
err = lfs_dir_commit(lfs, &newcwd,
LFS_MKATTR(LFS_TYPE_SOFTTAIL, 0x3ff,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
newcwd.tail, sizeof(newcwd.tail),
NULL));
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
return 0;
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
lfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path,
uint8_t type, void *buffer, lfs_size_t size) {
lfs_mdir_t cwd;
int32_t res = lfs_dir_lookup(lfs, &cwd, &path);
if (res < 0) {
return res;
}
res = lfs_dir_get(lfs, &cwd, 0x7ffff000,
LFS_MKTAG(0x100 | type, lfs_tagid(res),
lfs_min(size, lfs->attr_size)), buffer);
Updated custom attribute documentation and tweaked nonexistant attributes Because of limitations in how littlefs manages attributes on disk, littlefs views zero-length attributes and missing attributes as the same thing. The simpliest implementation of attributes mirrors this behaviour transparently for the user.
2018-08-05 00:23:49 +00:00
if (res < 0 && res != LFS_ERR_NOENT) {
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
return res;
}
Updated custom attribute documentation and tweaked nonexistant attributes Because of limitations in how littlefs manages attributes on disk, littlefs views zero-length attributes and missing attributes as the same thing. The simpliest implementation of attributes mirrors this behaviour transparently for the user.
2018-08-05 00:23:49 +00:00
return (res == LFS_ERR_NOENT) ? 0 : lfs_tagsize(res);
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
}
int lfs_setattr(lfs_t *lfs, const char *path,
uint8_t type, const void *buffer, lfs_size_t size) {
if (size > lfs->attr_size) {
return LFS_ERR_NOSPC;
}
lfs_mdir_t cwd;
int32_t res = lfs_dir_lookup(lfs, &cwd, &path);
if (res < 0) {
return res;
}
return lfs_dir_commit(lfs, &cwd,
LFS_MKATTR(0x100 | type, lfs_tagid(res), buffer, size,
NULL));
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
/// Filesystem operations ///
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
static inline void lfs_superblockfromle32(lfs_superblock_t *superblock) {
superblock->version = lfs_fromle32(superblock->version);
superblock->block_size = lfs_fromle32(superblock->block_size);
superblock->block_count = lfs_fromle32(superblock->block_count);
superblock->inline_size = lfs_fromle32(superblock->inline_size);
superblock->attr_size = lfs_fromle32(superblock->attr_size);
superblock->name_size = lfs_fromle32(superblock->name_size);
}
static inline void lfs_superblocktole32(lfs_superblock_t *superblock) {
superblock->version = lfs_tole32(superblock->version);
superblock->block_size = lfs_tole32(superblock->block_size);
superblock->block_count = lfs_tole32(superblock->block_count);
superblock->inline_size = lfs_tole32(superblock->inline_size);
superblock->attr_size = lfs_tole32(superblock->attr_size);
superblock->name_size = lfs_tole32(superblock->name_size);
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
lfs->cfg = cfg;
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
// check that block size is a multiple of cache size is a multiple
// of prog and read sizes
LFS_ASSERT(lfs->cfg->cache_size % lfs->cfg->read_size == 0);
LFS_ASSERT(lfs->cfg->cache_size % lfs->cfg->prog_size == 0);
LFS_ASSERT(lfs->cfg->block_size % lfs->cfg->cache_size == 0);
// check that the block size is large enough to fit ctz pointers
LFS_ASSERT(4*lfs_npw2(0xffffffff / (lfs->cfg->block_size-2*4))
<= lfs->cfg->block_size);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup read cache
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
lfs->rcache.block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->read_buffer) {
lfs->rcache.buffer = lfs->cfg->read_buffer;
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->rcache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->rcache.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup program cache
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
lfs->pcache.block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->prog_buffer) {
lfs->pcache.buffer = lfs->cfg->prog_buffer;
} else {
Introduced cache_size as alternative to hardware read/write sizes The introduction of an explicit cache_size configuration allows customization of the cache buffers independently from the hardware read/write sizes. This has been one of littlefs's main handicaps. Without a distinction between cache units and hardware limitations, littlefs isn't able to read or program _less_ than the cache size. This leads to the counter-intuitive case where larger cache sizes can actually be harmful, since larger read/prog sizes require sending more data over the bus if we're only accessing a small set of data (for example the CTZ skip-list traversal). This is compounded with metadata logging, since a large program size limits the number of commits we can write out in a single metadata block. It really doesn't make sense to link program size + cache size here. With a separate cache_size configuration, we can be much smarter about what we actually read/write from disk. This also simplifies cache handling a bit. Before there were two possible cache sizes, but these were rarely used. Note that the cache_size is NOT written to the superblock and can be freely changed without breaking backwards compatibility.
2018-08-04 19:48:27 +00:00
lfs->pcache.buffer = lfs_malloc(lfs->cfg->cache_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->pcache.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
// setup lookahead, must be multiple of 32-bits
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(lfs->cfg->lookahead % 32 == 0);
LFS_ASSERT(lfs->cfg->lookahead > 0);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
if (lfs->cfg->lookahead_buffer) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer = lfs->cfg->lookahead_buffer;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->free.buffer = lfs_malloc(lfs->cfg->lookahead/8);
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!lfs->free.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that the size limits are sane
LFS_ASSERT(lfs->cfg->inline_size <= LFS_INLINE_MAX);
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
LFS_ASSERT(lfs->cfg->inline_size <= lfs->cfg->cache_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
lfs->inline_size = lfs->cfg->inline_size;
if (!lfs->inline_size) {
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
lfs->inline_size = lfs_min(LFS_INLINE_MAX, lfs->cfg->cache_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
LFS_ASSERT(lfs->cfg->attr_size <= LFS_ATTR_MAX);
lfs->attr_size = lfs->cfg->attr_size;
if (!lfs->attr_size) {
lfs->attr_size = LFS_ATTR_MAX;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
LFS_ASSERT(lfs->cfg->name_size <= LFS_NAME_MAX);
lfs->name_size = lfs->cfg->name_size;
if (!lfs->name_size) {
lfs->name_size = LFS_NAME_MAX;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// setup default state
lfs->root[0] = 0xffffffff;
lfs->root[1] = 0xffffffff;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
lfs->mlist = NULL;
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs->globals.s.movepair[0] = 0xffffffff;
lfs->globals.s.movepair[1] = 0xffffffff;
lfs->globals.s.moveid = 0x3ff;
lfs->globals.s.deorphaned = true;
Removed redundant lfs_scan in lfs_init Interestingly enough, lfs_scan is only needed during mount, as the state of the filesystem is well know during format.
2018-08-01 06:05:04 +00:00
lfs_globalzero(&lfs->locals);
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
static int lfs_deinit(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// free allocated memory
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->cfg->read_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->rcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
if (!lfs->cfg->prog_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->pcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
if (!lfs->cfg->lookahead_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->free.buffer);
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
int lfs_format(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// create free lookahead
Fixed issue with aggressively rounding down lookahead configuration The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
2017-11-10 01:10:08 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead/8);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
lfs->free.off = 0;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.size = lfs_min(lfs->cfg->lookahead, lfs->cfg->block_count);
lfs->free.i = 0;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs_alloc_ack(lfs);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// create superblock dir
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = lfs_dir_alloc(lfs, &dir);
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// write root directory
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t root;
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
err = lfs_dir_alloc(lfs, &root);
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
err = lfs_dir_commit(lfs, &root, NULL);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
lfs->root[0] = root.pair[0];
lfs->root[1] = root.pair[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir.tail[0] = lfs->root[0];
dir.tail[1] = lfs->root[1];
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// write one superblock
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_superblock_t superblock = {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
.magic = {"littlefs"},
.version = LFS_DISK_VERSION,
.block_size = lfs->cfg->block_size,
.block_count = lfs->cfg->block_count,
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
.attr_size = lfs->attr_size,
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
.name_size = lfs->name_size,
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
.inline_size = lfs->inline_size,
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
};
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_superblocktole32(&superblock);
lfs_pairtole32(lfs->root);
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
err = lfs_dir_commit(lfs, &dir,
LFS_MKATTR(LFS_TYPE_SUPERBLOCK, 0, &superblock, sizeof(superblock),
LFS_MKATTR(LFS_TYPE_DIRSTRUCT, 0, lfs->root, sizeof(lfs->root),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
NULL)));
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(lfs->root);
lfs_superblockfromle32(&superblock);
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
if (err) {
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
// sanity check that fetch works
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
return lfs_deinit(lfs);
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup free lookahead
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs->free.off = 0;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.size = 0;
lfs->free.i = 0;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs_alloc_ack(lfs);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// load superblock
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs_mdir_t superdir;
err = lfs_dir_fetch(lfs, &superdir, (const lfs_block_t[2]){0, 1});
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if (err) {
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_superblock_t superblock;
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
int32_t res = lfs_dir_get(lfs, &superdir, 0x7ffff000,
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
LFS_MKTAG(LFS_TYPE_SUPERBLOCK, 0, sizeof(superblock)),
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
&superblock);
if (res < 0) {
return res;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_superblockfromle32(&superblock);
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (memcmp(superblock.magic, "littlefs", 8) != 0) {
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
LFS_ERROR("Invalid superblock \"%.8s\"", superblock.magic);
return LFS_ERR_INVAL;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
uint16_t major_version = (0xffff & (superblock.version >> 16));
uint16_t minor_version = (0xffff & (superblock.version >> 0));
Add version info for software library and on-disk structures An annoying part of filesystems is that the software library can change independently of the on-disk structures. For this reason versioning is very important, and must be handled separately for the software and on-disk parts. In this patch, littlefs provides two version numbers at compile time, with major and minor parts, in the form of 6 macros. LFS_VERSION // Library version, uint32_t encoded LFS_VERSION_MAJOR // Major - Backwards incompatible changes LFS_VERSION_MINOR // Minor - Feature additions LFS_DISK_VERSION // On-disk version, uint32_t encoded LFS_DISK_VERSION_MAJOR // Major - Backwards incompatible changes LFS_DISK_VERSION_MINOR // Minor - Feature additions Note that littlefs will error if it finds a major version number that is different, or a minor version number that has regressed.
2018-01-26 20:26:25 +00:00
if ((major_version != LFS_DISK_VERSION_MAJOR ||
minor_version > LFS_DISK_VERSION_MINOR)) {
LFS_ERROR("Invalid version %d.%d", major_version, minor_version);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_INVAL;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
res = lfs_dir_get(lfs, &superdir, 0x7ffff000,
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 0, sizeof(lfs->root)),
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
&lfs->root);
if (res < 0) {
return res;
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(lfs->root);
Changed type info to be retrieved from name tag instead of struct tag Originally, I had type info encoded in the struct tag. This initially made sense because the type info only directly impacts the struct tag. However this was a case of focusing too much on the details instead of the bigger picture. A more file operations need to figure out the type of a file, but it's only actually a small number of file operations that need to interact with the file's structure. For the common case, providing the type of the file early shortens operations by a full tag access. Additionally, but storing the type in the file name tag, this opens up the struct tag to use those bits for storing more struct descriptions.
2018-07-09 17:51:31 +00:00
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
// check superblock configuration
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
if (superblock.attr_size) {
if (superblock.attr_size > lfs->attr_size) {
LFS_ERROR("Unsupported attr size (%d > %d)",
superblock.attr_size, lfs->attr_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return LFS_ERR_INVAL;
}
Added support for custom attributes leveraging the new metadata logging Now that littlefs has been rebuilt almost from the ground up with the intention to support custom attributes, adding in custom attribute support is relatively easy. The highest bit in the 9-bit type structure indicates that an attribute is a user-specified custom attribute. The user then has a full 8-bits to specify the attribute type. Other than that, custom attributes are treated the same as system-level attributes. Also made some tweaks to custom attributes: - Adopted the opencfg for file-level attributes provided by dpgeorge - Changed setattrs/getattrs to the simpler setattr/getattr functions users will probably be more familiar with. Note that multiple attributes can still be committed atomically with files, though not with directories. - Changed LFS_ATTRS_MAX -> LFS_ATTR_MAX since there's no longer a global limit on the sum of attribute sizes, which was rather confusing. Though they are still limited by what can fit in a metadata-pair.
2018-07-29 20:03:23 +00:00
lfs->attr_size = superblock.attr_size;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (superblock.name_size) {
if (superblock.name_size > lfs->name_size) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Unsupported name size (%d > %d)",
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
superblock.name_size, lfs->name_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return LFS_ERR_INVAL;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->name_size = superblock.name_size;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Cleaned up config options - Updated documentation where needed - Added asserts which take into account relationships with the new cache_size configuration - Restructured ordering to be consistent for the three main configurables: LFS_ATTR_MAX, LFS_NAME_MAX, and LFS_INLINE_MAX
2018-08-04 21:04:24 +00:00
if (superblock.inline_size) {
if (superblock.inline_size > lfs->inline_size) {
LFS_ERROR("Unsupported inline size (%d > %d)",
superblock.inline_size, lfs->inline_size);
return LFS_ERR_INVAL;
}
lfs->inline_size = superblock.inline_size;
}
Removed redundant lfs_scan in lfs_init Interestingly enough, lfs_scan is only needed during mount, as the state of the filesystem is well know during format.
2018-08-01 06:05:04 +00:00
// scan for any global updates
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs_mdir_t dir = {.tail = {0, 1}};
while (!lfs_pairisnull(dir.tail)) {
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
if (err) {
return err;
}
// xor together indirect deletes
lfs_globalxor(&lfs->locals, &dir.locals);
}
// update littlefs with globals
lfs_globalfromle32(&lfs->locals);
lfs_globalxor(&lfs->globals, &lfs->locals);
lfs_globalzero(&lfs->locals);
if (!lfs_pairisnull(lfs->globals.s.movepair)) {
LFS_DEBUG("Found move %d %d %d",
lfs->globals.s.movepair[0],
lfs->globals.s.movepair[1],
lfs->globals.s.moveid);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
int lfs_unmount(lfs_t *lfs) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return lfs_deinit(lfs);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
/// Filesystem filesystem operations ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_fs_traverse(lfs_t *lfs,
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int (*cb)(void *data, lfs_block_t block), void *data) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
// iterate over metadata pairs
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir = {.tail = {0, 1}};
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
while (!lfs_pairisnull(dir.tail)) {
for (int i = 0; i < 2; i++) {
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
int err = cb(data, dir.tail[i]);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
// iterate through ids in directory
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
for (uint16_t id = 0; id < dir.count; id++) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
struct lfs_ctz ctz;
int32_t tag = lfs_dir_get(lfs, &dir, 0x7c3ff000,
LFS_MKTAG(LFS_TYPE_STRUCT, id, sizeof(ctz)), &ctz);
if (tag < 0) {
if (tag == LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
continue;
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return tag;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_ctzfromle32(&ctz);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (lfs_tagtype(tag) == LFS_TYPE_CTZSTRUCT) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_ctztraverse(lfs, &lfs->rcache, NULL,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
ctz.head, ctz.size, cb, data);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
}
// iterate over any open files
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
for (lfs_file_t *f = (lfs_file_t*)lfs->mlist; f; f = f->next) {
if (f->type != LFS_TYPE_REG) {
continue;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if ((f->flags & LFS_F_DIRTY) && !(f->flags & LFS_F_INLINE)) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_ctztraverse(lfs, &lfs->rcache, &f->cache,
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
f->ctz.head, f->ctz.size, cb, data);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
if ((f->flags & LFS_F_WRITING) && !(f->flags & LFS_F_INLINE)) {
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
int err = lfs_ctztraverse(lfs, &lfs->rcache, &f->cache,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
f->block, f->pos, cb, data);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
return 0;
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_pred(lfs_t *lfs,
const lfs_block_t pair[2], lfs_mdir_t *pdir) {
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
if (lfs_pairisnull(lfs->root)) {
return LFS_ERR_NOENT;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// iterate over all directory directory entries
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
pdir->tail[0] = 0;
pdir->tail[1] = 1;
while (!lfs_pairisnull(pdir->tail)) {
if (lfs_paircmp(pdir->tail, pair) == 0) {
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return 0;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, pdir, pdir->tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
return LFS_ERR_NOENT;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int32_t lfs_fs_parent(lfs_t *lfs, const lfs_block_t pair[2],
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
lfs_mdir_t *parent) {
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
if (lfs_pairisnull(lfs->root)) {
return LFS_ERR_NOENT;
}
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
// search for both orderings so we can reuse the find function
lfs_block_t child[2] = {pair[0], pair[1]};
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairtole32(child);
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
for (int i = 0; i < 2; i++) {
// iterate over all directory directory entries
parent->tail[0] = 0;
parent->tail[1] = 1;
while (!lfs_pairisnull(parent->tail)) {
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
int32_t tag = lfs_dir_find(lfs, parent, parent->tail, 0x7fc00fff,
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
LFS_MKTAG(LFS_TYPE_DIRSTRUCT, 0, sizeof(child)),
Changed internal functions to return tags over pointers One neat (if gimmicky) trick, is that each tag has a valid bit in the highest bit position of the 32-bit word. This is used to determine when to stop a fetch operation, but after fetch, the bit is free to use in the driver. This means we can create a typed-union of sorts with error codes and tags, returning both as the return value from a function. Say what you will about this trick, it does have a significant impact on code size. I suspect this is primarily due to the compiler having a hard time optimizing around pointer access.
2018-07-13 01:22:06 +00:00
child);
if (tag != LFS_ERR_NOENT) {
return tag;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
}
Consolidated find/parent scanning functions An interesting observation about the find and parent scanning functions is that at their core, they're both actually doing the same operation. They search a metadata-pair during fetch for an entry uses the entry data instead of the entry tag. This means we can combine these functions and get a decent code savings. It's a little bit trickier because pair ordering isn't guaranteed. But to work around that we can simply search for both pair orderings. It's a bit more expensive but may be worth the code savings. A fancier implementation in the future can avoid the 2x lfs_parent scans.
2018-07-09 22:29:40 +00:00
lfs_pairswap(child);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Modified results from find-like functions to use tags Tags offer all of the necessary info from the find functions (which makes sense, this is the structure that stores the info on disk). Passing around a single tag instead of separate id and type fields simplifies the internal functions while leverages the tag's compactness.
2018-07-11 12:18:30 +00:00
return LFS_ERR_NOENT;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_relocate(lfs_t *lfs,
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
const lfs_block_t oldpair[2], lfs_block_t newpair[2]) {
Squished in-flight files/dirs into single list This is an effort to try to consolidate the handling of in-flight files and dirs opened by the user (and possibly opened internally). Both files and dirs have metadata state that need to be kept in sync by the commit logic. This metadata state is mostly contained in the lfs_mdir_t type, which is present in both the lfs_file_t and lfs_dir_t. Unfortunately both of these structs have some relatively unrelated metadata that needs to be kept in sync: - Files store an id representing the open file - Dirs store an id during iteration While these take up the same space, they unfortunately need to be managed differently by the commit logic. The best solution I can come up with is to simple store a general purpose list and tag both structures with LFS_TYPE_REG and LFS_TYPE_DIR respectively. This is kinda funky, but wins out over duplicated the commit logic.
2018-08-01 15:24:59 +00:00
// update internal root
if (lfs_paircmp(oldpair, lfs->root) == 0) {
LFS_DEBUG("Relocating root %d %d", newpair[0], newpair[1]);
lfs->root[0] = newpair[0];
lfs->root[1] = newpair[1];
}
// update internally tracked dirs
for (lfs_mlist_t *d = lfs->mlist; d; d = d->next) {
if (lfs_paircmp(oldpair, d->m.pair) == 0) {
d->m.pair[0] = newpair[0];
d->m.pair[1] = newpair[1];
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// find parent
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t parent;
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int32_t tag = lfs_fs_parent(lfs, oldpair, &parent);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag < 0 && tag != LFS_ERR_NOENT) {
return tag;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (tag != LFS_ERR_NOENT) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// update disk, this creates a desync
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairtole32(newpair);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
int err = lfs_dir_commit(lfs, &parent,
&(lfs_mattr_t){.tag=tag, .buffer=newpair});
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(newpair);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// clean up bad block, which should now be a desync
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
return lfs_fs_forceconsistency(lfs);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
// find pred
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int err = lfs_fs_pred(lfs, oldpair, &parent);
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Cleaned up commit logic and function organization Restrctured function organization to make a bit more sense, and made some small refactoring tweaks, specifically around the commit logic and global related functions.
2018-07-13 20:04:31 +00:00
// if we can't find dir, it must be new
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
if (err != LFS_ERR_NOENT) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// just replace bad pair, no desync can occur
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
parent.tail[0] = newpair[0];
parent.tail[1] = newpair[1];
Renamed tag functions and macros - lfs_tagverb -> lfs_tag_verb - lfs_mktag -> LFS_MKTAG (it's a macro now) - LFS_STRUCT_THING -> LFS_THINGSTRUCT
2018-07-13 00:07:56 +00:00
int err = lfs_dir_commit(lfs, &parent,
LFS_MKATTR(LFS_TYPE_TAIL + parent.split, 0x3ff,
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
parent.tail, sizeof(parent.tail),
Cleaned up attributes and related logic The biggest change here is to make littlefs less obsessed with the lfs_mattr_t struct. It was limiting our flexibility and can be entirely replaced by passing the tag + data explicitly. The remaining use of lfs_mattr_t is specific to the commit logic, where it replaces the lfs_mattrlist_t struct. Other changes: - Added global lfs_diskoff struct for embedding disk references inside the lfs_mattr_t. - Reordered lfs_mattrlist_t to squeeze out some code savings - Added commit_get for explicit access to entries from unfinished metadata-pairs - Parameterized the "stop_at_commit" flag instead of hackily storing it in the lfs_mdir_t temporarily - Changed return value of lfs_pred to error-only with ENOENT representing a missing predecessor - Adopted const where possible
2018-07-13 01:43:55 +00:00
NULL));
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (err) {
return err;
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return 0;
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
static int lfs_fs_forceconsistency(lfs_t *lfs) {
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
if (!lfs->globals.s.deorphaned) {
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
LFS_DEBUG("Found orphans %d",
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs->globals.s.deorphaned);
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// Fix any orphans
lfs_mdir_t pdir = {.split = true};
lfs_mdir_t dir = {.tail = {0, 1}};
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// iterate over all directory directory entries
while (!lfs_pairisnull(dir.tail)) {
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
if (err) {
return err;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// check head blocks for orphans
if (!pdir.split) {
// check if we have a parent
lfs_mdir_t parent;
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
int32_t tag = lfs_fs_parent(lfs, pdir.tail, &parent);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (tag < 0 && tag != LFS_ERR_NOENT) {
return tag;
}
Fixed move handling when caught in a relocate This was a surprisingly tricky issue. One of the subtle requirements for the new move handling to work is that the block containing the move does not change until the move is resolved. Initially, this seemed easy to implement, given that a move is always immediately followed by its resolution. However, the extra metadata-pair operations needed to maintain integrity present a challenge. At any commit, a directory block may end up moved as a side effect of relocation due to a bad block. The fix here is to move the move resolution directly into the commit logic. This means that any commit to a block containing a move will be implicitly resolved, leaving the later attempt at move resolution as a noop. This fix required quite a bit of restructuring, but as a nice side-effect some of the complexity around moves actually went away. Additionally, the new move handling is surprisingly powerful at combining moves with nearby commits. And we now get same-metadata-pair renames for free! A win for procrasination on that minor feature.
2018-07-17 23:31:30 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (tag == LFS_ERR_NOENT) {
// we are an orphan
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
LFS_DEBUG("Fixing orphan %d %d",
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
pdir.tail[0], pdir.tail[1]);
pdir.tail[0] = dir.tail[0];
pdir.tail[1] = dir.tail[1];
err = lfs_dir_commit(lfs, &pdir,
LFS_MKATTR(LFS_TYPE_SOFTTAIL, 0x3ff,
pdir.tail, sizeof(pdir.tail),
NULL));
if (err) {
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
break;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
lfs_block_t pair[2];
int32_t res = lfs_dir_get(lfs, &parent, 0x7ffff000, tag, pair);
if (res < 0) {
return res;
}
Added back big-endian support again on the new metadata structures The only interesting thing to note is that we now have to also support le16 due to storing the id outside of tags in the globals structure.
2018-08-01 23:10:24 +00:00
lfs_pairfromle32(pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (!lfs_pairsync(pair, pdir.tail)) {
// we have desynced
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
LFS_DEBUG("Fixing half-orphan %d %d", pair[0], pair[1]);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
pdir.tail[0] = pair[0];
pdir.tail[1] = pair[1];
err = lfs_dir_commit(lfs, &pdir,
LFS_MKATTR(LFS_TYPE_SOFTTAIL, 0x3ff,
pdir.tail, sizeof(pdir.tail),
NULL));
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
break;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
memcpy(&pdir, &dir, sizeof(pdir));
}
Modified results from find-like functions to use tags Tags offer all of the necessary info from the find functions (which makes sense, this is the structure that stores the info on disk). Passing around a single tag instead of separate id and type fields simplifies the internal functions while leverages the tag's compactness.
2018-07-11 12:18:30 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// mark orphan as fixed
lfs_globaldeorphaned(lfs, false);
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
if (lfs->globals.s.moveid != 0x3ff) {
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// Fix bad moves
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
LFS_DEBUG("Fixing move %d %d %d",
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
lfs->globals.s.movepair[0],
lfs->globals.s.movepair[1],
lfs->globals.s.moveid);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// fetch and delete the moved entry
lfs_mdir_t movedir;
Simplified the internal xored-globals implementation There wasn't much use (and inconsistent compiler support) for storing small values next to the unaligned lfs_global_t struct. So instead, I've rounded the struct up to the nearest word to try to take advantage of the alignment in xor and memset operations. I've also moved the global fetching into lfs_mount, since that was the only use of the operation. This allows for some variable reuse in the mount function.
2018-08-04 00:01:27 +00:00
int err = lfs_dir_fetch(lfs, &movedir, lfs->globals.s.movepair);
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
if (err) {
return err;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Added orphan bit to xored-globals Unfortunately for us, even with the new ability to store global state, orphans can not be handled as gracefully as moves. This is due to the fact that directory operations can create an unbounded number of orphans. It's usually small, the fact that it's unbounded means we can't store the orphan info in xored-globals. However, one thing we can do to leverage the xored-global state is store a bit indicating if _any_ orphans are present. This means in the common case we can completely avoid the deorphan step, while only using a single bit of the global state, which is effectively free since we can store it in the globals tag itself. If a littlefs drive does not want to consider the orphan bit, it's free to use the previous behaviour of always checking for orphans on first write.
2018-07-31 13:07:36 +00:00
// rely on cancel logic inside commit
err = lfs_dir_commit(lfs, &movedir, NULL);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
return 0;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
lfs_ssize_t lfs_fs_getattr(lfs_t *lfs,
uint8_t type, void *buffer, lfs_size_t size) {
lfs_mdir_t superdir;
int err = lfs_dir_fetch(lfs, &superdir, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
int32_t res = lfs_dir_get(lfs, &superdir, 0x7ffff000,
LFS_MKTAG(0x100 | type, 0,
lfs_min(size, lfs->attr_size)), buffer);
if (res < 0) {
return res;
}
Updated custom attribute documentation and tweaked nonexistant attributes Because of limitations in how littlefs manages attributes on disk, littlefs views zero-length attributes and missing attributes as the same thing. The simpliest implementation of attributes mirrors this behaviour transparently for the user.
2018-08-05 00:23:49 +00:00
return (res == LFS_ERR_NOENT) ? 0 : lfs_tagsize(res);
Cleaned up several TODOs Other than removed outdated TODOs, there are several tweaks: - Standardized naming of fs-level functions (mostly internal names) - Tweaked low-level use of subtype to hopefully take advantage of redundant code removal - Moved root-handling into lfs_dir_getinfo - Updated DEBUG statements around move/orphan fixes - Removed trailing 1s in type fields - Removed unused code
2018-08-01 10:52:48 +00:00
}
int lfs_fs_setattr(lfs_t *lfs,
uint8_t type, const void *buffer, lfs_size_t size) {
if (size > lfs->attr_size) {
return LFS_ERR_NOSPC;
}
lfs_mdir_t superdir;
int err = lfs_dir_fetch(lfs, &superdir, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
return lfs_dir_commit(lfs, &superdir,
LFS_MKATTR(0x100 | type, 0, buffer, size,
NULL));
}
Removed the implicit lfs_t parameter to lfs_traverse This is a very minor thing but it has been bugging me. On one hand, all a callback ever needs is a single pointer for context. On the other hand, you could make the argument that in the context of littlefs, the lfs_t struct represents global state and should always be available to callbacks passed to littlefs. In the end I'm sticking with only a single context pointer, since this is satisfies the minimum requirements and has the highest chance of function reuse. If a user needs access to the lfs_t struct, it can be passed by reference in the context provided to the callback. This also matches callbacks used in other languages with more emphasis on objects and classes. Usually the callback doesn't get a reference to the caller.
2018-07-30 19:40:27 +00:00
static int lfs_fs_size_count(void *p, lfs_block_t block) {
Fixed the orphan test to handle logging metadata-pairs The main issue here was that the old orphan test relied on deleting the block that contained the most recent update. In the new design this doesn't really work since updates get appended to metadata-pairs incrementally. This is fixed by instead using the truncate command on the appropriate block. We're now passing orphan tests.
2018-07-30 14:10:04 +00:00
lfs_size_t *size = p;
*size += 1;
return 0;
}
lfs_ssize_t lfs_fs_size(lfs_t *lfs) {
lfs_size_t size = 0;
int err = lfs_fs_traverse(lfs, lfs_fs_size_count, &size);
if (err) {
return err;
}
return size;
}
Reference in New Issue Copy Permalink