openharmony/third_party_littlefs
1
0
Fork 0
mirror of https://gitee.com/openharmony/third_party_littlefs synced 2024-11-23 14:59:50 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
2ff32d2dfb
third_party_littlefs/lfs.c

4128 lines
116 KiB
C
Raw Normal View History

Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
/*
* The little filesystem
*
Updated copyright Due to employee contract Per ARM license remains under Apache 2.0
2017-10-13 01:27:33 +00:00
* Copyright (c) 2017 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
*/
#include "lfs.h"
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
#include "lfs_util.h"
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// Caching block device operations ///
static int lfs_cache_read(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
lfs_off_t off, void *buffer, lfs_size_t size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (pcache && block == pcache->block && off >= pcache->off &&
off < pcache->off + lfs->cfg->prog_size) {
// is already in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs->cfg->prog_size - (off-pcache->off));
memcpy(data, &pcache->buffer[off-pcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
if (block == rcache->block && off >= rcache->off &&
off < rcache->off + lfs->cfg->read_size) {
// is already in rcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs->cfg->read_size - (off-rcache->off));
memcpy(data, &rcache->buffer[off-rcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (off % lfs->cfg->read_size == 0 && size >= lfs->cfg->read_size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
// bypass cache?
lfs_size_t diff = size - (size % lfs->cfg->read_size);
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
int err = lfs->cfg->read(lfs->cfg, block, off, data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
data += diff;
off += diff;
size -= diff;
continue;
}
// load to cache, first condition can no longer fail
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
rcache->block = block;
rcache->off = off - (off % lfs->cfg->read_size);
int err = lfs->cfg->read(lfs->cfg, rcache->block,
rcache->off, rcache->buffer, lfs->cfg->read_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_cache_cmp(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
lfs_off_t off, const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
int err = lfs_cache_read(lfs, rcache, pcache,
block, off+i, &c, 1);
if (err) {
return err;
}
if (c != data[i]) {
return false;
}
}
return true;
}
static int lfs_cache_crc(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
int err = lfs_cache_read(lfs, rcache, pcache,
block, off+i, &c, 1);
if (err) {
return err;
}
lfs_crc(crc, &c, 1);
}
return 0;
}
static int lfs_cache_flush(lfs_t *lfs,
lfs_cache_t *pcache, lfs_cache_t *rcache) {
if (pcache->block != 0xffffffff) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(pcache->block < lfs->cfg->block_count);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs->cfg->prog(lfs->cfg, pcache->block,
pcache->off, pcache->buffer, lfs->cfg->prog_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (rcache) {
int res = lfs_cache_cmp(lfs, rcache, NULL, pcache->block,
pcache->off, pcache->buffer, lfs->cfg->prog_size);
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_CORRUPT;
}
}
pcache->block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
return 0;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_cache_prog(lfs_t *lfs, lfs_cache_t *pcache,
lfs_cache_t *rcache, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
LFS_ASSERT(off + size <= lfs->cfg->block_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (block == pcache->block && off >= pcache->off &&
off < pcache->off + lfs->cfg->prog_size) {
// is already in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
lfs->cfg->prog_size - (off-pcache->off));
memcpy(&pcache->buffer[off-pcache->off], data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (off % lfs->cfg->prog_size == 0) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// eagerly flush out pcache if we fill up
int err = lfs_cache_flush(lfs, pcache, rcache);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// pcache must have been flushed, either by programming and
// entire block or manually flushing the pcache
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(pcache->block == 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (off % lfs->cfg->prog_size == 0 &&
size >= lfs->cfg->prog_size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// bypass pcache?
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = size - (size % lfs->cfg->prog_size);
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
int err = lfs->cfg->prog(lfs->cfg, block, off, data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (rcache) {
int res = lfs_cache_cmp(lfs, rcache, NULL,
block, off, data, diff);
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_CORRUPT;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// prepare pcache, first condition can no longer fail
pcache->block = block;
pcache->off = off - (off % lfs->cfg->prog_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// General lfs block device operations ///
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_read(lfs_t *lfs, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, void *buffer, lfs_size_t size) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return lfs_cache_read(lfs, &lfs->rcache, &lfs->pcache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
block, off, buffer, size);
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_prog(lfs_t *lfs, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, const void *buffer, lfs_size_t size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
return lfs_cache_prog(lfs, &lfs->pcache, NULL,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
block, off, buffer, size);
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_cmp(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, const void *buffer, lfs_size_t size) {
return lfs_cache_cmp(lfs, &lfs->rcache, NULL, block, off, buffer, size);
}
static int lfs_bd_crc(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
return lfs_cache_crc(lfs, &lfs->rcache, NULL, block, off, size, crc);
}
static int lfs_bd_erase(lfs_t *lfs, lfs_block_t block) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->erase(lfs->cfg, block);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_sync(lfs_t *lfs) {
Fixed dirty rcache during directory commit An interesting side-effect of adding internal checks to the littlefs for block errors, is that the littlefs starts to cover up its own flaws. Probably out of embarrassment. In this case, the relocation logic for directories left the littlefs rcache dirty with invalid data. The littlefs detected the error, treated it as a corrupted write, and just moved the "corrupted" block to a new block, which as a side-effect flushes the rcache. Since committing a dir will end up flushing the rcache to check for errors anyways, we can just drop the rcache in lfs_bd_sync.
2017-06-25 21:21:14 +00:00
lfs->rcache.block = 0xffffffff;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs_cache_flush(lfs, &lfs->pcache, NULL);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->sync(lfs->cfg);
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
/// Internal operations predeclared here ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_fs_traverse(lfs_t *lfs,
int (*cb)(lfs_t*, void*, lfs_block_t), void *data);
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_pred(lfs_t *lfs, const lfs_block_t dir[2], lfs_mdir_t *pdir);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_parent(lfs_t *lfs, const lfs_block_t dir[2],
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *parent, lfs_mattr_t *attr);
static int lfs_moved(lfs_t *lfs, lfs_mdir_t *fromdir, uint16_t fromid);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_relocate(lfs_t *lfs,
const lfs_block_t oldpair[2], const lfs_block_t newpair[2]);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
int lfs_scan(lfs_t *lfs);
int lfs_fixmove(lfs_t *lfs);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
int lfs_deorphan(lfs_t *lfs);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
/// Block allocator ///
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static int lfs_alloc_lookahead(lfs_t *lfs, void *p, lfs_block_t block) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs_block_t off = ((block - lfs->free.off)
Fixed issue with negative modulo with unaligned lookaheads When the lookahead buffer wraps around in an unaligned filesystem, it's possible for blocks at the beginning of the disk to have a negative distance from the lookahead, but still reside in the lookahead buffer. Switching to signed modulo doesn't quite work due to how negative modulo is implemented in C, so the simple solution is to shift the region to be positive.
2017-09-17 19:52:25 +00:00
+ lfs->cfg->block_count) % lfs->cfg->block_count;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
if (off < lfs->free.size) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer[off / 32] |= 1U << (off % 32);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
return 0;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) {
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
while (true) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
while (lfs->free.i != lfs->free.size) {
lfs_block_t off = lfs->free.i;
lfs->free.i += 1;
lfs->free.ack -= 1;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!(lfs->free.buffer[off / 32] & (1U << (off % 32)))) {
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// found a free block
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
*block = (lfs->free.off + off) % lfs->cfg->block_count;
Fixed issue with lookahead trusting old lookahead blocks One of the big simplifications in littlefs's implementation is the complete lack of tracking free blocks, allowing operations to simply drop blocks that are no longer in use. However, this means the lookahead buffer can easily contain outdated blocks that were previously deleted. This is usually fine, as littlefs will rescan the storage if it can't find a free block in the lookahead buffer, but after changes that caused littlefs to more conservatively respect the alloc acks (e611cf5), any scanned blocks after an ack would be incorrectly trusted. The fix is to eagerly scan ahead in the lookahead when we allocate so that alloc acks are better able to discredit old lookahead blocks. Since usually alloc acks are tightly coupled to allocations of one or two blocks, this allows littlefs to properly rescan every set of allocations. This may still be a concern if there is a long series of worn out blocks, but in the worst case littlefs will conservatively avoid using blocks it's not sure about. Found by davidefer
2018-04-06 22:00:29 +00:00
// eagerly find next off so an alloc ack can
// discredit old lookahead blocks
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
while (lfs->free.i != lfs->free.size &&
(lfs->free.buffer[lfs->free.i / 32]
& (1U << (lfs->free.i % 32)))) {
lfs->free.i += 1;
lfs->free.ack -= 1;
Fixed issue with lookahead trusting old lookahead blocks One of the big simplifications in littlefs's implementation is the complete lack of tracking free blocks, allowing operations to simply drop blocks that are no longer in use. However, this means the lookahead buffer can easily contain outdated blocks that were previously deleted. This is usually fine, as littlefs will rescan the storage if it can't find a free block in the lookahead buffer, but after changes that caused littlefs to more conservatively respect the alloc acks (e611cf5), any scanned blocks after an ack would be incorrectly trusted. The fix is to eagerly scan ahead in the lookahead when we allocate so that alloc acks are better able to discredit old lookahead blocks. Since usually alloc acks are tightly coupled to allocations of one or two blocks, this allows littlefs to properly rescan every set of allocations. This may still be a concern if there is a long series of worn out blocks, but in the worst case littlefs will conservatively avoid using blocks it's not sure about. Found by davidefer
2018-04-06 22:00:29 +00:00
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
// check if we have looked at all blocks since last ack
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
if (lfs->free.ack == 0) {
LFS_WARN("No more free space %d", lfs->free.i + lfs->free.off);
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
return LFS_ERR_NOSPC;
}
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.off = (lfs->free.off + lfs->free.size)
% lfs->cfg->block_count;
lfs->free.size = lfs_min(lfs->cfg->lookahead, lfs->free.ack);
lfs->free.i = 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// find mask of free blocks from tree
Fixed issue with aggressively rounding down lookahead configuration The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
2017-11-10 01:10:08 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead/8);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_fs_traverse(lfs, lfs_alloc_lookahead, NULL);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
if (err) {
return err;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static void lfs_alloc_ack(lfs_t *lfs) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.ack = lfs->cfg->block_count;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
/// Endian swapping functions ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//static void lfs_dir_fromle32(struct lfs_disk_dir *d) {
// d->rev = lfs_fromle32(d->rev);
// d->size = lfs_fromle32(d->size);
// d->tail[0] = lfs_fromle32(d->tail[0]);
// d->tail[1] = lfs_fromle32(d->tail[1]);
//}
//
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
//static void lfs_mdir_tole32(struct lfs_disk_dir *d) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// d->rev = lfs_tole32(d->rev);
// d->size = lfs_tole32(d->size);
// d->tail[0] = lfs_tole32(d->tail[0]);
// d->tail[1] = lfs_tole32(d->tail[1]);
//}
//
//static void lfs_entry_fromle32(struct lfs_disk_entry *d) {
// d->u.dir[0] = lfs_fromle32(d->u.dir[0]);
// d->u.dir[1] = lfs_fromle32(d->u.dir[1]);
//}
//
//static void lfs_entry_tole32(struct lfs_disk_entry *d) {
// d->u.dir[0] = lfs_tole32(d->u.dir[0]);
// d->u.dir[1] = lfs_tole32(d->u.dir[1]);
//}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
///*static*/ void lfs_superblock_fromle32(struct lfs_disk_superblock *d) {
// d->root[0] = lfs_fromle32(d->root[0]);
// d->root[1] = lfs_fromle32(d->root[1]);
// d->block_size = lfs_fromle32(d->block_size);
// d->block_count = lfs_fromle32(d->block_count);
// d->version = lfs_fromle32(d->version);
// d->inline_size = lfs_fromle32(d->inline_size);
// d->attrs_size = lfs_fromle32(d->attrs_size);
// d->name_size = lfs_fromle32(d->name_size);
//}
//
///*static*/ void lfs_superblock_tole32(struct lfs_disk_superblock *d) {
// d->root[0] = lfs_tole32(d->root[0]);
// d->root[1] = lfs_tole32(d->root[1]);
// d->block_size = lfs_tole32(d->block_size);
// d->block_count = lfs_tole32(d->block_count);
// d->version = lfs_tole32(d->version);
// d->inline_size = lfs_tole32(d->inline_size);
// d->attrs_size = lfs_tole32(d->attrs_size);
// d->name_size = lfs_tole32(d->name_size);
//}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
/// Other struct functions ///
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
//static inline lfs_size_t lfs_entry_elen(const lfs_mattr_t *attr) {
// return (lfs_size_t)(attr->d.elen) |
// ((lfs_size_t)(attr->d.alen & 0xc0) << 2);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//}
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
//static inline lfs_size_t lfs_entry_alen(const lfs_mattr_t *attr) {
// return attr->d.alen & 0x3f;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//}
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
//static inline lfs_size_t lfs_entry_nlen(const lfs_mattr_t *attr) {
// return attr->d.nlen;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//}
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
//static inline lfs_size_t lfs_entry_size(const lfs_mattr_t *attr) {
// return 4 + lfs_entry_elen(attr) +
// lfs_entry_alen(attr) +
// lfs_entry_nlen(attr);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Metadata pair and directory operations ///
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static inline void lfs_pairswap(lfs_block_t pair[2]) {
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
lfs_block_t t = pair[0];
pair[0] = pair[1];
pair[1] = t;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
static inline bool lfs_pairisnull(const lfs_block_t pair[2]) {
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
return pair[0] == 0xffffffff || pair[1] == 0xffffffff;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static inline int lfs_paircmp(
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
return !(paira[0] == pairb[0] || paira[1] == pairb[1] ||
paira[0] == pairb[1] || paira[1] == pairb[0]);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static inline bool lfs_pairsync(
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
return (paira[0] == pairb[0] && paira[1] == pairb[1]) ||
(paira[0] == pairb[1] && paira[1] == pairb[0]);
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
/// Entry tag operations ///
static inline lfs_tag_t lfs_mktag(
uint16_t type, uint16_t id, lfs_size_t size) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return (type << 22) | (id << 12) | size;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static inline bool lfs_tag_valid(lfs_tag_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return !(tag & 0x80000000);
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static inline uint16_t lfs_tag_type(lfs_tag_t tag) {
return (tag & 0x7fc00000) >> 22;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
static inline uint16_t lfs_tag_scope(lfs_tag_t tag) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return (tag & 0x70000000) >> 22;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
static inline uint16_t lfs_tag_subtype(lfs_tag_t tag) {
return (tag & 0x7e000000) >> 22;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
static inline uint16_t lfs_tag_struct(lfs_tag_t tag) {
return (tag & 0x71c00000) >> 22;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
static inline uint16_t lfs_tag_id(lfs_tag_t tag) {
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
return (tag & 0x003ff000) >> 12;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static inline lfs_size_t lfs_tag_size(lfs_tag_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return tag & 0x00000fff;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// operations on globals
static lfs_globals_t lfs_globals_xor(
const lfs_globals_t *a, const lfs_globals_t *b) {
lfs_globals_t res;
res.move.pair[0] = a->move.pair[0] ^ b->move.pair[0];
res.move.pair[1] = a->move.pair[1] ^ b->move.pair[1];
res.move.id = a->move.id ^ b->move.id;
return res;
}
static bool lfs_globals_iszero(const lfs_globals_t *a) {
return (a->move.pair[0] == 0 && a->move.pair[1] == 0 && a->move.id == 0);
}
// commit logic
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
struct lfs_commit {
lfs_block_t block;
lfs_off_t off;
lfs_off_t begin;
lfs_off_t end;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t ptag;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint32_t crc;
struct {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
uint16_t begin;
uint16_t end;
} filter;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
};
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// TODO predelcare
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
static int lfs_commit_move(lfs_t *lfs, struct lfs_commit *commit,
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
uint16_t fromid, uint16_t toid,
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *dir, lfs_mattrlist_t *list);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
static int lfs_commit_commit(lfs_t *lfs,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
struct lfs_commit *commit, lfs_mattr_t attr) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// filter out ids
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_scope(attr.tag) <= LFS_SCOPE_ENTRY && (
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_tag_id(attr.tag) < commit->filter.begin ||
lfs_tag_id(attr.tag) >= commit->filter.end)) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
return 0;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// special cases
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_type(attr.tag) == LFS_FROM_MOVE) {
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
return lfs_commit_move(lfs, commit,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_tag_size(attr.tag), lfs_tag_id(attr.tag),
attr.u.dir, NULL);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
uint16_t id = lfs_tag_id(attr.tag) - commit->filter.begin;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
attr.tag = lfs_mktag(0, id, 0) | (attr.tag & 0xffc00fff);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// check if we fit
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_size_t size = lfs_tag_size(attr.tag);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (commit->off + sizeof(lfs_tag_t)+size > commit->end) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return LFS_ERR_NOSPC;
}
// write out tag
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// TODO rm me
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
//printf("tag w %#010x (%x:%x %03x %03x %03x)\n", attr.tag, commit->block, commit->off+sizeof(lfs_tag_t), lfs_tag_type(attr.tag), lfs_tag_id(attr.tag), lfs_tag_size(attr.tag));
lfs_tag_t tag = lfs_tole32((attr.tag & 0x7fffffff) ^ commit->ptag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
lfs_crc(&commit->crc, &tag, sizeof(tag));
int err = lfs_bd_prog(lfs, commit->block, commit->off, &tag, sizeof(tag));
if (err) {
return err;
}
commit->off += sizeof(tag);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (!(attr.tag & 0x80000000)) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// from memory
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_crc(&commit->crc, attr.u.buffer, size);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
err = lfs_bd_prog(lfs, commit->block, commit->off,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
attr.u.buffer, size);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
} else {
// from disk
for (lfs_off_t i = 0; i < size; i++) {
uint8_t dat;
int err = lfs_bd_read(lfs,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
attr.u.d.block, attr.u.d.off+i, &dat, 1);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
lfs_crc(&commit->crc, &dat, 1);
err = lfs_bd_prog(lfs, commit->block, commit->off+i, &dat, 1);
if (err) {
return err;
}
}
}
commit->off += size;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
commit->ptag = attr.tag & 0x7fffffff; // TODO do this once
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return 0;
}
static int lfs_commit_crc(lfs_t *lfs, struct lfs_commit *commit) {
// align to program units
lfs_off_t noff = lfs_alignup(
commit->off + 2*sizeof(uint32_t), lfs->cfg->prog_size);
// read erased state from next program unit
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t tag;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
int err = lfs_bd_read(lfs, commit->block, noff, &tag, sizeof(tag));
if (err) {
return err;
}
// build crc tag
tag = (0x80000000 & ~lfs_fromle32(tag)) |
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
lfs_mktag(LFS_TYPE_CRC, 0x3ff,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
noff - (commit->off+sizeof(uint32_t)));
// write out crc
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
//printf("tag w %#010x (%x:%x %03x %03x %03x)\n", tag, commit->block, commit->off+sizeof(tag), lfs_tag_type(tag), lfs_tag_id(tag), lfs_tag_size(tag));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint32_t footer[2];
footer[0] = lfs_tole32(tag ^ commit->ptag);
lfs_crc(&commit->crc, &footer[0], sizeof(footer[0]));
footer[1] = lfs_tole32(commit->crc);
err = lfs_bd_prog(lfs, commit->block, commit->off,
footer, sizeof(footer));
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
commit->off += sizeof(tag)+lfs_tag_size(tag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
commit->ptag = tag;
// flush buffers
err = lfs_bd_sync(lfs);
if (err) {
return err;
}
// successful commit, check checksum to make sure
uint32_t crc = 0xffffffff;
err = lfs_bd_crc(lfs, commit->block, commit->begin,
commit->off-lfs_tag_size(tag) - commit->begin, &crc);
if (err) {
return err;
}
if (crc != commit->crc) {
return LFS_ERR_CORRUPT;
}
return 0;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
static int lfs_commit_list(lfs_t *lfs, struct lfs_commit *commit,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattrlist_t *list) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
for (; list; list = list->next) {
int err = lfs_commit_commit(lfs, commit, list->e);
if (err) {
return err;
}
}
return 0;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// committer for moves
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// TODO rename?
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
struct lfs_commit_move {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *dir; // TODO need dir?
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
struct {
uint16_t from;
uint16_t to;
} id;
struct lfs_commit *commit;
};
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// TODO redeclare
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
static int lfs_dir_traverse(lfs_t *lfs, lfs_mdir_t *dir,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int (*cb)(lfs_t *lfs, void *data, lfs_mattr_t attr),
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
void *data);
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_get(lfs_t *lfs, lfs_mdir_t *dir,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
uint32_t mask, lfs_mattr_t *attr);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
static int lfs_commit_movescan(lfs_t *lfs, void *p, lfs_mattr_t attr) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
struct lfs_commit_move *move = p;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_struct(attr.tag) == LFS_STRUCT_DELETE &&
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_tag_id(attr.tag) <= move->id.from) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// something was deleted, we need to move around it
move->id.from += 1;
return 0;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_type(attr.tag) == LFS_STRUCT_MOVE) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// TODO need this?
// ignore moves
return 0;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
Fixed bug where globals were poisoning move commits The issue lies in the reuse of the id field for globals. Before globals, the only tags with a non-null (0x3ff) id field were names, structs, and other file-specific metadata. But globals are also using this field for the indirect delete, since otherwise the globals structure would be very unaligned (74-bits long). To make matters worse, the id field for globals contains the delta used to reconstruct the globals at mount time. Which means the id field could take on very absurd values and break the dir fetch logic if we're not careful. Solution is to use the scope portion of the type field where necessary, although unforunately this does add some code cost.
2018-07-08 19:21:29 +00:00
// TODO AHHHH, scopes, what about user scope above?
if (lfs_tag_scope(attr.tag) == LFS_SCOPE_FS ||
lfs_tag_scope(attr.tag) == LFS_SCOPE_DIR) {
// ignore non-matching ids
return 0;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_id(attr.tag) != move->id.from) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// ignore non-matching ids
return 0;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// check if type has already been committed
int err = lfs_dir_get(lfs,
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
&(lfs_mdir_t){
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
.pair[0]=move->commit->block,
.off=move->commit->off,
.etag=move->commit->ptag,
.stop_at_commit=true},
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
(lfs_tag_scope(attr.tag) == LFS_SCOPE_STRUCT) ? // TODO also entry?
0x7e3ff000 : 0x7ffff000,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
&(lfs_mattr_t){
lfs_mktag(lfs_tag_type(attr.tag),
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
move->id.to - move->commit->filter.begin, 0)}); // TODO can all these filter adjustments be consolidated?
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err != LFS_ERR_NOENT) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// already committed
return 0;
}
// update id and commit, as we are currently unique
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
attr.tag = lfs_mktag(0, move->id.to, 0) | (attr.tag & 0xffc00fff);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
return lfs_commit_commit(lfs, move->commit, attr);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
static int lfs_commit_move(lfs_t *lfs, struct lfs_commit *commit,
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
uint16_t fromid, uint16_t toid,
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *dir, lfs_mattrlist_t *list) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
struct lfs_commit_move move = {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
.id.from = fromid,
.id.to = toid,
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
.commit = commit,
};
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
for (; list; list = list->next) {
int err = lfs_commit_movescan(lfs, &move, list->e);
if (err) {
return err;
}
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
int err = lfs_dir_traverse(lfs, dir, lfs_commit_movescan, &move);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
return err;
}
return 0;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
static int lfs_commit_globals(lfs_t *lfs, struct lfs_commit *commit,
const lfs_globals_t *source, const lfs_globals_t *diff) {
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
if (lfs_globals_iszero(diff)) {
return 0;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
// TODO check performance/complexity of different strategies here
lfs_globals_t res = lfs_globals_xor(source, diff);
int err = lfs_commit_commit(lfs, commit, (lfs_mattr_t){
lfs_mktag(LFS_TYPE_IDELETE,
res.move.id, sizeof(res.move.pair)),
.u.buffer=res.move.pair});
if (err) {
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
return 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_alloc(lfs_t *lfs, lfs_mdir_t *dir,
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
bool split, const lfs_block_t tail[2]) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// allocate pair of dir blocks (backwards, so we write to block 1 first)
for (int i = 0; i < 2; i++) {
int err = lfs_alloc(lfs, &dir->pair[(i+1)%2]);
if (err) {
return err;
}
}
// rather than clobbering one of the blocks we just pretend
// the revision may be valid
int err = lfs_bd_read(lfs, dir->pair[0], 0, &dir->rev, 4);
dir->rev = lfs_fromle32(dir->rev);
if (err) {
return err;
}
// set defaults
dir->off = sizeof(dir->rev);
dir->etag = 0;
dir->count = 0;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
dir->tail[0] = tail[0];
dir->tail[1] = tail[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->erased = false;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->split = split;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
dir->globals = (lfs_globals_t){0};
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// don't write out yet, let caller take care of that
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_fetchwith(lfs_t *lfs,
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *dir, const lfs_block_t pair[2],
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int (*cb)(lfs_t *lfs, void *data, lfs_mattr_t attr), void *data) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->pair[0] = pair[0];
dir->pair[1] = pair[1];
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
dir->stop_at_commit = false;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// find the block with the most recent revision
uint32_t rev[2];
for (int i = 0; i < 2; i++) {
int err = lfs_bd_read(lfs, dir->pair[i], 0, &rev[i], sizeof(rev[i]));
rev[i] = lfs_fromle32(rev[i]);
if (err) {
return err;
}
}
if (lfs_scmp(rev[1], rev[0]) > 0) {
lfs_pairswap(dir->pair);
lfs_pairswap(rev);
}
// load blocks and check crc
for (int i = 0; i < 2; i++) {
lfs_off_t off = sizeof(dir->rev);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t ptag = 0;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint32_t crc = 0xffffffff;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->tail[0] = 0xffffffff;
dir->tail[1] = 0xffffffff;
dir->count = 0;
dir->split = false;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
dir->globals = (lfs_globals_t){0};
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->moveid = -1;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->rev = lfs_tole32(rev[0]);
lfs_crc(&crc, &dir->rev, sizeof(dir->rev));
dir->rev = lfs_fromle32(dir->rev);
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t temp = *dir;
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
while (true) {
// extract next tag
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t tag;
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
int err = lfs_bd_read(lfs, temp.pair[0], off, &tag, sizeof(tag));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
lfs_crc(&crc, &tag, sizeof(tag));
tag = lfs_fromle32(tag) ^ ptag;
// next commit not yet programmed
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_type(ptag) == LFS_TYPE_CRC && !lfs_tag_valid(tag)) {
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// synthetic move
if (lfs_paircmp(dir->pair, lfs->globals.move.pair) == 0
&& cb) {
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
int err = cb(lfs, data, (lfs_mattr_t){
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs_mktag(LFS_STRUCT_DELETE,
lfs->globals.move.id, 0)});
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
if (err) {
return err;
}
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->erased = true;
return 0;
}
// check we're in valid range
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (off + sizeof(tag)+lfs_tag_size(tag) > lfs->cfg->block_size) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
break;
}
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
//printf("tag r %#010x (%x:%x %03x %03x %03x)\n", tag, temp.pair[0], off+sizeof(tag), lfs_tag_type(tag), lfs_tag_id(tag), lfs_tag_size(tag));
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_type(tag) == LFS_TYPE_CRC) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// check the crc attr
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint32_t dcrc;
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
int err = lfs_bd_read(lfs, temp.pair[0],
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
off+sizeof(tag), &dcrc, sizeof(dcrc));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
if (crc != lfs_fromle32(dcrc)) {
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
if (off == sizeof(temp.rev)) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// try other block
break;
} else {
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// snythetic move
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
// TODO combine with above?
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
if (lfs_paircmp(dir->pair, lfs->globals.move.pair) == 0
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
&& cb) {
int err = cb(lfs, data, (lfs_mattr_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
lfs_mktag(LFS_STRUCT_DELETE,
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs->globals.move.id, 0)});
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
if (err) {
return err;
}
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// consider what we have good enough
dir->erased = false;
return 0;
}
}
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
temp.off = off + sizeof(tag)+lfs_tag_size(tag);
temp.etag = tag;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
crc = 0xffffffff;
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
*dir = temp;
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
// TODO simplify this?
if (cb) {
err = cb(lfs, data, (lfs_mattr_t){
(tag | 0x80000000),
.u.d.block=temp.pair[0],
.u.d.off=off+sizeof(tag)});
if (err) {
return err;
}
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
} else {
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
// TODO crc before callback???
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
err = lfs_bd_crc(lfs, temp.pair[0],
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
off+sizeof(tag), lfs_tag_size(tag), &crc);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_struct(tag) == LFS_STRUCT_TAIL) {
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
temp.split = lfs_tag_type(tag) == LFS_TYPE_HARDTAIL;
err = lfs_bd_read(lfs, temp.pair[0], off+sizeof(tag),
temp.tail, sizeof(temp.tail));
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
} else if (lfs_tag_type(tag) == LFS_TYPE_IDELETE) {
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
temp.globals.move.id = lfs_tag_id(tag);
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
err = lfs_bd_read(lfs, temp.pair[0], off+sizeof(tag),
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
&temp.globals.move.pair,
sizeof(temp.globals.move.pair));
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
if (err) {
return err;
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
} else if (lfs_tag_type(tag) == LFS_STRUCT_MOVE) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// TODO handle moves correctly?
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
temp.moveid = lfs_tag_id(tag);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
} else {
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
if (lfs_tag_scope(tag) <= LFS_SCOPE_ENTRY &&
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
lfs_tag_id(tag) >= temp.count) {
temp.count = lfs_tag_id(tag)+1;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_struct(tag) == LFS_STRUCT_DELETE) {
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
temp.count -= 1;
if (temp.moveid != -1) {
//printf("RENAME DEL %d (%d)\n", lfs_tag_id(tag), temp.moveid);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
if (lfs_tag_id(tag) == temp.moveid) {
temp.moveid = -1;
} else if (lfs_tag_id(tag) < temp.moveid) {
temp.moveid -= 1;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
}
if (cb) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = cb(lfs, data, (lfs_mattr_t){
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
(tag | 0x80000000),
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
.u.d.block=temp.pair[0],
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
.u.d.off=off+sizeof(tag)});
if (err) {
return err;
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
}
}
ptag = tag;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
off += sizeof(tag)+lfs_tag_size(tag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
// failed, try the other crc?
lfs_pairswap(dir->pair);
lfs_pairswap(rev);
}
LFS_ERROR("Corrupted dir pair at %d %d", dir->pair[0], dir->pair[1]);
return LFS_ERR_CORRUPT;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_fetch(lfs_t *lfs,
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *dir, const lfs_block_t pair[2]) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return lfs_dir_fetchwith(lfs, dir, pair, NULL, NULL);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
static int lfs_dir_traverse(lfs_t *lfs, lfs_mdir_t *dir,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int (*cb)(lfs_t *lfs, void *data, lfs_mattr_t attr), void *data) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// iterate over dir block backwards (for faster lookups)
lfs_block_t block = dir->pair[0];
lfs_off_t off = dir->off;
lfs_tag_t tag = dir->etag;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// synthetic move
if (lfs_paircmp(dir->pair, lfs->globals.move.pair) == 0) {
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
int err = cb(lfs, data, (lfs_mattr_t){
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs_mktag(LFS_STRUCT_DELETE, lfs->globals.move.id, 0)});
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
if (err) {
return err;
}
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
while (off != sizeof(uint32_t)) {
// TODO rm me
//printf("tag r %#010x (%x:%x %03x %03x %03x)\n", tag, block, off-lfs_tag_size(tag), lfs_tag_type(tag), lfs_tag_id(tag), lfs_tag_size(tag));
// TODO hmm
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
if (lfs_tag_type(tag) == LFS_TYPE_CRC) {
if (dir->stop_at_commit) {
break;
}
} else {
int err = cb(lfs, data, (lfs_mattr_t){
(0x80000000 | tag),
.u.d.block=block,
.u.d.off=off-lfs_tag_size(tag)});
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
LFS_ASSERT(off > sizeof(tag)+lfs_tag_size(tag));
off -= sizeof(tag)+lfs_tag_size(tag);
lfs_tag_t ntag;
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
int err = lfs_bd_read(lfs, block, off, &ntag, sizeof(ntag));
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
tag ^= lfs_fromle32(ntag);
}
return 0;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_compact(lfs_t *lfs, lfs_mdir_t *dir, lfs_mattrlist_t *list,
lfs_mdir_t *source, uint16_t begin, uint16_t end) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// save some state in case block is bad
const lfs_block_t oldpair[2] = {dir->pair[1], dir->pair[0]};
bool relocated = false;
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
// There's nothing special about our global delta, so feed it back
// into the global global delta
Fixed bug where globals were poisoning move commits The issue lies in the reuse of the id field for globals. Before globals, the only tags with a non-null (0x3ff) id field were names, structs, and other file-specific metadata. But globals are also using this field for the indirect delete, since otherwise the globals structure would be very unaligned (74-bits long). To make matters worse, the id field for globals contains the delta used to reconstruct the globals at mount time. Which means the id field could take on very absurd values and break the dir fetch logic if we're not careful. Solution is to use the scope portion of the type field where necessary, although unforunately this does add some code cost.
2018-07-08 19:21:29 +00:00
// TODO IMMENSE HMM globals get bleed into from above, need to be fixed after commits due to potential moves
lfs_globals_t gtemp = dir->globals; // TODO hmm, why did we have different variables then?
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
lfs->diff = lfs_globals_xor(&lfs->diff, &dir->globals);
dir->globals = (lfs_globals_t){0};
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// increment revision count
dir->rev += 1;
while (true) {
// last complete id
int16_t ack = -1;
dir->count = end - begin;
if (true) {
// erase block to write to
int err = lfs_bd_erase(lfs, dir->pair[1]);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// write out header
uint32_t crc = 0xffffffff;
uint32_t rev = lfs_tole32(dir->rev);
lfs_crc(&crc, &rev, sizeof(rev));
err = lfs_bd_prog(lfs, dir->pair[1], 0, &rev, sizeof(rev));
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// setup compaction
struct lfs_commit commit = {
.block = dir->pair[1],
.off = sizeof(dir->rev),
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
// space is complicated, we need room for tail, crc, idelete,
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// and we keep cap at around half a block
.begin = 0,
.end = lfs_min(
lfs_alignup(lfs->cfg->block_size / 2,
lfs->cfg->prog_size),
lfs->cfg->block_size - 5*sizeof(uint32_t)),
.crc = crc,
.ptag = 0,
// filter out ids
.filter.begin = begin,
.filter.end = end,
};
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
// commit with a move
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
for (uint16_t id = begin; id < end; id++) {
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
err = lfs_commit_move(lfs, &commit, id, id, source, list);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
if (err == LFS_ERR_NOSPC) {
goto split;
} else if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
ack = id;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// reopen the reserved space at the end
// TODO can I just commit these first?
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
commit.end = lfs->cfg->block_size - 2*sizeof(uint32_t);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
if (!relocated) {
err = lfs_commit_globals(lfs, &commit,
&dir->globals, &lfs->diff);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
if (err == LFS_ERR_NOSPC) {
goto split;
} else if (err == LFS_ERR_CORRUPT) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
goto relocate;
}
return err;
}
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
if (!lfs_pairisnull(dir->tail)) {
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
// TODO le32
err = lfs_commit_commit(lfs, &commit, (lfs_mattr_t){
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs_mktag(LFS_STRUCT_TAIL + dir->split*0x8,
0x3ff, sizeof(dir->tail)),
.u.buffer=dir->tail});
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_commit_crc(lfs, &commit);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// successful compaction, swap dir pair to indicate most recent
lfs_pairswap(dir->pair);
dir->off = commit.off;
dir->etag = commit.ptag;
dir->erased = true;
}
break;
split:
// commit no longer fits, need to split dir,
// drop caches and create tail
lfs->pcache.block = 0xffffffff;
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t tail;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
int err = lfs_dir_alloc(lfs, &tail, dir->split, dir->tail);
if (err) {
return err;
}
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
err = lfs_dir_compact(lfs, &tail, list, dir, ack+1, end);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
return err;
}
end = ack+1;
dir->tail[0] = tail.pair[0];
dir->tail[1] = tail.pair[1];
dir->split = true;
continue;
relocate:
//commit was corrupted
LFS_DEBUG("Bad block at %d", dir->pair[1]);
// drop caches and prepare to relocate block
relocated = true;
lfs->pcache.block = 0xffffffff;
// can't relocate superblock, filesystem is now frozen
if (lfs_paircmp(oldpair, (const lfs_block_t[2]){0, 1}) == 0) {
LFS_WARN("Superblock %d has become unwritable", oldpair[1]);
return LFS_ERR_CORRUPT;
}
// relocate half of pair
err = lfs_alloc(lfs, &dir->pair[1]);
if (err) {
return err;
}
continue;
}
if (relocated) {
// update references if we relocated
LFS_DEBUG("Relocating %d %d to %d %d",
oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]);
int err = lfs_relocate(lfs, oldpair, dir->pair);
if (err) {
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
} else {
lfs->globals = lfs_globals_xor(&lfs->globals, &lfs->diff);
lfs->diff = (lfs_globals_t){0};
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Fixed bug where globals were poisoning move commits The issue lies in the reuse of the id field for globals. Before globals, the only tags with a non-null (0x3ff) id field were names, structs, and other file-specific metadata. But globals are also using this field for the indirect delete, since otherwise the globals structure would be very unaligned (74-bits long). To make matters worse, the id field for globals contains the delta used to reconstruct the globals at mount time. Which means the id field could take on very absurd values and break the dir fetch logic if we're not careful. Solution is to use the scope portion of the type field where necessary, although unforunately this does add some code cost.
2018-07-08 19:21:29 +00:00
lfs->globals = lfs_globals_xor(&lfs->globals, &gtemp); // TODO hmm, why did we have different variables then?
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
return 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_commit(lfs_t *lfs, lfs_mdir_t *dir, lfs_mattrlist_t *list) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
while (true) {
if (!dir->erased) {
// not erased, must compact
goto compact;
}
struct lfs_commit commit = {
.block = dir->pair[0],
.begin = dir->off,
.off = dir->off,
.end = lfs->cfg->block_size - 2*sizeof(uint32_t),
.crc = 0xffffffff,
.ptag = dir->etag,
.filter.begin = 0,
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
.filter.end = 0x3ff,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
};
int err = lfs_commit_list(lfs, &commit, list);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
return err;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
err = lfs_commit_globals(lfs, &commit, &dir->globals, &lfs->diff);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
return err;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_commit_crc(lfs, &commit);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
goto compact;
}
return err;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// successful commit, lets update dir
dir->off = commit.off;
dir->etag = commit.ptag;
Fixed bug where globals were poisoning move commits The issue lies in the reuse of the id field for globals. Before globals, the only tags with a non-null (0x3ff) id field were names, structs, and other file-specific metadata. But globals are also using this field for the indirect delete, since otherwise the globals structure would be very unaligned (74-bits long). To make matters worse, the id field for globals contains the delta used to reconstruct the globals at mount time. Which means the id field could take on very absurd values and break the dir fetch logic if we're not careful. Solution is to use the scope portion of the type field where necessary, although unforunately this does add some code cost.
2018-07-08 19:21:29 +00:00
// // TODO hm
// dir->globals = lfs_globals_xor(&dir->globals, &lfs->diff);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs->globals = lfs_globals_xor(&lfs->globals, &lfs->diff);
lfs->diff = (lfs_globals_t){0};
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
break;
compact:
lfs->pcache.block = 0xffffffff;
err = lfs_dir_compact(lfs, dir, list, dir, 0, dir->count);
if (err) {
return err;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
break;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// update any directories that are affected
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
// TODO what about pairs? what if we're splitting??
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (lfs_paircmp(d->m.pair, dir->pair) == 0) {
d->m = *dir;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
}
return 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_append(lfs_t *lfs, lfs_mdir_t *dir, uint16_t *id) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
*id = dir->count;
dir->count += 1;
return 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_delete(lfs_t *lfs, lfs_mdir_t *dir, uint16_t id) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->count -= 1;
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
// check if we should drop the directory block
if (dir->count == 0) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t pdir;
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
int res = lfs_pred(lfs, dir->pair, &pdir);
if (res < 0) {
return res;
}
if (res && pdir.split) {
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// steal tail, and global state
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
pdir.split = dir->split;
pdir.tail[0] = dir->tail[0];
pdir.tail[1] = dir->tail[1];
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs->diff = dir->globals;
lfs->globals = lfs_globals_xor(&lfs->globals, &dir->globals);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int err = lfs_dir_commit(lfs, &pdir, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_TAIL + pdir.split*0x8,
0x3ff, sizeof(pdir.tail)),
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
.u.buffer=pdir.tail}});
return err;
}
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int err = lfs_dir_commit(lfs, dir, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_DELETE, id, 0)}});
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
if (err) {
return err;
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
// shift over any dirs/files that are affected
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (lfs_paircmp(d->m.pair, dir->pair) == 0) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (d->id > id) {
d->id -= 1;
d->pos -= 1;
}
}
}
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
for (lfs_file_t *f = lfs->files; f; f = f->next) {
if (lfs_paircmp(f->pair, dir->pair) == 0) {
if (f->id == id) {
f->pair[0] = 0xffffffff;
f->pair[1] = 0xffffffff;
} else if (f->id > id) {
f->id -= 1;
}
}
}
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
return 0;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
struct lfs_dir_get {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
uint32_t mask;
lfs_tag_t tag;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t *attr;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
};
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
static int lfs_dir_getscan(lfs_t *lfs, void *p, lfs_mattr_t attr) {
struct lfs_dir_get *get = p;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if ((attr.tag & get->mask) == (get->tag & get->mask)) {
*get->attr = attr;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
return true;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
} else if (lfs_tag_type(attr.tag) == LFS_STRUCT_DELETE) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_id(attr.tag) <= lfs_tag_id(get->tag)) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
get->tag += lfs_mktag(0, 1, 0);
}
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
return false;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_get(lfs_t *lfs, lfs_mdir_t *dir,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
uint32_t mask, lfs_mattr_t *attr) {
uint16_t id = lfs_tag_id(attr->tag);
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
int res = lfs_dir_traverse(lfs, dir, lfs_dir_getscan,
&(struct lfs_dir_get){mask, attr->tag, attr});
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_NOENT;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// TODO hmm, stop at commit? maybe we need to handle this elsewhere?
// Should commit get be its own thing? commit traverse?
if (id == dir->moveid && !dir->stop_at_commit) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int moved = lfs_moved(lfs, dir, dir->moveid);
if (moved < 0) {
return moved;
}
if (moved) {
return LFS_ERR_NOENT;
}
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
attr->tag = lfs_mktag(0, id, 0) | (attr->tag & 0xffc00fff);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
return 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_getbuffer(lfs_t *lfs, lfs_mdir_t *dir,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
uint32_t mask, lfs_mattr_t *attr) {
void *buffer = attr->u.buffer;
lfs_size_t size = lfs_tag_size(attr->tag);
int err = lfs_dir_get(lfs, dir, mask, attr);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_size_t diff = lfs_min(size, lfs_tag_size(attr->tag));
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
memset((uint8_t*)buffer + diff, 0, size - diff);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_bd_read(lfs, attr->u.d.block, attr->u.d.off, buffer, diff);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_size(attr->tag) > size) {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
return LFS_ERR_RANGE;
}
return 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_getentry(lfs_t *lfs, lfs_mdir_t *dir,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
uint32_t mask, lfs_tag_t tag, lfs_mattr_t *attr) {
attr->tag = tag | sizeof(attr->u);
attr->u.buffer = &attr->u;
int err = lfs_dir_getbuffer(lfs, dir, mask, attr);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err && err != LFS_ERR_RANGE) {
return err;
}
return 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_getinfo(lfs_t *lfs, lfs_mdir_t *dir,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int16_t id, struct lfs_info *info) {
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
if (id == dir->moveid) {
int moved = lfs_moved(lfs, dir, dir->moveid);
if (moved < 0) {
return moved;
}
if (moved) {
return LFS_ERR_NOENT;
}
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t attr;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
int err = lfs_dir_getentry(lfs, dir, 0x703ff000,
lfs_mktag(LFS_SCOPE_STRUCT, id, 0), &attr);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
info->type = lfs_tag_subtype(attr.tag);
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_struct(attr.tag) == LFS_STRUCT_CTZ) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
info->size = attr.u.ctz.size;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
} else if (lfs_tag_struct(attr.tag) == LFS_STRUCT_INLINE) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
info->size = lfs_tag_size(attr.tag);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
err = lfs_dir_getbuffer(lfs, dir, 0x71fff000, &(lfs_mattr_t){
lfs_mktag(LFS_STRUCT_NAME, id, lfs->name_size+1),
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
.u.buffer=info->name});
if (err) {
return err;
}
return 0;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
struct lfs_dir_find {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
const char *name;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
uint16_t len;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
int16_t id;
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
int16_t tempid;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
};
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
static int lfs_dir_findscan(lfs_t *lfs, void *p, lfs_mattr_t attr) {
struct lfs_dir_find *find = p;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_struct(attr.tag) == LFS_STRUCT_NAME &&
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_tag_size(attr.tag) == find->len) {
int res = lfs_bd_cmp(lfs, attr.u.d.block, attr.u.d.off,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
find->name, find->len);
if (res < 0) {
return res;
}
if (res) {
// found a match
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
find->tempid = lfs_tag_id(attr.tag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
} else if (lfs_tag_type(attr.tag) == LFS_STRUCT_DELETE) {
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
if (lfs_tag_id(attr.tag) == find->tempid) {
find->tempid = -1;
} else if (lfs_tag_id(attr.tag) < find->tempid) {
find->tempid -= 1;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
} else if (lfs_tag_type(attr.tag) == LFS_TYPE_CRC) {
find->id = find->tempid;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// TODO drop others, make this only return id, also make get take in only entry to populate (with embedded tag)
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_dir_find(lfs_t *lfs, lfs_mdir_t *dir,
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
const char **path, uint16_t *id) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t attr = {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
.u.pair[0] = lfs->root[0],
.u.pair[1] = lfs->root[1],
};
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
struct lfs_dir_find find = {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
.name = *path,
};
while (true) {
nextname:
// skip slashes
find.name += strspn(find.name, "/");
find.len = strcspn(find.name, "/");
// special case for root dir
if (find.name[0] == '\0') {
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
// Return ISDIR when we hit root
return LFS_ERR_ISDIR;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
// skip '.' and root '..'
if ((find.len == 1 && memcmp(find.name, ".", 1) == 0) ||
(find.len == 2 && memcmp(find.name, "..", 2) == 0)) {
find.name += find.len;
goto nextname;
}
// skip if matched by '..' in name
const char *suffix = find.name + find.len;
lfs_size_t sufflen;
int depth = 1;
while (true) {
suffix += strspn(suffix, "/");
sufflen = strcspn(suffix, "/");
if (sufflen == 0) {
break;
}
if (sufflen == 2 && memcmp(suffix, "..", 2) == 0) {
depth -= 1;
if (depth == 0) {
find.name = suffix + sufflen;
goto nextname;
}
} else {
depth += 1;
}
suffix += sufflen;
}
// update what we've found
*path = find.name;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// find path
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
while (true) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
//printf("checking %d %d for %s\n", attr.u.pair[0], attr.u.pair[1], *path);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
find.id = -1;
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
find.tempid = -1;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int err = lfs_dir_fetchwith(lfs, dir, attr.u.pair,
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
lfs_dir_findscan, &find);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
if (find.id >= 0) {
// found it
break;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (!dir->split) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return LFS_ERR_NOENT;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
attr.u.pair[0] = dir->tail[0];
attr.u.pair[1] = dir->tail[1];
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (find.id == dir->moveid) {
int moved = lfs_moved(lfs, dir, dir->moveid);
if (moved < 0) {
return moved;
}
if (moved) {
return LFS_ERR_NOENT;
}
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*id = find.id;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
find.name += find.len;
find.name += strspn(find.name, "/");
if (find.name[0] == '\0') {
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// TODO optimize grab for inline files and like?
// TODO would this mean more code?
// grab the entry data
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
int err = lfs_dir_getentry(lfs, dir, 0x703ff000,
lfs_mktag(LFS_SCOPE_STRUCT, find.id, 0), &attr);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (err) {
return err;
Added initial superblock definition Really started working out how the internal structure of the driver will be organized. There are a few hazy lines between the intended data structures with the goal of code reuse, so the function boundaries may end up a bit weird.
2017-03-05 20:11:52 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// continue on if we hit a directory
// TODO update with what's on master?
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_subtype(attr.tag) != LFS_TYPE_DIR) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return LFS_ERR_NOTDIR;
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
}
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level directory operations ///
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
int lfs_mkdir(lfs_t *lfs, const char *path) {
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_deorphan(lfs);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
int err = lfs_dir_find(lfs, &cwd, &path, &(uint16_t){0});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err != LFS_ERR_NOENT || strchr(path, '/') != NULL) {
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
if (!err || err == LFS_ERR_ISDIR) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return LFS_ERR_EXIST;
}
return err;
}
// check that name fits
lfs_size_t nlen = strlen(path);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
// build up new directory
lfs_alloc_ack(lfs);
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_alloc(lfs, &dir, false, cwd.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
err = lfs_dir_commit(lfs, &dir, NULL);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
// get next slot and commit
uint16_t id;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_append(lfs, &cwd, &id);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
cwd.tail[0] = dir.pair[0];
cwd.tail[1] = dir.pair[1];
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_dir_commit(lfs, &cwd, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_NAME | LFS_TYPE_DIR, id, nlen),
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
.u.buffer=(void*)path}, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_DIR | LFS_TYPE_DIR, id, sizeof(dir.pair)),
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
.u.buffer=dir.pair}, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_TYPE_SOFTTAIL, 0x3ff, sizeof(cwd.tail)),
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
.u.buffer=cwd.tail}}}});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// TODO need ack here?
lfs_alloc_ack(lfs);
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_open(lfs_t *lfs, lfs_dir_t *dir, const char *path) {
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
uint16_t id;
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_find(lfs, &dir->m, &path, &id);
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
if (err && err != LFS_ERR_ISDIR) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t attr;
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
if (err == LFS_ERR_ISDIR) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// handle root dir separately
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
attr.u.pair[0] = lfs->root[0];
attr.u.pair[1] = lfs->root[1];
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else {
// get dir pair from parent
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
err = lfs_dir_getentry(lfs, &dir->m, 0x703ff000,
lfs_mktag(LFS_SCOPE_STRUCT, id, 0), &attr);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_subtype(attr.tag) != LFS_TYPE_DIR) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return LFS_ERR_NOTDIR;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// fetch first pair
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
err = lfs_dir_fetch(lfs, &dir->m, attr.u.pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// setup entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->head[0] = dir->m.pair[0];
dir->head[1] = dir->m.pair[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->id = 0;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->pos = 0;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// add to list of directories
dir->next = lfs->dirs;
lfs->dirs = dir;
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// remove from list of directories
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
for (lfs_dir_t **p = &lfs->dirs; *p; p = &(*p)->next) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (*p == dir) {
*p = dir->next;
break;
}
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_read(lfs_t *lfs, lfs_dir_t *dir, struct lfs_info *info) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
memset(info, 0, sizeof(*info));
// special offset for '.' and '..'
if (dir->pos == 0) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, ".");
dir->pos += 1;
return 1;
} else if (dir->pos == 1) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, "..");
dir->pos += 1;
return 1;
}
while (true) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (dir->id == dir->m.count) {
if (!dir->m.split) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return false;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->m.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
dir->id = 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_getinfo(lfs, &dir->m, dir->id, info);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err && err != LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
dir->id += 1;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err != LFS_ERR_NOENT) {
break;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
dir->pos += 1;
return true;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// TODO does this work?
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_seek(lfs_t *lfs, lfs_dir_t *dir, lfs_off_t off) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// simply walk from head dir
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_rewind(lfs, dir);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// first two for ./..
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->pos = lfs_min(2, off);
off -= dir->pos;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
while (off != 0) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->id = lfs_min(dir->m.count, off);
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->pos += dir->id;
off -= dir->id;
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (dir->id == dir->m.count) {
if (!dir->m.split) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
return LFS_ERR_INVAL;
}
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->m.tail);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (err) {
return err;
}
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_soff_t lfs_dir_tell(lfs_t *lfs, lfs_dir_t *dir) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return dir->pos;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_rewind(lfs_t *lfs, lfs_dir_t *dir) {
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
// reload the head dir
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
int err = lfs_dir_fetch(lfs, &dir->m, dir->head);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
return err;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
dir->m.pair[0] = dir->head[0];
dir->m.pair[1] = dir->head[1];
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->id = 0;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
dir->pos = 0;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// File index list operations ///
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_index(lfs_t *lfs, lfs_off_t *off) {
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
lfs_off_t size = *off;
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
lfs_off_t b = lfs->cfg->block_size - 2*4;
lfs_off_t i = size / b;
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
if (i == 0) {
return 0;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
i = (size - 4*(lfs_popc(i-1)+2)) / b;
*off = size - b*i - 4*lfs_popc(i);
return i;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_find(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, const lfs_cache_t *pcache,
lfs_block_t head, lfs_size_t size,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t pos, lfs_block_t *block, lfs_off_t *off) {
if (size == 0) {
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
*off = 0;
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
lfs_off_t current = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
lfs_off_t target = lfs_ctz_index(lfs, &pos);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
while (current > target) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t skip = lfs_min(
lfs_npw2(current-target+1) - 1,
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
lfs_ctz(current));
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
int err = lfs_cache_read(lfs, rcache, pcache, head, 4*skip, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
current -= 1 << skip;
}
*block = head;
*off = pos;
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_extend(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, lfs_cache_t *pcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
lfs_block_t *block, lfs_off_t *off) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
// go ahead and grab a block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(nblock >= 2 && nblock <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
if (true) {
err = lfs_bd_erase(lfs, nblock);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (size == 0) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 0;
return 0;
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
size -= 1;
lfs_off_t index = lfs_ctz_index(lfs, &size);
size += 1;
// just copy out the last block if it is incomplete
if (size != lfs->cfg->block_size) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t data;
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_cache_read(lfs, rcache, NULL,
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
head, i, &data, 1);
if (err) {
return err;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
err = lfs_cache_prog(lfs, pcache, rcache,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, i, &data, 1);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = size;
return 0;
}
// append block
index += 1;
lfs_size_t skips = lfs_ctz(index) + 1;
for (lfs_off_t i = 0; i < skips; i++) {
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_tole32(head);
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_cache_prog(lfs, pcache, rcache,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (i != skips-1) {
err = lfs_cache_read(lfs, rcache, NULL,
head, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 4*skips;
return 0;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
relocate:
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
LFS_DEBUG("Bad block at %d", nblock);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// just clear cache and try a new block
pcache->block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_traverse(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, const lfs_cache_t *pcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int (*cb)(lfs_t*, void*, lfs_block_t), void *data) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (size == 0) {
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
lfs_off_t index = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
while (true) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = cb(lfs, data, head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
if (index == 0) {
return 0;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
lfs_block_t heads[2];
int count = 2 - (index & 1);
err = lfs_cache_read(lfs, rcache, pcache, head, 0, &heads, count*4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
heads[0] = lfs_fromle32(heads[0]);
heads[1] = lfs_fromle32(heads[1]);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
for (int i = 0; i < count-1; i++) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = cb(lfs, data, heads[i]);
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
if (err) {
return err;
}
}
head = heads[count-1];
index -= count;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level file operations ///
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_open(lfs_t *lfs, lfs_file_t *file,
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
const char *path, int flags) {
// deorphan if we haven't yet, needed at most once after poweron
if ((flags & 3) != LFS_O_RDONLY && !lfs->deorphaned) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_deorphan(lfs);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
}
// allocate entry for file if it doesn't exist
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
uint16_t id;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_find(lfs, &cwd, &path, &id);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err && (err != LFS_ERR_NOENT || strchr(path, '/') != NULL)) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t attr;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err == LFS_ERR_NOENT) {
if (!(flags & LFS_O_CREAT)) {
return LFS_ERR_NOENT;
}
// check that name fits
lfs_size_t nlen = strlen(path);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
// get next slot and create entry to remember name
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_append(lfs, &cwd, &id);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
// TODO do we need to make file registered to list to catch updates from this commit? ie if id/cwd change
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_dir_commit(lfs, &cwd, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_NAME | LFS_TYPE_REG, id, nlen),
.u.buffer=(void*)path}, &(lfs_mattrlist_t){
{lfs_mktag(LFS_STRUCT_INLINE | LFS_TYPE_REG, id, 0)}}});
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
// TODO eh
if (id >= cwd.count) {
// catch updates from a compact in the above commit
id -= cwd.count;
cwd.pair[0] = cwd.tail[0];
cwd.pair[1] = cwd.tail[1];
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
attr.tag = lfs_mktag(LFS_STRUCT_INLINE | LFS_TYPE_REG, id, 0);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
if (id == -1) {
return LFS_ERR_ISDIR;
} else if (flags & LFS_O_EXCL) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return LFS_ERR_EXIST;
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
attr.tag = lfs_mktag(LFS_SCOPE_STRUCT, id, 0);
err = lfs_dir_get(lfs, &cwd, 0x703ff000, &attr);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_subtype(attr.tag) != LFS_TYPE_REG) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return LFS_ERR_ISDIR;
}
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
// setup file struct
file->pair[0] = cwd.pair[0];
file->pair[1] = cwd.pair[1];
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
file->id = id;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
file->flags = flags;
file->pos = 0;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
file->attrs = NULL;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
// allocate buffer if needed
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
file->cache.block = 0xffffffff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (lfs->cfg->file_buffer) {
file->cache.buffer = lfs->cfg->file_buffer;
} else if ((file->flags & 3) == LFS_O_RDONLY) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->read_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
return LFS_ERR_NOMEM;
}
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->prog_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
return LFS_ERR_NOMEM;
}
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_struct(attr.tag) == LFS_STRUCT_INLINE) {
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
// load inline files
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->head = 0xfffffffe;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
file->size = lfs_tag_size(attr.tag);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->flags |= LFS_F_INLINE;
file->cache.block = file->head;
file->cache.off = 0;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// don't always read (may be new file)
if (file->size > 0) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_bd_read(lfs, attr.u.d.block, attr.u.d.off,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
file->cache.buffer, file->size);
if (err) {
lfs_free(file->cache.buffer);
return err;
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
} else {
// use ctz list from entry
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_bd_read(lfs, attr.u.d.block, attr.u.d.off,
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
&file->head, 2*sizeof(uint32_t));
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
}
// truncate if requested
if (flags & LFS_O_TRUNC) {
if (file->size != 0) {
file->flags |= LFS_F_DIRTY;
}
file->head = 0xfffffffe;
file->size = 0;
file->flags |= LFS_F_INLINE;
file->cache.block = file->head;
file->cache.off = 0;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
// add to list of files
file->next = lfs->files;
lfs->files = file;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_close(lfs_t *lfs, lfs_file_t *file) {
int err = lfs_file_sync(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// remove from list of files
for (lfs_file_t **p = &lfs->files; *p; p = &(*p)->next) {
if (*p == file) {
*p = file->next;
break;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// clean up memory
if (!lfs->cfg->file_buffer) {
lfs_free(file->cache.buffer);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
return err;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_relocate(lfs_t *lfs, lfs_file_t *file) {
relocate:;
// just relocate what exists into new block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
if (err) {
Fixed missing erase during file relocation This was an easy fix, but highlighted the fact that the current testing framework doesn't detect when a block is written to without an associated erase. Added a quick solution that creates an empty file during an erase.
2017-06-25 21:56:12 +00:00
return err;
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
err = lfs_bd_erase(lfs, nblock);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// either read from dirty cache or disk
for (lfs_off_t i = 0; i < file->off; i++) {
uint8_t data;
err = lfs_cache_read(lfs, &lfs->rcache, &file->cache,
file->block, i, &data, 1);
Adopted redundant cache read in lfs_file_relocate Previously had some custom logic that could be reduced
2017-06-25 22:23:40 +00:00
if (err) {
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
err = lfs_cache_prog(lfs, &lfs->pcache, &lfs->rcache,
nblock, i, &data, 1);
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
// copy over new state of file
memcpy(file->cache.buffer, lfs->pcache.buffer, lfs->cfg->prog_size);
file->cache.block = lfs->pcache.block;
file->cache.off = lfs->pcache.off;
lfs->pcache.block = 0xffffffff;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
file->block = nblock;
return 0;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file) {
if (file->flags & LFS_F_READING) {
file->flags &= ~LFS_F_READING;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (file->flags & LFS_F_WRITING) {
lfs_off_t pos = file->pos;
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (!(file->flags & LFS_F_INLINE)) {
// copy over anything after current branch
lfs_file_t orig = {
.head = file->head,
.size = file->size,
.flags = LFS_O_RDONLY,
.pos = file->pos,
.cache = lfs->rcache,
};
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
lfs->rcache.block = 0xffffffff;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
while (file->pos < file->size) {
// copy over a byte at a time, leave it up to caching
// to make this efficient
uint8_t data;
lfs_ssize_t res = lfs_file_read(lfs, &orig, &data, 1);
if (res < 0) {
return res;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
res = lfs_file_write(lfs, file, &data, 1);
if (res < 0) {
return res;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// keep our reference to the rcache in sync
if (lfs->rcache.block != 0xffffffff) {
orig.cache.block = 0xffffffff;
lfs->rcache.block = 0xffffffff;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// write out what we have
while (true) {
int err = lfs_cache_flush(lfs, &file->cache, &lfs->rcache);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
break;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
relocate:
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
LFS_DEBUG("Bad block at %d", file->block);
err = lfs_file_relocate(lfs, file);
if (err) {
return err;
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->size = lfs_max(file->pos, file->size);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// actual file updates
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->head = file->block;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->size = file->pos;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_WRITING;
file->flags |= LFS_F_DIRTY;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = pos;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) {
int err = lfs_file_flush(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
if ((file->flags & LFS_F_DIRTY) &&
!(file->flags & LFS_F_ERRED) &&
!lfs_pairisnull(file->pair)) {
// update dir entry
// TODO keep list of dirs including these guys for no
// need of another reload?
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_fetch(lfs, &cwd, file->pair);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
// either update the references or inline the whole file
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int err = lfs_dir_commit(lfs, &cwd, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_CTZ | LFS_TYPE_REG,
file->id, 2*sizeof(uint32_t)), .u.buffer=&file->head},
file->attrs});
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
if (err) {
return err;
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
} else {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int err = lfs_dir_commit(lfs, &cwd, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_INLINE | LFS_TYPE_REG,
file->id, file->size), .u.buffer=file->cache.buffer},
file->attrs});
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_DIRTY;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
void *buffer, lfs_size_t size) {
uint8_t *data = buffer;
lfs_size_t nsize = size;
if ((file->flags & 3) == LFS_O_WRONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_WRITING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// flush out any writes
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
if (file->pos >= file->size) {
// eof if past end
return 0;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
size = lfs_min(size, file->size - file->pos);
nsize = size;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
while (nsize > 0) {
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_READING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
int err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
file->pos, &file->block, &file->off);
if (err) {
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags |= LFS_F_READING;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
// read as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
int err = lfs_cache_read(lfs, &file->cache, NULL,
file->block, file->off, data, diff);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
data += diff;
nsize -= diff;
}
return size;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
lfs_size_t nsize = size;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_READING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// drop any reads
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
if ((file->flags & LFS_O_APPEND) && file->pos < file->size) {
file->pos = file->size;
}
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > file->size) {
// fill with zeros
lfs_off_t pos = file->pos;
file->pos = file->size;
while (file->pos < pos) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if ((file->flags & LFS_F_INLINE) &&
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
file->pos + nsize >= lfs->cfg->inline_size) {
Bumped versions, cleaned up some TODOs and missing comments
2018-04-03 13:29:28 +00:00
// inline file doesn't fit anymore
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
file->block = 0xfffffffe;
file->off = file->pos;
lfs_alloc_ack(lfs);
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
int err = lfs_file_relocate(lfs, file);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
file->flags &= ~LFS_F_INLINE;
file->flags |= LFS_F_WRITING;
}
while (nsize > 0) {
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_WRITING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > 0) {
// find out which block we're extending from
int err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
file->pos-1, &file->block, &file->off);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
}
// mark cache as dirty since we may have read data into it
file->cache.block = 0xffffffff;
}
// extend file with new blocks
lfs_alloc_ack(lfs);
int err = lfs_ctz_extend(lfs, &lfs->rcache, &file->cache,
file->block, file->pos,
&file->block, &file->off);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
file->flags |= LFS_F_WRITING;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// program as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs_cache_prog(lfs, &file->cache, &lfs->rcache,
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
file->block, file->off, data, diff);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
break;
relocate:
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
err = lfs_file_relocate(lfs, file);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
data += diff;
nsize -= diff;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_alloc_ack(lfs);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags &= ~LFS_F_ERRED;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
return size;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t off, int whence) {
// write out everything beforehand, may be noop if rdonly
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// update pos
if (whence == LFS_SEEK_SET) {
file->pos = off;
} else if (whence == LFS_SEEK_CUR) {
Fixed positive seek bounds checking This bug was a result of an annoying corner case around intermingling signed and unsigned offsets. The boundary check that prevents seeking a file to a position before the file was preventing valid seeks with positive offsets. This corner case is a bit more complicated than it looks because the offset is signed, while the size of the file is unsigned. Simply casting both to signed or unsigned offsets won't handle large files.
2018-01-03 21:00:04 +00:00
if (off < 0 && (lfs_off_t)-off > file->pos) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return LFS_ERR_INVAL;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = file->pos + off;
} else if (whence == LFS_SEEK_END) {
Fixed positive seek bounds checking This bug was a result of an annoying corner case around intermingling signed and unsigned offsets. The boundary check that prevents seeking a file to a position before the file was preventing valid seeks with positive offsets. This corner case is a bit more complicated than it looks because the offset is signed, while the size of the file is unsigned. Simply casting both to signed or unsigned offsets won't handle large files.
2018-01-03 21:00:04 +00:00
if (off < 0 && (lfs_off_t)-off > file->size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return LFS_ERR_INVAL;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = file->size + off;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Fixed incorrect return value from lfs_file_seek lfs_file_seek returned the _previous_ file offset on success, where most standards return the _calculated_ offset on success. This just falls into me not noticing a mistake, and shows why it's always helpful to have a second set of eyes on code.
2017-09-27 00:50:39 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size) {
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
lfs_off_t oldsize = lfs_file_size(lfs, file);
if (size < oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
// need to flush since directly changing metadata
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
// lookup new head in ctz skip list
err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
size, &file->head, &(lfs_off_t){0});
if (err) {
return err;
}
file->size = size;
file->flags |= LFS_F_DIRTY;
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
} else if (size > oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
lfs_off_t pos = file->pos;
// flush+seek if not already at end
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
if (file->pos != oldsize) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
int err = lfs_file_seek(lfs, file, 0, LFS_SEEK_END);
Fixed error check when truncating files to larger size
2018-02-04 19:48:44 +00:00
if (err < 0) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
return err;
}
}
// fill with zeros
while (file->pos < size) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
// restore pos
int err = lfs_file_seek(lfs, file, pos, LFS_SEEK_SET);
if (err < 0) {
return err;
}
}
return 0;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_tell(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
int lfs_file_rewind(lfs_t *lfs, lfs_file_t *file) {
lfs_soff_t res = lfs_file_seek(lfs, file, 0, LFS_SEEK_SET);
if (res < 0) {
return res;
}
return 0;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (file->flags & LFS_F_WRITING) {
return lfs_max(file->pos, file->size);
} else {
return file->size;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//int lfs_file_getattrs(lfs_t *lfs, lfs_file_t *file,
// const struct lfs_attr *attrs, int count) {
// // set to null in case we can't find the attrs (missing file?)
// for (int j = 0; j < count; j++) {
// memset(attrs[j].buffer, 0, attrs[j].size);
// }
//
// // load from disk if we haven't already been deleted
// if (!lfs_pairisnull(file->pair)) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
// lfs_mdir_t cwd;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// int err = lfs_dir_fetch(lfs, &cwd, file->pair);
// if (err) {
// return err;
// }
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// lfs_mattr_t entry = {.off = file->pairoff};
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// err = lfs_dir_get(lfs, &cwd, entry.off, &entry.d, 4);
// if (err) {
// return err;
// }
// entry.size = lfs_entry_size(&entry);
//
// err = lfs_dir_getattrs(lfs, &cwd, &entry, attrs, count);
// if (err) {
// return err;
// }
// }
//
// // override an attrs we have stored locally
// for (int i = 0; i < file->attrcount; i++) {
// for (int j = 0; j < count; j++) {
// if (attrs[j].type == file->attrs[i].type) {
// if (attrs[j].size < file->attrs[i].size) {
// return LFS_ERR_RANGE;
// }
//
// memset(attrs[j].buffer, 0, attrs[j].size);
// memcpy(attrs[j].buffer,
// file->attrs[i].buffer, file->attrs[i].size);
// }
// }
// }
//
// return 0;
//}
//int lfs_file_setattrs(lfs_t *lfs, lfs_file_t *file,
// const struct lfs_attr *attrs, int count) {
// if ((file->flags & 3) == LFS_O_RDONLY) {
// return LFS_ERR_BADF;
// }
//
// // at least make sure attributes fit
// if (!lfs_pairisnull(file->pair)) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
// lfs_mdir_t cwd;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// int err = lfs_dir_fetch(lfs, &cwd, file->pair);
// if (err) {
// return err;
// }
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// lfs_mattr_t entry = {.off = file->pairoff};
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// err = lfs_dir_get(lfs, &cwd, entry.off, &entry.d, 4);
// if (err) {
// return err;
// }
// entry.size = lfs_entry_size(&entry);
//
// lfs_ssize_t res = lfs_dir_checkattrs(lfs, &cwd, &entry, attrs, count);
// if (res < 0) {
// return res;
// }
// }
//
// // just tack to the file, will be written at sync time
// file->attrs = attrs;
// file->attrcount = count;
// file->flags |= LFS_F_DIRTY;
//
// return 0;
//}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Commentary typo fix
2018-01-30 19:07:37 +00:00
/// General fs operations ///
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
int lfs_stat(lfs_t *lfs, const char *path, struct lfs_info *info) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
uint16_t id;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_find(lfs, &cwd, &path, &id);
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
if (err && err != LFS_ERR_ISDIR) {
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
return err;
}
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
if (err == LFS_ERR_ISDIR) {
// special case for root
strcpy(info->name, "/");
info->type = LFS_TYPE_DIR;
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return lfs_dir_getinfo(lfs, &cwd, id, info);
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int lfs_remove(lfs_t *lfs, const char *path) {
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
int err = lfs_deorphan(lfs);
if (err) {
return err;
}
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
uint16_t id;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
err = lfs_dir_find(lfs, &cwd, &path, &id);
if (err) {
return err;
}
// grab entry to see if we're dealing with a dir
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t attr;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
err = lfs_dir_getentry(lfs, &cwd, 0x703ff000,
lfs_mktag(LFS_SCOPE_STRUCT, id, 0), &attr);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs_mdir_t dir;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_subtype(attr.tag) == LFS_TYPE_DIR) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// must be empty before removal
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_dir_fetch(lfs, &dir, attr.u.pair);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// TODO lfs_dir_empty?
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (dir.count > 0 || dir.split) {
return LFS_ERR_NOTEMPTY;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
//
// // unlink from tail chain and create move to fix
// lfs->diff.move.pair[0] = cwd.pair[0] ^ lfs->globals.move.pair[0];
// lfs->diff.move.pair[1] = cwd.pair[1] ^ lfs->globals.move.pair[1];
// lfs->diff.move.id = id ^ lfs->globals.move.id;
//
// // xor over our child's global state
// lfs->diff = lfs_globals_xor(&lfs->diff, &dir.globals);
// lfs->globals = lfs_globals_xor(&lfs->globals, &dir.globals);
//
// // find pred and remove
// // TODO handle dropped block?
// lfs_mdir_t pred;
// int res = lfs_pred(lfs, dir.pair, &pred);
// if (res < 0) {
// return res;
// }
//
// LFS_ASSERT(res); // must have pred
// pred.tail[0] = dir.tail[0];
// pred.tail[1] = dir.tail[1];
// err = lfs_dir_commit(lfs, &pred, &(lfs_mattrlist_t){
// {lfs_mktag(LFS_TYPE_SOFTTAIL, 0x3ff, sizeof(pred.tail)),
// .u.buffer=pred.tail}});
// if (err) {
// return err;
// }
//
// // mark global state to clear move entry
// lfs->diff.move.pair[0] = 0xffffffff ^ lfs->globals.move.pair[0];
// lfs->diff.move.pair[1] = 0xffffffff ^ lfs->globals.move.pair[1];
// lfs->diff.move.id = 0x3ff ^ lfs->globals.move.id;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
// delete the entry
err = lfs_dir_delete(lfs, &cwd, id);
if (err) {
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// // if we were a directory, fix the move we just created
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// if (lfs_tag_subtype(attr.tag) == LFS_TYPE_DIR) {
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// err = lfs_deorphan(lfs);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// if (err) {
// return err;
// }
// }
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
if (lfs_tag_subtype(attr.tag) == LFS_TYPE_DIR) {
int res = lfs_pred(lfs, dir.pair, &cwd);
if (res < 0) {
return res;
}
LFS_ASSERT(res); // must have pred
cwd.tail[0] = dir.tail[0];
cwd.tail[1] = dir.tail[1];
err = lfs_dir_commit(lfs, &cwd, &(lfs_mattrlist_t){
{lfs_mktag(LFS_TYPE_SOFTTAIL, 0x3ff, sizeof(cwd.tail)),
.u.buffer=cwd.tail}});
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return 0;
}
int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) {
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
int err = lfs_deorphan(lfs);
if (err) {
return err;
}
}
// find old entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t oldcwd;
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
uint16_t oldid;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int err = lfs_dir_find(lfs, &oldcwd, &oldpath, &oldid);
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t oldattr;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
err = lfs_dir_getentry(lfs, &oldcwd, 0x703ff000,
lfs_mktag(LFS_SCOPE_STRUCT, oldid, 0), &oldattr);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
// find new entry
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t newcwd;
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
uint16_t newid;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
err = lfs_dir_find(lfs, &newcwd, &newpath, &newid);
if (err && err != LFS_ERR_NOENT) {
return err;
}
bool prevexists = (err != LFS_ERR_NOENT);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
//bool samepair = (lfs_paircmp(oldcwd.pair, newcwd.pair) == 0);
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t prevattr;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs_mdir_t prevdir;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (prevexists) {
// get prev entry, check that we have same type
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
err = lfs_dir_getentry(lfs, &newcwd, 0x703ff000,
lfs_mktag(LFS_SCOPE_STRUCT, newid, 0), &prevattr);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_subtype(prevattr.tag) != lfs_tag_subtype(oldattr.tag)) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return LFS_ERR_ISDIR;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_subtype(prevattr.tag) == LFS_TYPE_DIR) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// must be empty before removal
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_dir_fetch(lfs, &prevdir, prevattr.u.pair);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
if (prevdir.count > 0 || prevdir.split) {
return LFS_ERR_NOTEMPTY;
}
}
} else {
// check that name fits
lfs_size_t nlen = strlen(newpath);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
// get next id
err = lfs_dir_append(lfs, &newcwd, &newid);
if (err) {
return err;
}
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// create move to fix later
lfs->diff.move.pair[0] = oldcwd.pair[0] ^ lfs->globals.move.pair[0];
lfs->diff.move.pair[1] = oldcwd.pair[1] ^ lfs->globals.move.pair[1];
lfs->diff.move.id = oldid ^ lfs->globals.move.id;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// // mark as moving
// //printf("RENAME MOVE %d %d %d\n", oldcwd.pair[0], oldcwd.pair[1], oldid);
// err = lfs_dir_commit(lfs, &oldcwd, &(lfs_mattrlist_t){
// {lfs_mktag(LFS_STRUCT_MOVE, oldid, 0)}});
// if (err) {
// return err;
// }
//
// if (samepair) {
// // update pair if newcwd == oldcwd
// newcwd = oldcwd;
// }
//
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// TODO check that all complaints are fixed
// // move to new location
// // TODO NAME?????
// // TODO HAH, move doesn't want to override things (due
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
// // to its use in compaction), but that's _exactly what we want here
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// err = lfs_dir_commitwith(lfs, &newcwd, lfs_commit_move,
// &(struct lfs_commit_move){.dir=&oldcwd, .id={oldid, newid}});
// if (err) {
// return err;
// }
// // TODO NONONONONO
// // TODO also don't call strlen twice (see prev name check)
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// err = lfs_dir_commit(lfs, &newcwd, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
// {lfs_mktag(LFS_STRUCT_NAME, newid, strlen(newpath)),
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// .u.buffer=(void*)newpath}});
// if (err) {
// return err;
// }
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// move over all attributes
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_dir_commit(lfs, &newcwd, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_NAME | lfs_tag_subtype(oldattr.tag),
newid, strlen(newpath)),
.u.buffer=(void*)newpath}, &(lfs_mattrlist_t){
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
{lfs_mktag(LFS_FROM_MOVE, newid, oldid),
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
.u.dir=&oldcwd}}});
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// clean up after ourselves
err = lfs_fixmove(lfs);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (prevexists && lfs_tag_subtype(prevattr.tag) == LFS_TYPE_DIR) {
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
int res = lfs_pred(lfs, prevdir.pair, &newcwd);
if (res < 0) {
return res;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
Fixed bug where globals were poisoning move commits The issue lies in the reuse of the id field for globals. Before globals, the only tags with a non-null (0x3ff) id field were names, structs, and other file-specific metadata. But globals are also using this field for the indirect delete, since otherwise the globals structure would be very unaligned (74-bits long). To make matters worse, the id field for globals contains the delta used to reconstruct the globals at mount time. Which means the id field could take on very absurd values and break the dir fetch logic if we're not careful. Solution is to use the scope portion of the type field where necessary, although unforunately this does add some code cost.
2018-07-08 19:21:29 +00:00
// TODO test for global state stealing?
// steal global state
lfs->globals = lfs_globals_xor(&lfs->globals, &prevdir.globals);
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
LFS_ASSERT(res); // must have pred
newcwd.tail[0] = prevdir.tail[0];
newcwd.tail[1] = prevdir.tail[1];
err = lfs_dir_commit(lfs, &newcwd, &(lfs_mattrlist_t){
{lfs_mktag(LFS_TYPE_SOFTTAIL, 0x3ff, sizeof(newcwd.tail)),
.u.buffer=newcwd.tail}});
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
return 0;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// if (samepair) {
// // update pair if newcwd == oldcwd
// oldcwd = newcwd;
// }
//
// err = fix
//
// // remove old entry
// //printf("RENAME DELETE %d %d %d\n", oldcwd.pair[0], oldcwd.pair[1], oldid);
// err = lfs_dir_delete(lfs, &oldcwd, oldid);
// if (err) {
// return err;
// }
//
// // if we were a directory, find pred, replace tail
// // TODO can this just deorphan?
// if (prevexists && lfs_tag_subtype(prevattr.tag) == LFS_TYPE_DIR) {
// err = lfs_deorphan(lfs);
// if (err) {
// return err;
// }
// }
//
return 0;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
//int lfs_getattrs(lfs_t *lfs, const char *path,
// const struct lfs_attr *attrs, int count) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
// lfs_mdir_t cwd;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
// if (err) {
// return err;
// }
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// lfs_mattr_t entry;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// err = lfs_dir_find(lfs, &cwd, &entry, &path);
// if (err) {
// return err;
// }
//
// return lfs_dir_getattrs(lfs, &cwd, &entry, attrs, count);
//}
//
//int lfs_setattrs(lfs_t *lfs, const char *path,
// const struct lfs_attr *attrs, int count) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
// lfs_mdir_t cwd;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
// if (err) {
// return err;
// }
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// lfs_mattr_t entry;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// err = lfs_dir_find(lfs, &cwd, &entry, &path);
// if (err) {
// return err;
// }
//
// return lfs_dir_setattrs(lfs, &cwd, &entry, attrs, count);
//}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
/// Filesystem operations ///
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
lfs->cfg = cfg;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup read cache
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
lfs->rcache.block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->read_buffer) {
lfs->rcache.buffer = lfs->cfg->read_buffer;
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->rcache.buffer = lfs_malloc(lfs->cfg->read_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->rcache.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup program cache
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
lfs->pcache.block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->prog_buffer) {
lfs->pcache.buffer = lfs->cfg->prog_buffer;
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->pcache.buffer = lfs_malloc(lfs->cfg->prog_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->pcache.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
// setup lookahead, round down to nearest 32-bits
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(lfs->cfg->lookahead % 32 == 0);
LFS_ASSERT(lfs->cfg->lookahead > 0);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
if (lfs->cfg->lookahead_buffer) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer = lfs->cfg->lookahead_buffer;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->free.buffer = lfs_malloc(lfs->cfg->lookahead/8);
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!lfs->free.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
}
}
Added asserts on geometry and updated config documentation littlefs had an unwritten assumption that the block device's program size would be a multiple of the read size, and the block size would be a multiple of the program size. This has already caused confusion for users. Added a note and assert to catch unexpected geometries early. Also found that the prog/erase functions indicated they must return LFS_ERR_CORRUPT to catch bad blocks. This is no longer true as errors are found by CRC.
2018-01-11 17:56:09 +00:00
// check that program and read sizes are multiples of the block size
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(lfs->cfg->prog_size % lfs->cfg->read_size == 0);
LFS_ASSERT(lfs->cfg->block_size % lfs->cfg->prog_size == 0);
Added asserts on geometry and updated config documentation littlefs had an unwritten assumption that the block device's program size would be a multiple of the read size, and the block size would be a multiple of the program size. This has already caused confusion for users. Added a note and assert to catch unexpected geometries early. Also found that the prog/erase functions indicated they must return LFS_ERR_CORRUPT to catch bad blocks. This is no longer true as errors are found by CRC.
2018-01-11 17:56:09 +00:00
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
// check that the block size is large enough to fit ctz pointers
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(4*lfs_npw2(0xffffffff / (lfs->cfg->block_size-2*4))
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
<= lfs->cfg->block_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that the size limits are sane
LFS_ASSERT(lfs->cfg->inline_size <= LFS_INLINE_MAX);
LFS_ASSERT(lfs->cfg->inline_size <= lfs->cfg->read_size);
lfs->inline_size = lfs->cfg->inline_size;
if (!lfs->inline_size) {
lfs->inline_size = lfs_min(LFS_INLINE_MAX, lfs->cfg->read_size);
}
LFS_ASSERT(lfs->cfg->attrs_size <= LFS_ATTRS_MAX);
lfs->attrs_size = lfs->cfg->attrs_size;
if (!lfs->attrs_size) {
lfs->attrs_size = LFS_ATTRS_MAX;
}
LFS_ASSERT(lfs->cfg->name_size <= LFS_NAME_MAX);
lfs->name_size = lfs->cfg->name_size;
if (!lfs->name_size) {
lfs->name_size = LFS_NAME_MAX;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// setup default state
lfs->root[0] = 0xffffffff;
lfs->root[1] = 0xffffffff;
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
lfs->files = NULL;
Added directory list for synchronizing in flight directories As it was, if a user operated on a directory while at the same time iterating over the directory, the directory objects could fall out of sync. In the best case, files may be skipped while removing everything in a file, in the worst case, a very poorly timed directory relocate could be missed. Simple fix is to add the same directory tracking that is currently in use for files, at a small code+complexity cost.
2017-11-22 02:53:15 +00:00
lfs->dirs = NULL;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs->deorphaned = false;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
lfs->globals.move.pair[0] = 0xffffffff;
lfs->globals.move.pair[1] = 0xffffffff;
lfs->globals.move.id = 0x3ff;
lfs->diff = (lfs_globals_t){0};
// scan for any global updates
// TODO rm me? need to grab any inits
int err = lfs_scan(lfs);
if (err) {
return err;
}
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
static int lfs_deinit(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// free allocated memory
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->cfg->read_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->rcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
if (!lfs->cfg->prog_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->pcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
if (!lfs->cfg->lookahead_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->free.buffer);
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
int lfs_format(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// create free lookahead
Fixed issue with aggressively rounding down lookahead configuration The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
2017-11-10 01:10:08 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead/8);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
lfs->free.off = 0;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.size = lfs_min(lfs->cfg->lookahead, lfs->cfg->block_count);
lfs->free.i = 0;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs_alloc_ack(lfs);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// create superblock dir
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_alloc(lfs, &dir, false,
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
(const lfs_block_t[2]){0xffffffff, 0xffffffff});
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// write root directory
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t root;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_alloc(lfs, &root, false,
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
(const lfs_block_t[2]){0xffffffff, 0xffffffff});
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Trimmed old names and functions from the code base I've found using temporary names to duplicate functions temporarily (lfs_dir_commit + lfs_dir_commit_) is a great way to introduce sweeping changes while keeping the code base functional and (mostly) passing tests. It does mean at some point I need to deduplicate all these functions.
2018-05-29 00:49:20 +00:00
err = lfs_dir_commit(lfs, &root, NULL);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
lfs->root[0] = root.pair[0];
lfs->root[1] = root.pair[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir.tail[0] = lfs->root[0];
dir.tail[1] = lfs->root[1];
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// write one superblock
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_superblock_t superblock = {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
.root[0] = lfs->root[0],
.root[1] = lfs->root[1],
.magic = {"littlefs"},
.version = LFS_DISK_VERSION,
.block_size = lfs->cfg->block_size,
.block_count = lfs->cfg->block_count,
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
.inline_size = lfs->inline_size,
.attrs_size = lfs->attrs_size,
.name_size = lfs->name_size,
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
};
dir.count += 1;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_dir_commit(lfs, &dir, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_TYPE_SUPERBLOCK | LFS_STRUCT_DIR,
0, sizeof(superblock)), .u.buffer=&superblock}});
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
if (err) {
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
// sanity check that fetch works
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
return lfs_deinit(lfs);
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup free lookahead
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs->free.off = 0;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.size = 0;
lfs->free.i = 0;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs_alloc_ack(lfs);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// load superblock
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
LFS_ERROR("Invalid superblock at %d %d", 0, 1);
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_superblock_t superblock;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
err = lfs_dir_getbuffer(lfs, &dir, 0x7e3ff000, &(lfs_mattr_t){
lfs_mktag(LFS_TYPE_SUPERBLOCK, 0, sizeof(superblock)),
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
.u.buffer=&superblock});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err && err != LFS_ERR_RANGE) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return err;
}
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (memcmp(superblock.magic, "littlefs", 8) != 0) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Invalid superblock at %d %d", 0, 1);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_CORRUPT;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
uint16_t major_version = (0xffff & (superblock.version >> 16));
uint16_t minor_version = (0xffff & (superblock.version >> 0));
Add version info for software library and on-disk structures An annoying part of filesystems is that the software library can change independently of the on-disk structures. For this reason versioning is very important, and must be handled separately for the software and on-disk parts. In this patch, littlefs provides two version numbers at compile time, with major and minor parts, in the form of 6 macros. LFS_VERSION // Library version, uint32_t encoded LFS_VERSION_MAJOR // Major - Backwards incompatible changes LFS_VERSION_MINOR // Minor - Feature additions LFS_DISK_VERSION // On-disk version, uint32_t encoded LFS_DISK_VERSION_MAJOR // Major - Backwards incompatible changes LFS_DISK_VERSION_MINOR // Minor - Feature additions Note that littlefs will error if it finds a major version number that is different, or a minor version number that has regressed.
2018-01-26 20:26:25 +00:00
if ((major_version != LFS_DISK_VERSION_MAJOR ||
minor_version > LFS_DISK_VERSION_MINOR)) {
LFS_ERROR("Invalid version %d.%d", major_version, minor_version);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_INVAL;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (superblock.inline_size) {
if (superblock.inline_size > lfs->inline_size) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Unsupported inline size (%d > %d)",
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
superblock.inline_size, lfs->inline_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return LFS_ERR_INVAL;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->inline_size = superblock.inline_size;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (superblock.attrs_size) {
if (superblock.attrs_size > lfs->attrs_size) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Unsupported attrs size (%d > %d)",
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
superblock.attrs_size, lfs->attrs_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return LFS_ERR_INVAL;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->attrs_size = superblock.attrs_size;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (superblock.name_size) {
if (superblock.name_size > lfs->name_size) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Unsupported name size (%d > %d)",
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
superblock.name_size, lfs->name_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return LFS_ERR_INVAL;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->name_size = superblock.name_size;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->root[0] = superblock.root[0];
lfs->root[1] = superblock.root[1];
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
err = lfs_scan(lfs);
if (err) {
return err;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
int lfs_unmount(lfs_t *lfs) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return lfs_deinit(lfs);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
/// Internal filesystem filesystem operations ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_fs_traverse(lfs_t *lfs,
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
int (*cb)(lfs_t *lfs, void *data, lfs_block_t block), void *data) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
// iterate over metadata pairs
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir = {.tail = {0, 1}};
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
while (!lfs_pairisnull(dir.tail)) {
for (int i = 0; i < 2; i++) {
int err = cb(lfs, data, dir.tail[i]);
if (err) {
return err;
}
}
// iterate through ids in directory
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
for (uint16_t id = 0; id < dir.count; id++) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t attr;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
int err = lfs_dir_getentry(lfs, &dir, 0x703ff000,
lfs_mktag(LFS_SCOPE_STRUCT, id, 0), &attr);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
if (err == LFS_ERR_NOENT) {
continue;
}
return err;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (lfs_tag_struct(attr.tag) == LFS_STRUCT_CTZ) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_ctz_traverse(lfs, &lfs->rcache, NULL,
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
attr.u.ctz.head, attr.u.ctz.size, cb, data);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
}
// iterate over any open files
for (lfs_file_t *f = lfs->files; f; f = f->next) {
if ((f->flags & LFS_F_DIRTY) && !(f->flags & LFS_F_INLINE)) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
f->head, f->size, cb, data);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
if ((f->flags & LFS_F_WRITING) && !(f->flags & LFS_F_INLINE)) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
f->block, f->pos, cb, data);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
// iterate over metadata pairs
lfs_block_t cwd[2] = {0, 1};
while (true) {
for (int i = 0; i < 2; i++) {
int err = cb(data, cwd[i]);
if (err) {
return err;
}
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t dir;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
int err = lfs_dir_fetch(lfs, &dir, cwd);
if (err) {
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
// iterate over contents
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t entry;
Shrinked on-disk directory program size Directories still consume two full erase blocks, but now only program the exact on-disk region to store the directory contents. This results in a decent improvement in the amount of data written and read to the device when doing directory operations. Calculating the checksum of dynamically sized data is surprisingly tricky, since the size of the data could also contain errors. For the littlefs, we can assume the data size must fit in an erase block. If the data size is invalid, we can just treat the block as corrupted.
2017-06-24 01:03:44 +00:00
while (dir.off + sizeof(entry.d) <= (0x7fffffff & dir.d.size)-4) {
Added internal lfs_dir_get to consolidate logic for reading dir entries It's a relatively simple function but offers some code reuse as well as making the dir entry operations a bit more readable.
2018-03-23 23:35:55 +00:00
err = lfs_dir_get(lfs, &dir,
dir.off, &entry.d, sizeof(entry.d));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_entry_fromle32(&entry.d);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
if (err) {
return err;
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
dir.off += lfs_entry_size(&entry);
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if ((0x70 & entry.d.type) == LFS_STRUCT_CTZ) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_ctz_traverse(lfs, &lfs->rcache, NULL,
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
entry.d.u.file.head, entry.d.u.file.size, cb, data);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
if (err) {
return err;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
}
cwd[0] = dir.d.tail[0];
cwd[1] = dir.d.tail[1];
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (lfs_pairisnull(cwd)) {
break;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
// iterate over any open files
for (lfs_file_t *f = lfs->files; f; f = f->next) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if ((f->flags & LFS_F_DIRTY) && !(f->flags & LFS_F_INLINE)) {
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
f->head, f->size, cb, data);
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (err) {
return err;
}
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if ((f->flags & LFS_F_WRITING) && !(f->flags & LFS_F_INLINE)) {
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
f->block, f->pos, cb, data);
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (err) {
return err;
}
}
}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_pred(lfs_t *lfs, const lfs_block_t pair[2], lfs_mdir_t *pdir) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// iterate over all directory directory entries
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
pdir->tail[0] = 0;
pdir->tail[1] = 1;
while (!lfs_pairisnull(pdir->tail)) {
if (lfs_paircmp(pdir->tail, pair) == 0) {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
return true; // TODO should we return true only if pred is part of dir?
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, pdir, pdir->tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_pred(lfs_t *lfs, const lfs_block_t dir[2], lfs_mdir_t *pdir) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// iterate directories
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
int err = lfs_dir_fetch(lfs, pdir, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (!lfs_pairisnull(pdir->d.tail)) {
if (lfs_paircmp(pdir->d.tail, dir) == 0) {
return true;
}
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, pdir, pdir->d.tail);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
// TODO combine parentscan and findscan?
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
struct lfs_dir_parentscan {
lfs_block_t pair[2];
int16_t id;
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
int16_t tempid;
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
};
static int lfs_parentscan(lfs_t *lfs, void *p, lfs_mattr_t attr) {
struct lfs_dir_parentscan *parentscan = p;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
if (lfs_tag_struct(attr.tag) == LFS_STRUCT_DIR) {
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
int err = lfs_bd_read(lfs, attr.u.d.block, attr.u.d.off,
&attr.u, sizeof(attr.u));
if (err) {
return err;
}
if (lfs_paircmp(attr.u.pair, parentscan->pair) == 0) {
// found a match
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
parentscan->tempid = lfs_tag_id(attr.tag);
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
}
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
} else if (lfs_tag_struct(attr.tag) == LFS_STRUCT_DELETE) {
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
if (lfs_tag_id(attr.tag) == parentscan->tempid) {
parentscan->tempid = -1;
} else if (lfs_tag_id(attr.tag) < parentscan->tempid) {
parentscan->tempid -= 1;
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
}
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
} else if (lfs_tag_type(attr.tag) == LFS_TYPE_CRC) {
parentscan->id = parentscan->tempid;
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_parent(lfs_t *lfs, const lfs_block_t pair[2],
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *parent, lfs_mattr_t *attr) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// iterate over all directory directory entries
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
parent->tail[0] = 0;
parent->tail[1] = 1;
while (!lfs_pairisnull(parent->tail)) {
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
struct lfs_dir_parentscan parentscan = {
.pair[0] = pair[0],
.pair[1] = pair[1],
Fixed issues with finding wrong ids after bad commits Unfortunately, the behaviour needed of lfs_dir_fetchwith is as subtle as it is important. When fetching from a block corrupted by power-loss, lfs_dir_fetch must be able to rewind any state it picks up to before the corruption. This is not limited to the directory state, but includes find results and other side-effects. This gets a bit complicated when trying to generalize littlefs's fetchwith mechanics. Being able to scan a directory block during a fetch greatly impacts the runtime of littlefs operations, but if the state is generic how do we know what to rollback to? The fix here is to leave the management of rolling back state to the fetchwith match functions, and transparently pass a CRC tag to indicate the temporary state can be saved.
2018-07-04 06:35:04 +00:00
.id = -1,
.tempid = -1,
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
};
int err = lfs_dir_fetchwith(lfs, parent, parent->tail,
lfs_parentscan, &parentscan);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
if (parentscan.id != -1) {
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
int err = lfs_dir_getentry(lfs, parent, 0x71fff000,
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
lfs_mktag(LFS_STRUCT_DIR, parentscan.id, 0), attr);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
return true;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
return false;
}
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
//
//static int lfs_parent(lfs_t *lfs, const lfs_block_t pair[2],
// lfs_mdir_t *parent, lfs_mattr_t *attr) {
// // iterate over all directory directory entries
// parent->tail[0] = 0;
// parent->tail[1] = 1;
// while (!lfs_pairisnull(parent->tail)) {
// int err = lfs_dir_fetch(lfs, parent, parent->tail);
// if (err) {
// return err;
// }
//
// // TODO make this O(n) by using fetchwith to match the pointers
// for (uint16_t id = 0; id < parent->count; id++) {
// int err = lfs_dir_getentry(lfs, parent, 0x43dff000,
// lfs_mktag(LFS_STRUCT_DIR, id, 0), attr);
// if (err) {
// if (err == LFS_ERR_NOENT) {
// continue;
// }
// return err;
// }
//
// if (lfs_paircmp(attr->u.pair, pair) == 0) {
// return true;
// }
// }
// }
//
// return false;
//}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_parent(lfs_t *lfs, const lfs_block_t dir[2],
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t *parent, lfs_mattr_t *attr) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
parent->d.tail[0] = 0;
parent->d.tail[1] = 1;
// iterate over all directory directory entries
while (!lfs_pairisnull(parent->d.tail)) {
int err = lfs_dir_fetch(lfs, parent, parent->d.tail);
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
if (err) {
return err;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
while (true) {
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_dir_next(lfs, parent, attr);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
if (err && err != LFS_ERR_NOENT) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
if (err == LFS_ERR_NOENT) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
break;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (((0x70 & attr->d.type) == LFS_STRUCT_DIR) &&
lfs_paircmp(attr->d.u.dir, dir) == 0) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
return true;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
static int lfs_moved(lfs_t *lfs, lfs_mdir_t *fromdir, uint16_t fromid) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// grab entry pair we're looking for
fromdir->moveid = -1;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t fromentry;
Fixed issues discovered around testing moves lfs_dir_fetchwith did not recover from failed dir fetches correctly, added a temporary dir variable to hold dir contents while being populated, allowing us to fall back to a known good dir state if a commit is corrupted. There is a RAM cost, but the upside is that our lfs_dir_fetchwith actually works. Also added better handling of move ids during some get functions.
2018-05-28 22:46:32 +00:00
// TODO what about inline files?
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
int err = lfs_dir_getentry(lfs, fromdir, 0x71fff000,
lfs_mktag(LFS_STRUCT_DIR, fromid, 0), &fromentry);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
fromdir->moveid = fromid;
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// skip superblock
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t todir;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
err = lfs_dir_fetch(lfs, &todir, (const lfs_block_t[2]){0, 1});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
// iterate over all directory directory entries
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
while (!lfs_pairisnull(todir.tail)) {
int err = lfs_dir_fetch(lfs, &todir, todir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
for (int toid = 0; toid < todir.count; toid++) {
if (lfs_paircmp(todir.pair, fromdir->pair) == 0 &&
toid == fromid) {
continue;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t toentry;
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
int err = lfs_dir_getentry(lfs, &todir, 0x71fff000,
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
lfs_mktag(LFS_STRUCT_DIR, toid, 0), &toentry);
if (err) {
if (err == LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
continue;
}
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (lfs_paircmp(toentry.u.pair, fromentry.u.pair) == 0) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return true;
}
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
static int lfs_moved(lfs_t *lfs, const void *e) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
// skip superblock
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t cwd;
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
int err = lfs_dir_fetch(lfs, &cwd, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
// iterate over all directory directory entries
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t entry;
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
while (!lfs_pairisnull(cwd.d.tail)) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, &cwd, cwd.d.tail);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
while (true) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_next(lfs, &cwd, &entry);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err == LFS_ERR_NOENT) {
break;
}
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (!(LFS_STRUCT_MOVED & entry.d.type) &&
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
memcmp(&entry.d.u, e, sizeof(entry.d.u)) == 0) {
return true;
}
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// TODO rename to lfs_dir_relocate?
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_relocate(lfs_t *lfs,
const lfs_block_t oldpair[2], const lfs_block_t newpair[2]) {
// find parent
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t parent;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t attr;
int res = lfs_parent(lfs, oldpair, &parent, &attr);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (res < 0) {
return res;
}
if (res) {
// update disk, this creates a desync
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
attr.u.pair[0] = newpair[0];
attr.u.pair[1] = newpair[1];
int err = lfs_dir_commit(lfs, &parent, &(lfs_mattrlist_t){attr});
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// update internal root
if (lfs_paircmp(oldpair, lfs->root) == 0) {
LFS_DEBUG("Relocating root %d %d", newpair[0], newpair[1]);
lfs->root[0] = newpair[0];
lfs->root[1] = newpair[1];
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// TODO update dir list!!?
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// clean up bad block, which should now be a desync
return lfs_deorphan(lfs);
}
// find pred
res = lfs_pred(lfs, oldpair, &parent);
if (res < 0) {
return res;
}
if (res) {
// just replace bad pair, no desync can occur
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
parent.tail[0] = newpair[0];
parent.tail[1] = newpair[1];
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
int err = lfs_dir_commit(lfs, &parent, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_STRUCT_TAIL + parent.split*0x8, // TODO hm
0x3ff, sizeof(lfs_block_t[2])),
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
.u.pair[0]=newpair[0], .u.pair[1]=newpair[1]}});
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (err) {
return err;
}
}
// shift over any dirs/files that are affected
for (int i = 0; i < 2; i++) {
for (lfs_dir_t *d = ((void*[2]){lfs->dirs, lfs->files})[i];
d; d = d->next) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
if (lfs_paircmp(d->m.pair, oldpair) == 0) {
d->m.pair[0] = newpair[0];
d->m.pair[1] = newpair[1];
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
}
}
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// couldn't find dir, must be new
return 0;
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
int lfs_scan(lfs_t *lfs) {
if (lfs_pairisnull(lfs->root)) { // TODO rm me
return 0;
}
lfs_mdir_t dir = {.tail = {0, 1}};
lfs_globals_t globals = {{{0xffffffff, 0xffffffff}, 0x3ff}};
// iterate over all directory directory entries
while (!lfs_pairisnull(dir.tail)) {
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
if (err) {
return err;
}
// xor together indirect deletes
globals = lfs_globals_xor(&globals, &dir.globals);
}
// update littlefs with globals
lfs->globals = globals;
lfs->diff = (lfs_globals_t){0};
if (!lfs_pairisnull(lfs->globals.move.pair)) {
LFS_DEBUG("Found move %d %d %d",
lfs->globals.move.pair[0],
lfs->globals.move.pair[1],
lfs->globals.move.id);
}
return 0;
}
int lfs_fixmove(lfs_t *lfs) {
LFS_DEBUG("Fixing move %d %d %d", // TODO move to just deorphan
lfs->globals.move.pair[0],
lfs->globals.move.pair[1],
lfs->globals.move.id);
// mark global state to clear move entry
lfs->diff.move.pair[0] = 0xffffffff ^ lfs->globals.move.pair[0];
lfs->diff.move.pair[1] = 0xffffffff ^ lfs->globals.move.pair[1];
lfs->diff.move.id = 0x3ff ^ lfs->globals.move.id;
// fetch and delete the moved entry
lfs_mdir_t movedir;
int err = lfs_dir_fetch(lfs, &movedir, lfs->globals.move.pair);
if (err) {
return err;
}
err = lfs_dir_delete(lfs, &movedir, lfs->globals.move.id);
if (err) {
return err;
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_deorphan(lfs_t *lfs) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->deorphaned = true;
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
if (lfs_pairisnull(lfs->root)) { // TODO rm me?
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return 0;
}
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
// Fix bad moves
if (!lfs_pairisnull(lfs->globals.move.pair)) {
int err = lfs_fixmove(lfs);
if (err) {
return err;
}
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t pdir = {.split = true};
lfs_mdir_t dir = {.tail = {0, 1}};
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// iterate over all directory directory entries
while (!lfs_pairisnull(dir.tail)) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
// check head blocks for orphans
if (!pdir.split) {
// check if we have a parent
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t parent;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t attr;
int res = lfs_parent(lfs, pdir.tail, &parent, &attr);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (res < 0) {
return res;
}
if (!res) {
// we are an orphan
LFS_DEBUG("Found orphan %d %d",
pdir.tail[0], pdir.tail[1]);
pdir.tail[0] = dir.tail[0];
pdir.tail[1] = dir.tail[1];
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
err = lfs_dir_commit(lfs, &pdir, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_TYPE_SOFTTAIL,
0x3ff, sizeof(pdir.tail)),
.u.buffer=pdir.tail}});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
break;
}
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
if (!lfs_pairsync(attr.u.pair, pdir.tail)) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// we have desynced
Introduced xored-globals logic to fix fundamental problem with moves This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
2018-05-29 17:35:23 +00:00
LFS_DEBUG("Found half-orphan %d %d",
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
attr.u.pair[0], attr.u.pair[1]);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
pdir.tail[0] = attr.u.pair[0];
pdir.tail[1] = attr.u.pair[1];
err = lfs_dir_commit(lfs, &pdir, &(lfs_mattrlist_t){
Restructured tags to better support xored-globals 32-bit tag structure: [--- 32 ---] [1|- 9 -|- 10 -|-- 12 --] ^ ^ ^ ^- entry length | | \--------- file id | \--------------- tag type \------------------- valid In this tag, the type decomposes into some more information: [--- 9 ---] [1|- 2 -|- 3 -|- 3 -] ^ ^ ^ ^- struct | | \------- type | \------------- scope \----------------- user The change in this encoding is the addition of a global scope: LFS_SCOPE_STRUCT = 0 00 xxx xxx LFS_SCOPE_ENTRY = 0 01 xxx xxx LFS_SCOPE_DIR = 0 10 xxx xxx LFS_SCOPE_FS = 0 11 xxx xxx LFS_SCOPE_USER = 1 xx xxx xxx
2018-05-30 01:08:42 +00:00
{lfs_mktag(LFS_TYPE_SOFTTAIL,
0x3ff, sizeof(pdir.tail)),
.u.buffer=pdir.tail}});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
break;
}
}
// check entries for moves
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
//if (dir.moveid >= 0) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// TODO moves and stuff
// TODO need to load entry to find it
// // found moved entry
// int moved = lfs_moved(lfs, &entry.u);
// if (moved < 0) {
// return moved;
// }
//
// if (moved) {
// LFS_DEBUG("Found move %d %d",
// entry.d.u.dir[0], entry.d.u.dir[1]);
// err = lfs_dir_set(lfs, &dir, &entry, (struct lfs_region[]){
// {LFS_FROM_MEM, 0, entry.size, NULL, 0}}, 1);
// if (err) {
// return err;
// }
// } else {
// LFS_DEBUG("Found partial move %d %d",
// entry.d.u.dir[0], entry.d.u.dir[1]);
// entry.d.type &= ~LFS_STRUCT_MOVED;
// err = lfs_dir_set(lfs, &dir, &entry, (struct lfs_region[]){
// {LFS_FROM_MEM, 0, 1, &entry.d, 1}}, 1);
// if (err) {
// return err;
// }
// }
Switched back to simple deorphan-step on directory remove Originally I tried to reuse the indirect delete to accomplish truely atomic directory removes, however this fell apart when it came to implementing directory removes as a side-effect of renames. A single indirect-delete simply can't handle renames with removes as a side effects. When copying an entry to its destination, we need to atomically delete both the old entry, and the source of our copy. We can't delete both with only a single indirect-delete. It is possible to accomplish this with two indirect-deletes, but this is such an uncommon case that it's really not worth supporting efficiently due to how expensive globals are. I also dropped indirect-deletes for normal directory removes. I may add it back later, but at the moment it's extra code cost for that's not traveled very often. As a result, restructured the indirect delete handling to be a bit more generic, now with a multipurpose lfs_globals_t struct instead of the delete specific lfs_entry_t struct. Also worked on integrating xored-globals, now with several primitive global operations to manage fetching/updating globals on disk.
2018-07-02 03:29:42 +00:00
//}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
memcpy(&pdir, &dir, sizeof(pdir));
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
int lfs_deorphan(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs->deorphaned = true;
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t pdir = {.d.size = 0x80000000};
lfs_mdir_t cwd = {.d.tail[0] = 0, .d.tail[1] = 1};
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// iterate over all directory directory entries
while (!lfs_pairisnull(cwd.d.tail)) {
int err = lfs_dir_fetch(lfs, &cwd, cwd.d.tail);
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
if (err) {
return err;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// check head blocks for orphans
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
if (!(0x80000000 & pdir.d.size)) {
// check if we have a parent
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
lfs_mdir_t parent;
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t entry;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
int res = lfs_parent(lfs, pdir.d.tail, &parent, &entry);
if (res < 0) {
return res;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (!res) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
// we are an orphan
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
LFS_DEBUG("Found orphan %d %d",
pdir.d.tail[0], pdir.d.tail[1]);
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
pdir.d.tail[0] = cwd.d.tail[0];
pdir.d.tail[1] = cwd.d.tail[1];
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &pdir, NULL, 0);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
}
break;
}
if (!lfs_pairsync(entry.d.u.dir, pdir.d.tail)) {
// we have desynced
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
LFS_DEBUG("Found desync %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
pdir.d.tail[0] = entry.d.u.dir[0];
pdir.d.tail[1] = entry.d.u.dir[1];
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &pdir, NULL, 0);
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
if (err) {
return err;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
break;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// check entries for moves
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
lfs_mattr_t entry;
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
while (true) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_next(lfs, &cwd, &entry);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err == LFS_ERR_NOENT) {
break;
}
// found moved entry
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (entry.d.type & LFS_STRUCT_MOVED) {
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
int moved = lfs_moved(lfs, &entry.d.u);
if (moved < 0) {
return moved;
}
if (moved) {
LFS_DEBUG("Found move %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, entry.size, NULL, 0}}, 1);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
} else {
LFS_DEBUG("Found partial move %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
entry.d.type &= ~LFS_STRUCT_MOVED;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
{LFS_FROM_MEM, 0, 1, &entry.d, 1}}, 1);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
}
}
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
memcpy(&pdir, &cwd, sizeof(pdir));
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
/// External filesystem filesystem operations ///
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
//int lfs_fs_getattrs(lfs_t *lfs, const struct lfs_attr *attrs, int count) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
// lfs_mdir_t dir;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// int err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
// if (err) {
// return err;
// }
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// lfs_mattr_t entry = {.off = sizeof(dir.d)};
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// err = lfs_dir_get(lfs, &dir, entry.off, &entry.d, 4);
// if (err) {
// return err;
// }
// entry.size = lfs_entry_size(&entry);
//
// if (err != LFS_ERR_NOENT) {
// if (!err) {
// break;
// }
// return err;
// }
//
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
// lfs_mdir_t cwd;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
// if (err) {
// return err;
// }
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// lfs_mattr_t entry;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// err = lfs_dir_find(lfs, &cwd, &entry, &path);
// if (err) {
// return err;
// }
//
// return lfs_dir_getinfo(lfs, &cwd, &entry, info);
// return lfs_dir_getattrs(lfs, &dir, &entry, attrs, count);
//}
//
//int lfs_fs_setattrs(lfs_t *lfs, const struct lfs_attr *attrs, int count) {
Added the internal meta-directory structure Similarly to the internal "meta-attributes", I was finding quite a bit of need for an internal structure that mirrors the user-facing directory structure for when I need to do an operation on a metadata-pair, but don't need all of the state associated with a fully iterable directory chain. lfs_mdir_t - meta-directory, describes a single metadata-pair lfs_dir_t - directory, describes an iterable directory chain While it may seem complex to have all these structures lying around, they only complicate the code at compile time. To the machine, any number of nested structures all looks the same.
2018-05-29 06:11:26 +00:00
// lfs_mdir_t dir;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// int err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
// if (err) {
// return err;
// }
//
Renamed lfs_entry_t -> lfs_mattr_t Attributes are used to describe more than just entries, so calling these list of attributes "entries" was inaccurate. However, the name "attributes" would conflict with "user attributes", user-facing attributes with a very similar purpose. "user attributes" must be kept distinct due to differences in binary layout (internal attributes can use a more compact tag+buffer representation, but expecting users to jump through hoops to get their data to look like that isn't very user-friendly). Decided to go with "mattr" as shorthand for "meta-attributes", similar to "metadata".
2018-05-29 05:50:47 +00:00
// lfs_mattr_t entry = {.off = sizeof(dir.d)};
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// err = lfs_dir_get(lfs, &dir, entry.off, &entry.d, 4);
// if (err) {
// return err;
// }
// entry.size = lfs_entry_size(&entry);
//
// return lfs_dir_setattrs(lfs, &dir, &entry, attrs, count);
//}
Added lfs_fs_size for finding a count of used blocks This has existed for some time in the form of the lfs_traverse function, through which a user could provide a simple callback that would just count the number of blocks lfs_traverse finds. However, this approach is relatively unconventional and has proven to be confusing for most users.
2018-04-09 03:25:58 +00:00
Adopted EISDIR as internal error for root path as argument Unfortunately, it's hard to make directory lookups for root not a special case, and we don't like special cases when we're trying to keep code size small. Since there are a handful of code paths where opening root should return EISDIR (such as lfs_file_open("/")), using EISDIR to note that the argument is in fact a path to the root. This is needed because we no longer look up an entries contents in lfs_dir_find for free, since entries are more ephemeral.
2018-05-29 06:21:55 +00:00
//static int lfs_fs_size_count(void *p, lfs_block_t block) {
// lfs_size_t *size = p;
// *size += 1;
// return 0;
//}
//
//lfs_ssize_t lfs_fs_size(lfs_t *lfs) {
// lfs_size_t size = 0;
// int err = lfs_fs_traverse(lfs, lfs_fs_size_count, &size);
// if (err) {
// return err;
// }
//
// return size;
//}
Reference in New Issue Copy Permalink